| Psychometrika – Journal of Psychometrics Society Free Trial Access until July 31, 2010! |
| From now until July 31, 2010, please enjoy free access to the journal. Click here to begin your reading. |
Archive for the ‘Psychometric Test Research’ Category
Psychometric Journal – Free Trial Access Until July 31
Monday, June 28th, 2010
HRM Webinar: How Chinese are the Chinese? A look at Personality Tests for the China
Wednesday, May 19th, 2010| Join us for a Webinar on June 22 | |||||||||||||||||||||||||||||||||
|
 |
|
|||||||||||||||||||||||||||||||
Replacing psychometric tests with biological tests?
Friday, April 16th, 2010Last week the Singapore-based Straits Times published an article which reported a study carried out at NUS linking testosterone to advantages in leadership. PsyAsia’s Dr. Graham Tyler was asked to comment on the reasearch and he was asked if biological testing might replace psychometric testing in the future! A copy of the article can be downloaded here:
There are a number of limitations in the above thinking. Not least the fact that the study was carried out on students – hardly real-world organisational leaders! Also, given that this was a correlational analysis it cannot be inferred that testosterone increases leadership skills. It might actually be the opposite. Finding oneself in leadership scenarios may cause the body to produce more testosterone!
Biological testing is certainly not about to replace psychometric tests in Singapore or anywhere else. Not only is the area contentious it would also bring legal issues into play. We know already that biology plays a role in many aspects of who we are from personality traits to ability and more. However, we also know that the environment, access to resources and simply luck (ie where we find ourselves, who our teachers/mentors are) also play a huge part. Psychometric tests should never be used on their own to inform decision-making processes and neither should any other form of assessment – biological testing, interviews and so forth. However, the use of highly valid tools used in conjunction with each other is a good thing.
Note: In the article, there is a mention that aptitude tests can predict between 9-36% of performance. This is incorrect. The journalist asked about personality tests and was told that traditional personality tests can predict about 9-36% of performance. Aptitude tests are able to predict more than this and more modern personality assessments such as Identity or the Saville Consulting Wave do an even better job than traditonal personality tests!
Psychometric Webinar – Amazing Apollo Profile Personality Assessment
Wednesday, March 17th, 2010The Amazing Apollo Profile
This free webinar will be facilitated by Mr. Jim Bowden, the developer of the Apollo Profile. The session will be interactive (provided attendees kit themselves out with headphones and a mic!) and Jim will present numerous interesting case studies.
The webinar will cover the following:
• Introduction: The Amazing Apollo Profile- can transform Recruitment, Staff Development, and Organisation Performance –Client example
• Apollo Questionnaire – valid/reliable/comprehensive
• Why is Apollo amazing? Apollo Advantages
• Using and interpreting of Apollo reports with anecdotes
• Recruitment – Accurate, easy, low cost – Case Study using Apollo Best Match in China for filtering 12,000 applicants for 40 Graduate level jobs
• Training and Development – Unique Apollo report PLUS downloadable solutions. Convenient, low cost, motivating
• Organisation Development. Benchmarking: Can analyse and identify current corporate strengths and weaknesses – then create high performing models/culture, identify engagement issues – case studies
• Customising: Develop models that work specifically for your organisation. If your organisation is serious about leadership through people.
• Integrate everything together with flexible multi-purpose Internet Online solutions. Use your own competencies frameworks and vocabulary – examples
• Special Offer – have to listen to Webinar to find out!
Date: Monday, May 17, 2010
Time: 12:30 PM – 1:30 PM SGT
After registering you will receive a confirmation email containing information about joining the Webinar.
System Requirements
PC-based attendees
Required: Windows® 7, Vista, XP, 2003 Server or 2000
Macintosh®-based attendees
Required: Mac OS® X 10.4.11 (Tiger®) or newer
Space is limited.
Reserve your Webinar seat now at:
https://www1.gotomeeting.com/register/522465752
Identity Personality Test Research in Hong Kong
Wednesday, February 10th, 2010Identity Questionnaire Research Results – A synopsis
No reproduction without permission.
Introduction to the Study and Outline of the Phases
In September 2008, Quest Partnership Ltd, PsyAsia International, and the Hong Kong Institute of Vocational Education (HKIVE) embarked on a project to translate the Identity Self-Perception Questionnaire from English into traditional Chinese. The reason for translating the questionnaire was to produce an occupationally focused personality questionnaire that could be used in China and Hong Kong SAR. At the same time, Quest were also producing a new Careers Report for the Identity system. This enabled the volunteer students to gain useful feedback on their questionnaire. The project was headed up by Max Choi of Quest Partnership Ltd and Dr. Graham Tyler of PsyAsia International. Max Choi is an Occupational Psychologist with BPS chartered status and has substantial experience in designing and validating tests. Graham Tyler is a registered psychologist and has a PhD based on psychometric assessment and validating tools for predicting performance at work in Asia.
The research was split into several stages:
• Translations – involving the translation and back-translation of Identity into Simplified and Traditional Chinese by professional staff at HKIVE.
• Pilot Study – using the translated Identity questionnaire.
• Phase 1 Testing – a sample of participants at HKIVE completed the Chinese Identity questionnaire.
• Phase 2 Re-testing – participants were asked to complete the questionnaire for a second time one month later i.e. re-testing to determine the reliability of the questionnaire items.
• Data Cleansing – first to identify and remove ‘rogue’ answer sheets from students who did not complete the questionnaire seriously.
• Data Analysis & Results– analysis of the data and understanding the results.
• Producing Norms and Building this into the New Career Focus Report – norms were produced based on these Hong Kong students. This norm group was used for the new Career Focus Report which is now available for the Hong Kong education sector.
• Translation into Simplified Chinese – the project to translate the Identity Questionnaire into simplified Chinese and have it available online was completed in December 2009.
Translations
In September 2008, the questionnaire was translated into both Traditional and Simplified Chinese by 4 individuals at the HKIVE who hold the British Psychological Society’s Level A and B Certificates of Competence in Occupational Testing. This process was supervised by Dr. Graham Tyler, who has a good understanding of principles behind item construction. The translated questionnaire was sent to the test publishers (Quest Partnership Ltd) in the UK for evaluation and further refinement, working with Chinese natives now resident in the UK.
The translated questionnaire was then back-translated into English by lecturers in the English language department at the HKIVE. Independent back-translation provides the quality check of how effective the translation has been. The back-translation was checked against the original version of the questionnaire to ensure it retained its overall theme and meaning. A few items achieved poor back-translations and these were reviewed and improved and back-translated again to check that the translation had improved. The traditional Chinese translation took precedence on the basis that this would be evaluated first and then simplified Chinese would follow at a later date.
Pilot Sample
20 students at VTC completed the translated traditional Chinese questionnaire. They also completed a form which collected their feedback on items that they did not fully understand or where they felt the wording could be improved. This feedback was analysed and a few minor improvements were made for the next phase.
Phase 1 Testing
In October 2008, a large sample of 800+ Chinese students at HKIVE completed the Traditional Chinese Identity Questionnaire. Most of these administrations were conducted under standardised test administration conditions during classes. The final sample after data cleansing consisted of 421 students.
Phase 2 Testing
One month later many of the Chinese students from the Phase 1 testing were invited to complete the questionnaire again. The test-retest study is based on 206 students who completed the questionnaire again. Most of these administrations were conducted under standardised test administration conditions during classes.
Students were entered into a monetary prize draw as an incentive to take part in the research. Also, students received a Career Focus Report from their completed questionnaire.
Data Cleansing
Identifying ‘Rogue’ Responses
We placed stringent requirements on the data that could be used. It was evident that a proportion of the student responses were not usable. This may be as a consequence of asking the students to complete the questionnaire as part of class work. So although they were volunteers, the request during class time may have resulted in some slightly ‘reluctant’ volunteers. Also, others may have become bored after starting the questionnaire and may not have taken the whole questionnaire seriously, unlike real candidates applying for jobs. So a small minority will complete the questionnaire in a non-serious manner. Only a few rogue answer sheets can be visually identified (e.g. students who have put in the same response for the whole column or making neat zig-zag patterns on the answer sheet). So we needed to employ more sophisticated techniques to identify other ‘rogue’ respondents in order to remove these from our sample before conducting further analysis on the data.
Removing Answer Sheets with Too Many ‘3’ Responses
The instructions for completing the questionnaire clearly states that 3 should be used sparingly. But for this Chinese student sample, the mean number of ‘3’s chosen was 30.4, with a Standard Deviation (SD) of 32. For our UK sample however, the mean number of ‘3’s chosen was 9.85, with a SD of 15. It was decided that participants who responded with over 71 unsure ‘3’ responses would be removed from the sample i.e. this means that they are putting down ‘3’ to over a third of their questionnaire items – which is much too high. A caveat to this however is that given the “middle-way” philosophy in the East, it can generally be anticipated that central tendency responding will be higher in China than in the West.
Removing Answer Sheets with Random Responses
We employed two established methods to detect answer sheets which were being completed randomly i.e. the True Response Inconsistency (TRIN) and the Variable Response Inconsistency (VRIN) methods. Both methods are based on paired items which are highly associated in that knowing an individual’s response to one item will provide a very high level of prediction of their response to the other item. Therefore, when a person scores below a certain threshold with many paired items, we can be confident that their responses to the questionnaire have been random.
Data Analysis and Results
Test Re-Test Reliability
At Phase 2, students completed the Identity Questionnaire again about one month after the Phase 1 original completion of the questionnaire; we were able to conduct a Test-Retest analysis. This allows us to look into the stability or reliability of the questionnaire over time.
The final sample size for the test-retest was 206 after all the data cleansing procedures were conducted. Overall the vast majority of Identity scales were reliable. A small number of scales were below the benchmark of .70. However we need to be reminded that we are dealing with a translated questionnaire so we would expect some loss of reliability compared to the original questionnaire. So the original English Identity questionnaire sets the upper limits.
The original English Identity single scale test-retest coefficients ranged from .77 to .92 (based on a test-retest sample of 121). For the translated traditional Chinese questionnaire the test-retest coefficients ranged from .58 to .87. Seven of the 36 Identity questionnaire scales reported less than ideal test-retest coefficients:
• Consultative .57
• Psychological .61
• Empathy .57
• Adaptability .60
• Theoretical .62
• Rational .59
• Reflective .58
Interestingly, it might be argued that these scales are less meaningful to this student sample and different results are likely to be obtained in a business sample.
Internal Consistency Reliability
Another method to determine reliability is to look at internal consistency of each scale to see how well items within a scale correspond with one another. From this analysis we identified nine scales at a lower range of reliability coefficients than our ideal of 0.7:
• Social Presence .60
• Direct .61
• Empathy .58
• Adaptability .60
• Decisive .67
• Self Potency .53
• Self Protecting .62
• Social Desirability .63
• Reflective .43
Combining the two methods of establishing reliability it was useful to see if there were any scales that would have both low test-retest and low internal consistency reliability. The following 2 scales had lower reliabilities than ideal:
• Empathy
• Adaptability
We will be collecting more data so with more extensive use of the tool with participants who will be completing the questionnaire for non-research purposes we do expect the reliabilities to improve.
Study Results: Comparisons with UK Data
The results for this group of Hong Kong students were compared against the UK working population and also against a group of UK A Level applicants and Final Year Students for a Design & Technology course at a UK university.
The group of Hong Kong students compared to the other groups tended to be slightly lower on the following scales:
• Independent
• Critical
• Multi-Tasking
• Variety Seeking
• Determined
• Self Potency
• Positive
However, it is not possible to determine exactly why these differences are found as there are a range of variables as to how the groups differ from each other e.g. motivational aspects as the students were volunteers rather than real job applicants; age differences; cultural and educational experience differences; work experience differences.
Producing Norms & Developing the Career Focus Report
A set of Hong Kong student norms has been established (N= 421) and more data will be added to this at a later date when it becomes available.
At the same time as this research Quest Partnership also developed a new Career Focus Report for Identity and participating students were provided with a report. This new report has been developed with educational clients in mind but can be used by other clients supporting individuals with career guidance. Currently, the report can be normed against the UK working population and the Hong Kong students.
Translation into Simplified Chinese
The project then made traditional Chinese available as an online solution for clients with a view to collect on-going norms data and to work with any clients who can support with validation studies. In December 2009 the simplified Chinese version was also made available online.
If you are interested in training to use the Identity Questionnaire or if you would like to work with PsyAsia in distributing this assessment, please do get in touch with us.
Do men cheat more on internet selection tests?
Wednesday, November 18th, 2009MEN are considered much more likely to cheat on internet selection tests than women, a Saville Consulting
survey reveals. The report, commissioned by work psychologists at Saville Consulting, questioned over a
thousand UK students. In this sample, 42% thought men were more likely to cheat than women, compared to just 9% who thought that females were more likely to cheat than males.
Internet selection tests measure a respondent’s abilities in certain areas to predict their likely level of performance at work. Tests typically measure verbal, numerical or logical skills. Many organisations use such tests as part of their recruitment process, but the vast majority of such tests can be completed by job applicants on their home computer, in an unsupervised manner.
Worryingly, 14% of students openly admitted to receiving help from others on such internet tests. Overall, respondents also revealed that they knew more men who had cheated on internet tests than women. When students who rated themselves as being of lower ability were compared to those who reported being of a higher level of ability, 58% more students of lower ability admitted to receiving help on internet tests.
Previous research commissioned by Saville Consulting indicated that personality and intelligence tests were actually seen by people as being more useful selection methods than job interviews. Eight out of ten students in the current survey confirmed that many people exaggerate on job application forms. Moreover, the same proportion of respondents agreed that if an organisation followed unsupervised tests with longer supervised tests prior to appointment, this was an effective method for preventing cheats from being selected.
These findings undermine the use of unsupervised tests without the re-testing of applicants
under supervised conditions. Organisations may not re-test applicants because they believe this
offers a more economical recruitment process. However, recruiting dishonest and less intelligent
workers will inevitably be counterproductive for an organisation’s effectiveness.
Professor Peter Saville, International Chairman of Saville Consulting, said:
“Following up applicants who complete online unsupervised tests with supervised tests of equal or greater rigour is arguably the most effective way to prevent candidate cheating. Other methods, such as randomising the order of questions or adapting questions to the ability of a candidate simply do not get around the problem of unsupervised cheating. If job applicants know that they are going to be retested in a properly supervised environment, this can do much to reduce the problem of cheating. The supervised test at the later stage also provides a safeguard that the successfully appointed applicant has the abilities to match the requirements of the role.”
Can work performance be predicted from conscientiousness in personality?
Thursday, November 12th, 2009There has been much research conducted examining the relationship between Conscientiousness as defined as one of the dimensions in the “Big Five” Model of personality and job performance. Much of the research has indicated that Conscientiousness is a positive predictor for job performance across a range of job positions. Simply put and what many advocate, is that it appears that people who have been assessed to be higher on Conscientiousness tend to perform better at work. Nevertheless, an article by Tett (1998) highlights some considerations regarding accepting this conclusion and cautions against accepting this at face value.
In this article, a number of examples suggest that Conscientiousness and job performance may not be irrevocably linked as previous research has suggested. It provides two examples of work situations where Conscientiousness may actually hinder the job performance. Firstly, the author highlights situations where being conscientious may result in lower productivity where either tasks require more time to complete or where fewer tasks can be completed. This is illustrated by the example of managers who at times are required to make decisions even when they do not have all the relevant information (Tett, 1998). In such a situation, being high on conscientious may affect the speed at which decisions can be made.
The second example refers to rules and procedures; conscientious individuals tend to adhere to rules and procedures. Tett (1998) highlights that in certain professions, strictly following rules and procedures may affect creativity and innovation. It is proposed that such adherence to rules and procedures can affect productivity as such individuals would be unlikely to develop novel ideas and solutions.
Although Conscientiousness appears to remain as a significant predictor of job performance, it is important to note that there are situations where other factors can play a part.
Tett, R. P. (1998). Is Conscientiousness ALWAYS positively related to job performance? The Industrial-Organizational Psychologist, 36(1)
International Society for Intelligence Research provides grants to students
Thursday, August 6th, 20091. ISIR has instituted a new program to help defray travel costs for post-doctoral students and faculty members without travel support or with only partial support. We will be able to support ten or more persons. First priority will go to those presenting papers who are post-doctoral students or junior faculty without travel support. Travel will be funded for up to $1,500 per person. This program is being funded by ISIR.
2. The second program supports student travel and is fully explained in the conference announcement found at http://www.isironline.org/meeting/. This program is supported by the Templeton Foundation grant that we recently received. To apply for either program, send an email by September 25th to detterman@case.edu with an attachment stating 1) Name, 2) Affiliation, 3) Current Status (e.g., 3rd year graduate student, postdoctoral student, faculty member, etc.), 4) Travel support currently available to you, 5) Indicate if you have submitted a paper to be presented at the conference and if you will be the presenter, 6) Any other information that may be relevant to your application. Note that, because of the new program, the deadline for application to both programs has been extended to September 25, 2009.
