Selecting Talent: The Upshot from 85 Years of Research

I recently wrote about how the "talent wars" are likely to be returning soon in the U.S. (and indeed, there are signs they have already returned in places like China and Singapore), and how companies that have treated people well during the downturn will have an advantage in keeping and retaining the best people --and those that have not damn well better change their ways or will face the prospect of their best people running for the exits in concert with the inability to attract the best people.  A related question has to do with the problem of determining who the best people might be -- what does the best evidence say about the best way to pick new people? 

Its is always dangerous to say there is one definitive paper or study on any subject, but in this case there is candidate -- a paper I have blogged about before when taking on graphology (handwriting analysis). But there is one article that just might qualify. It was published by Frank Schmidt and the late John Hunter in the Psychological Bulletin in 1998. These two very skilled researchers analyzed the pattern of relationships observed in peer reviewed journals during the prior 85 years to identify which employee selection methods were best and worst as predictors of job performance. They used a method called "meta-analysis" to do this, which they helped to develop and spread. The advantage of this method is -- in the hands of skilled researchers like Schmidt and Hunter -- is it reveals the overall patterns revealed by the weight of evidence, rather than the particular quirks of any single study.

The upshot of this research is that work sample tests (e.g., seeing if people can actually do key elements of a job -- if a secretary can type or a programmer can write code ), general mental ability (IQ and related tests), and structured interviews had the highest validity of all methods examined (Arun, thanks for the corrections). As Arun also suggests, Schmidt and Hunter point out that three combinations of methods that were the most powerful predictors of job performance were GMA plus a work sample test (in other words, hiring someone smart and seeing if they could do the work),  GMA plus an integrity test, and GMA plus a structured interview (but note that unstructured interviews, the way they are usually done, are weaker).

Note that this information about combinations is probably more important than the pure rank ordering, as it shows what blend of methods works best, but here is also the rank order of the 19 predictors examined, rank ordered by the validity coefficient, an indicator of how strongly the individual method is linked to performance:

1. Work sample tests (.54)

2. GMA tests ..."General mental ability" (.51)

3. Employment interviews -- structured (.51) 

4. Peer ratings (.49)

5. Job knowledge tests (.48) Test to assess how much employees know about specific aspects of the job. 

6. T & E behavioral consistency method (.45) "Based on the principle that past behavior is the best predictor of future behavior. In practice, the method involves describing previous accomplishments gained through work, training, or other experience (e.g., school, community service, hobbies) and matching those accomplishments to the competencies required by the job. a method were past achievements that are thought to be important to behavior on the job are weighted and score

7. Job tryout procedure (.44) Where employees go through a trial period of doing the entire job.

8. Integrity tests (.41)  Designed to assess honesty ... I don't like them but they do appear to work. 

9. Employment interviews -- unstructured (.38)

10. Assessment centers (.37)

11. Biographical data measures(.35)

12. Conscientiousness tests (.31)  Essentially do people follow through on their promises, do what they say, and work doggedly and reliably to finish their work.

13. Reference checks (.26)

14. Job experience --years (.18)

15. T & E point method (.11)

16. Years of education (.10)

17. Interests (.10)

18. Graphology (.02) e.g., handwriting analysis.

19. Age (-01)

Certainly, this rank-ordering does not apply in every setting.  It is also important to recall that there is a lot of controversy about IQ, with many researchers now arguing that it is more malleable than previously thought. But I find it interesting to see what doesn't work very well -- years of education and age in particular. And note that unstructured interviews, although of some value, are not an especially powerful method, despite their widespread use. Interviews are strange in that people have excessive confidence in them, especially in their own abilities to pick winners and losers -- when in fact the real explanation is that most of us have poor and extremely self-serving memories.

Many of these methods are described in more detail here by the Society for Industrial and Organizational Psychology. Also note that I am not proposing that any boss or company just mindlessly apply this rank ordering, but I think it is useful to see the research.

The reference for this article is:

Schmidt, F.L. & Hunter, J.E. (1998) The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings,” Psychological Bulletin, 124, 262–274.

P.S. Note the corrections, thanks Arun! 

Comments

[watch a video]

I found your blog through a comment on 37Signals regarding interview questions involving puzzles. How I wish some of these companies saw this. I am pretty sure that the folks inside Google do almost similar tasks as I do. While I would like to work there, I don't think I'd be able to crack the puzzle they ask.

[Avidblogreader]

I can't help it, I think it's important to make the following comment:

You have a blog and write serious commentary about serious subjects. Don't insult your readers by posting something that you didn't seriously proofread.

[Skeptic]

As I recall from my statistics class, 0.5 or less was considered a weak correlation rapidly diminishing in significance. So even the top seven or eight tests listed here are marginally useful, the rest junk. Not very encouraging to say the least.

[Ellie]

Rookie question:

Can anyone tell me what what a structured interview consists off compared to an unstructured one?

[Brian]

Hi Bob,

I feel this post certainly passes the litmus test as not snake-oil. So many "consultants" out there focus on those less predictive measures because it would be very hard to charge a company to look for smart people that can demonstrate that they can perform their work. Perhaps hiring is not a black art after all.

[Arun]

Look up page 227 of Handbook of Industrial, Work and Organizational Psychology: Personnel psychology
By Neil Anderson on books.google.com

You will see Schmidt and Hunter's table reproduced.

Table 11.2 Meta-analytic predictive validity of g and a second predictor for job performance.

The column headings are

Predictor
Validity (r)
Multiple R
R - r (superscript l, subscript g)

I produce the first 3 columns of the first few rows

g, .51,
Work sample test, .54, .63
Integrity tests, .41, .65
Conscientiousness tests, .31, .60
Structured interviews, .51, .63

By itself, Work sample test seems to have more validity (0.54) than g (0.51)

It seems to me that this blog post may have high g, but it has low integrity.

[Arun]

This is the article abstract:

"This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity .63). A further advantage of the latter 2 combinations is that they can be used for both entry level selection and selection of experienced employees. The practical utility implications of these summary findings are substantial. The implications of these research findings for the development of theories of job performance are discussed."

http://aprtestingservice.com/ tes...hmidtHunter1998

Doesn't seem to be saying what the blog article says.

[juegos de estrategia]

nice article of sutton

greetings

[Rob G]

As you point out, this research was conducted in 1998, so myself and other I/O Psychologists have known about this for a while. To me one of the most interesting points on this is Years of Experience. In the rank order it is pretty low, and when reading the article the effect size is low as well. Yet go to your favorite career website (Monster, Careerbuilder, The Ladders, etc.) or go to your own internal job post site. Years of experience will almost always be listed first in terms of requirements, and utilized first in resume screening by recruiters and hiring managers. A sub-par predictor becomes the first and usually most important screening criteria. Talk about not utilizing evidence-based management!

[Jan]

As I understand it, you are very likely to hire someone who resembles your own personality. You want someone who looks upon the world the way you do, because then you will understand them, and they will understand you.

Management teams are known for something called homosocial regeneration, which basically is that everyone behaves, thinks and dresses alike, because it gives comfort and a false sense of security (not the same thing as group think).

Maybe we should look more at how creative a person is (if it's a creative job), and less on value-systems and personality. I think this is an awfully difficult subject.

[sk]

Very interesting. The fact that GMA tests are most predictive shows why filtering based on university is commonly used. A school is an imperfect proxy for SAT, GRE, or GMAT tests, which in turn are imperfect proxies for the controversial (at least to this layman) IQ.

Although interviews may not be the most successful predictor in determining the ability of a person to perform a job, it seems they would be the most effective way to insure employees fit the culture. In particular, a way to avoid (or attract) assholes as your culture sees fit.

So, I wonder how much of a role company cultural fit or social interactions played in the various studies used to perform the meta analysis.

It is also interesting that these results show age discrimination (a growing problem) to be the least effective way to create an able workforce.

That's a nice, thought-provoking blog entry for a Friday.