I'm open to the possibility that the traditional process is still the best way to select a colleague. But has anyone studied this question? Is anyone familiar with the personnel selection literature as applied to high-end positions like scientists, professors and researchers?
Here's a highly cited article that a quick search turned up. GMA = "General Mental Ability" :-)
The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings
Full paper
This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and Ihe 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity of .63). A further advantage of the latter 2 combinations is that they can be used for both entry level selection and selection of experienced employees. The practical utility implications of these summary findings are substantial. The implications of these research findings for the development of theories of job performance are discussed.
... As noted above, GMA is also an excellent predictor of job-related learning. It has been found to have high and essentially equal predictive validity for performance (amount learned) in job training programs for jobs at all job levels studied. In the U.S. Department of Labor research, the average predictive validity performance in job training programs was .56 (Hunter & Hunter, 1984, Table 2); this is the figure entered in Table 2. Thus, when an employer uses GMA to select employees who will have a high level of performance on the job, that employer is also selecting those who will learn the most from job training programs and will acquire job knowledge faster from experience ... Because of its special status, GMA can be considered the primary personnel measure for hiring decisions, and one can consider the remaining 18 personnel measures as supplements to GMA measures.
This comment has been removed by the author.
ReplyDeleteJob performance is evaluated in a perfectly objective manner. Am I wrong about that?
ReplyDeleteIt couldn't be that supervisors rate higher those from the same social background as themselves, could it?
True enough in general, but I don't think hiring is the best example of cognitive biases. I asked a management colleague, who says (in essence) that there are no sure-fire methods for things like academic recruiting. Maybe Google's in a different world, I don't know. Has there been serious study of their methods? How do they measure success? In the end, lots of this is random -- lots of work shows that there's not much you can do about that.
ReplyDeleteI'd add that part of success is your own environment: it's not always survival of the fittest. More than that: good environments attract good people -- see the pix from your back door.
It couldn't be that supervisors rate higher those from the same social background as themselves, could it?
ReplyDeleteWhat does that mean? "Social background" sounds like something you could redefine in each instance to fit your preconceived notions. I personally have seen very little evidence of that, particularly at higher levels where a poor performing subordinate could cost you big bucks come bonus time. I mean, would you string along some boob just because you feel more comfortable with his background? Not too many managers are going to last very long with that approach.
I get the part about error in measuring performance, but for some jobs one can do it pretty well. If you can at least estimate the error (uncertainty) you can correct for it in estimating validity.
ReplyDeleteDave: one thing that the literature did seem to agree on is that structured interviews are better than the kind of unstructured (get to know ya, talk some physics) thing that is usually done in all academic fields I am familiar with. Everyone in my field is sure they can "tell" who the good people are from talking to them and listening to a seminar. It might be true, but I don't know of any systematic analysis... In my own case I would have felt pretty confident in the past (with no real track record to go on), but studying cognitive bias stuff has made me wonder :-)
PS I know there are some Goldman types reading this blog ... let's hear how it's done in the pros :-)
ReplyDeleteInteresting, thanks for mentioning. Been wondering the same thing. I'll check this out.
ReplyDeleteOne thing that strikes me as counterproductive in academic searchers is (at least in my experience) everybody just wants to get done with it. And then there's the power fighters.
Anyway, I am vary of using selection methods based on measures in particular in science. The problem with any sort of measure is that once you start employing it, you're deviating people's goals. As a consequence, the measure will become less useful, even though it originally had a correlation with what you were trying to look for. A rather infamous example is the number of papers. There's means to increase the paper count that doesn't increase or even decrease the quality of scientific research.
I lean towards technical interviews (math and programming questions), and I'm occasionally shocked when somebody with a strong resume and positive self-evaluation can't draw a plot of f(x)=x^2.
ReplyDelete"Communication skills" means the same thing in the US that accent means in the UK.
ReplyDeleteSpeaking in RP in the UK is equivalent to having excellent "communication skills" in the US.
A well-known and much cited paper that discusses evaluation of candidates for admission to graduate school, among other applications of improper linear models, is the following:
ReplyDeleteDawes, Robyn M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, volume 34, pages 571-582.
It is cited in Wikipedia's article on unit-weighted regression. This subject has broader implications, and I think it deserves some attention on your blog.
[Robyn M. Dawes is currently emeritus professor of psychology in the Department of Social and Decision Sciences at Carnegie-Mellon University.]
One of the common reasons for skepticism about the value of linear models, and especially improper linear models, is the fact that they often aren't as predictive as we would like, or as we imagine competent human judges could be. One of Dawes' main points is that the latter assumption has rarely if ever been supported by evidence, despite extensive study of the question.
This question then arises: What kind of predictive success can we reasonably expect in such applications? With respect to evaluation of candidates for graduate school, where the criterion is successful completion of a Ph.D., Dawes reminds us that unanticipated events will inevitably play a role in the actual progress and performance of a candidate once they're admitted, e.g., the vicissitudes (good and bad) of their health, love lives, finances, and family dynamics. These events are, so to speak, not encompassed by a candidate's "past light cone" at the time of evaluation, so they can't play a role in the evaluation. Ultimately our expectations of predictive success need to be modest, notwithstanding the fact that there is pain associated with failed predictions—dashed hopes and disappointment on the part of the people and institutions with a stake in the outcome.
Structured interviews where you see if people can solve problems are better than the usual shoot-the-breeze interview.
ReplyDelete