All articles
7 min readentrepreneurship / productivity

Structured interviews hit 0.42, unstructured 0.19: how to run one

Jason Dana had students interview each other about their studies, then predict the interviewee's grade point average for the coming semester. Some interviewees were following a rule the interviewer never saw: answer each question at random. The interviewers made sense of those answers, felt as sure about them as about the honest ones, and predicted worse than the people who had seen the background data alone. I have walked out of interviews certain about a candidate, and the certainty is the part that study explains.

The conversation adds confidence and subtracts accuracy.

Dana, Robyn Dawes and Nathanial Peterson ran three studies for Judgment and Decision Making. Participants forecast a student's next-semester GPA from valid background data, with or without an unstructured interview. In the random condition the interviewee answered through a random response system, so the content carried no signal. Interviewers built a coherent impression anyway. Adding an interview to the background data lowered accuracy 1.

The authors name two mechanisms. Sensemaking lets an interviewer assemble a story out of almost any answer. Dilution describes what that story then does: non-diagnostic information weakens the predictive value of the quality information sitting beside it. Watching the interview instead of conducting it did little to protect participants, and in the third study they said they would prefer a random interview to no interview 1.

Confidence tracks none of this. Derek Chapman and David Zweig surveyed 812 interviewees and 592 interviewers in the field. The interviewers were confident they could identify the best candidate whatever amount of structure they used 2.

Structure moves the number.

Paul Sackett, Charlene Zhang, Christopher Berry and Filip Lievens re-estimated the validity of the standard selection methods in 2022, arguing that earlier meta-analyses had overcorrected for restriction of range. Structured interviews came out at 0.42, the top of their table, ahead of job knowledge tests at 0.40, work sample tests at 0.33 and cognitive ability tests at 0.31. Unstructured interviews sat at 0.19. The same reanalysis pulled the much-quoted Schmidt and Hunter values of 0.51 and 0.38 down to those two numbers 3.

The gap is older than the reanalysis. Michael McDaniel, Deborah Whetzel, Frank Schmidt and Steven Maurer pooled 245 validity coefficients covering 86,311 people in 1994 and found 0.44 for structured interviews against 0.33 for unstructured ones 4.

"Structured" has an operational definition behind it. Michael Campion, David Palmer and James Campion catalogued 15 components, split between the content of the interview (questions derived from a job analysis, the same questions for every candidate, limits on prompting and on candidate questions until the end) and the evaluation (rating each answer, anchored scales, several interviewers, interviewer training, notes) 5.

Question type carries part of the effect. Paul Taylor and Bruce Small meta-analysed the two families. Past-behaviour questions ("tell me about a time when") paired with descriptively anchored rating scales reached a mean validity of 0.63, against 0.47 for situational questions ("what would you do if") with the same scales. Interrater reliability ran 0.77 for past-behaviour and 0.79 for situational questions with anchored scales, and fell to 0.73 in past-behaviour studies without them 6.

The debrief undoes the structure.

Scoring every answer buys little if the decision gets made in a room afterwards. Nathan Kuncel, David Klieger, Brian Connelly and Deniz Ones compared two ways of turning assessment data into a choice: a formula, or an expert weighing the data in their head. Across advancement, supervisory ratings, training performance and GPA, the expert route lost validity every time, including when the people doing the weighing knew the jobs and the organisations well. On job performance the gap amounted to better than a 50% improvement in prediction for the formula 7.

Mitchell Hoffman, Lisa Kahn and Danielle Li watched the same decision inside 15 firms hiring low-skill service workers, where median tenure ran 99 days. Introducing a job test raised the duration of hires by about 15%. Managers varied in how often they hired against the test's recommendation, and those exceptions went badly: a firm staffed by managers at the 10th percentile of exception rates would see durations around 15% longer than one staffed at the 90th percentile. The authors checked whether the overriding managers held information the test lacked, and concluded that bias or poor judgment fits the data better 8.

The 0.42 is softer than the ordering.

Give the point estimate less weight than the direction. The 80% credibility interval around Sackett's 0.42 runs from 0.18 to 0.66, and the figure pools two earlier meta-analyses that coded structure differently 3. In-Sue Oh, Huy Le and Philip Roth disputed the decision not to correct for restriction of range in concurrent validation studies, and Sackett, Berry, Lievens and Zhang answered that they endorse the correction wherever the information to apply it exists, while holding severe restriction to be uncommon 910. Julia Levashina, Christopher Hartwell, Frederick Morgeson and Michael Campion opened their 2014 review of the structured interview on the definition of structure and closed it with 12 propositions and 19 open questions 11.

The ordering survives all of it. Every correction either side proposes leaves structured interviews above unstructured ones, and the components of structure cost an afternoon to build.

The protocol.

  1. Write the scorecard before the questions. Four to six criteria taken from the work the person will do in their first 90 days. Every content component of structure starts from that analysis 5.
  2. Fix one list of questions and ask it in the same order to everyone. Six to eight questions, with the follow-up probes written out so each candidate gets the same ones 5.
  3. Ask for past behaviour. "Tell me about the last time you shipped something with an unclear spec. What did you do in the first hour?" The hypothetical version of that question scored 0.47 against 0.63 for the past-behaviour version 6.
  4. Write the anchors before the first interview. One sentence per question describing a weak answer, an adequate answer and a strong answer. Reliability drops in the studies that skip the anchors 6.
  5. Score each answer before you ask the next question. A single rating at the end of the hour is the story you built, not the evidence you collected 1.
  6. Average the scores before anyone speaks. Each interviewer submits per-question ratings, the average sets the ranking, the discussion comes after. The expert debrief costs validity even among people who know the job well 7.
  7. Keep the unscored conversation, and keep it out of the decision. Candidates deserve a real talk about the team and the work. Give it its own slot, score nothing in it, and let no one bring it to the debrief. Non-diagnostic impressions dilute the data you worked to collect 1.
  8. Log every override the day you make it. Write the reason, name the criterion you overruled, and reopen the note at 6 months. Exception rates predicted worse outcomes across all 15 firms in Hoffman's data 8.

Take the role you are hiring for next. Before the first call, write the 5 criteria, the 6 questions and the 3 anchor answers per question, which runs to about 90 minutes of work. Then interview your next two candidates off that sheet and compare the per-question scores against the ranking your gut would have handed you.

Sources.

  1. Dana, J., Dawes, R., Peterson, N. (2013). Belief in the unstructured interview: The persistence of an illusion. Judgment and Decision Making, 8(5), 512-520. doi:10.1017/S1930297500003612
  2. Chapman, D. S., Zweig, D. I. (2005). Developing a nomological network for interview structure: Antecedents and consequences of the structured selection interview. Personnel Psychology, 58(3), 673-702. doi:10.1111/j.1744-6570.2005.00516.x
  3. Sackett, P. R., Zhang, C., Berry, C. M., Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040-2068. doi:10.1037/apl0000994
  4. McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79(4), 599-616. doi:10.1037/0021-9010.79.4.599
  5. Campion, M. A., Palmer, D. K., Campion, J. E. (1997). A review of structure in the selection interview. Personnel Psychology, 50(3), 655-702. doi:10.1111/j.1744-6570.1997.tb00709.x
  6. Taylor, P. J., Small, B. (2002). Asking applicants what they would do versus what they did do: A meta-analytic comparison of situational and past behaviour employment interview questions. Journal of Occupational and Organizational Psychology, 75(3), 277-294. doi:10.1348/096317902320369712
  7. Kuncel, N. R., Klieger, D. M., Connelly, B. S., Ones, D. S. (2013). Mechanical versus clinical data combination in selection and admissions decisions: A meta-analysis. Journal of Applied Psychology, 98(6), 1060-1072. doi:10.1037/a0034156
  8. Hoffman, M., Kahn, L. B., Li, D. (2018). Discretion in hiring. The Quarterly Journal of Economics, 133(2), 765-800. doi:10.1093/qje/qjx042
  9. Oh, I.-S., Le, H., Roth, P. L. (2023). Revisiting Sackett et al.'s (2022) rationale behind their recommendation against correcting for range restriction in concurrent validation studies. Journal of Applied Psychology, 108(8), 1300-1310. doi:10.1037/apl0001078
  10. Sackett, P. R., Berry, C. M., Lievens, F., Zhang, C. (2023). Correcting for range restriction in meta-analysis: A reply to Oh et al. (2023). Journal of Applied Psychology, 108(8), 1311-1315. doi:10.1037/apl0001116
  11. Levashina, J., Hartwell, C. J., Morgeson, F. P., Campion, M. A. (2014). The structured employment interview: Narrative and quantitative review of the research literature. Personnel Psychology, 67(1), 241-293. doi:10.1111/peps.12052

Keep reading