All articles
6 min readfocus / productivity

Teams missed 23% of steps from memory: how to write a checklist

Most teams I train leave the room with a checklist. Three months later it has become a box someone ticks after the work is finished. Surgery ran this experiment at scale with death as the outcome, once across 101 hospitals, and the effect tracked one variable: whether anyone read the list out loud and finished it.

What the research shows.

Alex Haynes and his co-authors took a 19-item checklist to eight hospitals on four continents, from Seattle to Ifakara in Tanzania, and tracked 7,688 non-cardiac surgical patients: 3,733 before the checklist, 3,955 after. Death fell from 1.5% to 0.8% (P = 0.003). Inpatient complications fell from 11.0% to 7.0% (P < 0.001) 1.

The Dutch SURPASS group ran a longer version of the same idea in six teaching hospitals that already held high baseline standards. Complications went from 27.3 to 16.7 per 100 patients, counted as events rather than patients affected, and in-hospital mortality from 1.5% to 0.8%. Control hospitals over the same period showed no change 2.

Both of those compare one period to the period before it. Haugen and colleagues randomized instead, in a stepped wedge design across two Norwegian hospitals: 2,212 procedures in the control stage against 2,263 with the checklist. Complications fell from 19.9% to 11.5%, an absolute risk reduction of 8.4 points (95% CI 6.3 to 10.5). Mortality dropped at one of the two hospitals and the overall change did not clear significance 3.

Bergs and colleagues pooled seven studies, none of them randomized, and reported risk ratios of 0.59 for any complication (95% CI 0.47 to 0.74), 0.77 for mortality (0.60 to 0.98) and 0.57 for surgical site infection (0.41 to 0.79) 4.

Alexander Arriaga's team tested the other kind of list, the one for when things go wrong. 17 operating-room teams worked 106 simulated crises, half with a set of crisis checklists and half from memory. With the checklists they missed 24 of 371 critical steps. From memory they missed 89 of 379. That is 6% against 23%, an adjusted relative risk of 0.28 (95% CI 0.18 to 0.42). Each of the 17 teams did better with the page in front of them 5.

Peter Pronovost's version carried only five items, on inserting a central line, and arrived with the authority and the feedback loop around it. Across 103 Michigan ICUs the median rate of catheter-related bloodstream infection went from 2.7 per 1,000 catheter-days to 0 in the first three months, the mean from 7.7 to 2.3, and the reduction held through 18 months 6.

A mandated rollout in Ontario moved nothing.

David Urbach and colleagues examined 101 Ontario hospitals after checklists became provincial policy, comparing three-month windows either side of each hospital's adoption date. Adjusted 30-day mortality went from 0.71% to 0.65%, odds ratio 0.91 (95% CI 0.80 to 1.03, P = 0.13). Complications sat at 3.86% and then 3.82%, odds ratio 0.97 (P = 0.29) 7.

Wilton van Klei's cohort of 25,513 surgical patients accounts for part of that gap. Crude 30-day in-hospital mortality moved from 3.13% to 2.85%, which did not clear significance (P = 0.19). Adjusted for case mix the odds ratio was 0.85 (0.73 to 0.98). Split by compliance the picture sharpens: full checklist completion carried an odds ratio of 0.44 (0.28 to 0.70), while partial completion and non-compliance carried no measurable effect 8.

Conley and colleagues interviewed implementation leaders and surgeons at five Washington State hospitals and heard the same division. Leaders who opened with the evidence and named the gains they expected in safety, efficiency and teamwork got the list used. Staff who received a form with no reason attached filled it in like a form 9.

The protocol.

  1. Keep only the items that get missed and that hurt when missed. Asaf Degani and Earl Wiener, studying flight-deck checklists for NASA, found that long lists push crews to run the handful of steps they judge critical and to drop the cross-check the list exists for 10. The surgical checklist spreads 19 items across three pause points, so six or seven per stop. Draft your candidates, then cut any step the team already completes unprompted.
  2. Attach the list to a pause point. Three of them work: before the task starts, before the step you cannot undo, before the handoff. A list run during the work gets bunched into one "yes".
  3. One person reads, another answers. Degani and Wiener call this call-do-response, and they recommend the answer carry the actual status or value instead of "set" or "checked" 10. On a deploy: "Last restore test?" answered with "02:14 this morning, 4.1 GB", never with "done".
  4. Log the completion rate, not the adoption rate. van Klei's effect sits entirely in full completion 8. Record which items got answered on the last ten runs. Below 80% full completion, shorten the list rather than lecturing the team.
  5. Introduce it with the numbers. Spend the first five minutes of the rollout on the evidence above and on the gains you expect. Conley's hospitals that skipped this step collected signatures 9.
  6. Write a crisis list for the three failures you have handled badly. Arriaga's teams missed close to a quarter of the critical steps when they worked from memory under pressure 5. One page per scenario, stored where the on-call engineer will find it at 03:00.
  7. Cut one item per quarter. Any item answered the same way on ten consecutive runs has turned into a habit, so take the slot back and give it to something that still fails.
  8. Give the list one owner who reads it aloud. In my trainings the lists that survive six months all have a named reader. No study behind this one; it comes from the sessions I run.

Open your deploy runbook and count the items that come before the first irreversible step. More than nine, cut it to seven. Run it out loud with one other person on the next deploy, and write down which of the seven came back with a number.

Sources.

  1. Haynes, A. B., Weiser, T. G., Berry, W. R., et al. (2009). A surgical safety checklist to reduce morbidity and mortality in a global population. New England Journal of Medicine, 360(5), 491-499. doi.org/10.1056/NEJMsa0810119
  2. de Vries, E. N., Prins, H. A., Crolla, R. M. P. H., et al. (2010). Effect of a comprehensive surgical safety system on patient outcomes. New England Journal of Medicine. doi.org/10.1056/NEJMsa0911535
  3. Haugen, A. S., Softeland, E., Almeland, S. K., et al. (2015). Effect of the World Health Organization checklist on patient outcomes: a stepped wedge cluster randomized controlled trial. Annals of Surgery, 261(5), 821-828. doi.org/10.1097/SLA.0000000000000716
  4. Bergs, J., Hellings, J., Cleemput, I., et al. (2014). Systematic review and meta-analysis of the effect of the World Health Organization surgical safety checklist on postoperative complications. British Journal of Surgery, 101(3), 150-158. doi.org/10.1002/bjs.9381
  5. Arriaga, A. F., Bader, A. M., Wong, J. M., et al. (2013). Simulation-based trial of surgical-crisis checklists. New England Journal of Medicine, 368(3), 246-253. doi.org/10.1056/NEJMsa1204720
  6. Pronovost, P., Needham, D., Berenholtz, S., et al. (2006). An intervention to decrease catheter-related bloodstream infections in the ICU. New England Journal of Medicine, 355(26), 2725-2732. doi.org/10.1056/NEJMoa061115
  7. Urbach, D. R., Govindarajan, A., Saskin, R., Wilton, A. S., & Baxter, N. N. (2014). Introduction of surgical safety checklists in Ontario, Canada. New England Journal of Medicine, 370(11), 1029-1038. doi.org/10.1056/NEJMsa1308261
  8. van Klei, W. A., Hoff, R. G., van Aarnhem, E. E. H. L., et al. (2012). Effects of the introduction of the WHO "Surgical Safety Checklist" on in-hospital mortality: a cohort study. Annals of Surgery, 255(1), 44-49. doi.org/10.1097/SLA.0b013e31823779ae
  9. Conley, D. M., Singer, S. J., Edmondson, L., Berry, W. R., & Gawande, A. A. (2011). Effective surgical safety checklist implementation. Journal of the American College of Surgeons, 212(5), 873-879. doi.org/10.1016/j.jamcollsurg.2011.01.052
  10. Degani, A., & Wiener, E. L. (1990). Human factors of flight-deck checklists: the normal checklist. NASA Contractor Report 177549. ntrs.nasa.gov

Keep reading