All articles
5 min readai / productivity

30.4% of meeting summaries had a factual error: how to review yours

An AI notetaker joins meetings by default in most of the teams I train. It produces a transcript, then a summary, and the summary is what someone acts on a week later. Two layers of machinery sit between the sentence a person spoke and the bullet point in your follow-up email. Researchers have measured both layers, and the error rates are high enough to change how teams should handle the output.

The transcript is wrong before any summary runs.

Salman Seyedi and 11 co-authors compared transcription services on recorded research psychiatric interviews, scoring every output against transcripts the research team produced by hand. The human service, Rev, had a median word error rate of 7.6%. The best automatic service, Amazon Transcribe, reached 8.9%, and the gap held up statistically (P<.001). Whisper came in at 14.8% 1.

A word error rate of 14.8% puts roughly 1 word in 7 in the wrong place, missing or invented. Most of those errors are harmless: a dropped article, a mangled filler word. Some land on a number, a name or a negation.

Some lines were never spoken.

Allison Koenecke, Anna Seo Gyeong Choi, Katelyn Mei, Hilke Schellmann and Mona Sloane ran Whisper over AphasiaBank, 13,140 English audio samples from speakers with aphasia and from control speakers. In 1.4% of transcriptions, the model produced a phrase or a whole sentence with no counterpart anywhere in the audio. Of the 312 hallucinated transcriptions, 38% carried an explicit harm: 19% perpetuated violence, 13% invented associations between people, 8% implied false authority such as a fabricated credential 2. Hallucinations clustered on speakers with longer non-vocal stretches.

Mateusz Barański and five co-authors pushed on that pattern at ICASSP 2025. Feeding non-speech audio to Whisper large-v3, they got a non-empty transcription back for 40.3% of those inputs 3. The frequent outputs are subtitle artifacts, "thanks for watching" and "subtitles by the amara.org community", left over from training on video transcripts 3.

Long pauses and room noise function as inputs the model tries to decode, so they carry their own error rate.

The summary adds a second set of errors.

Liyan Tang and 13 co-authors built TofuEval: 1.5K topic-focused summaries drawn from MediaSum interview transcripts and MeetingBank city council recordings, with two expert linguists judging each summary sentence for factual consistency against the source dialogue.

Averaged over the five summarizers they tested, 30.4% of MeetingBank summaries written on a main topic of the meeting contained at least one factually inconsistent sentence. On marginal topics, the side threads and asides, the average rose to 43.6% 4. GPT-3.5-Turbo was the most faithful of the five at 10.9% on main-topic MeetingBank summaries. Model size bought nothing: WizardLM-13B made more errors than WizardLM-7B on the same transcripts 4.

The same team tested those models as error detectors. All of them performed poorly at flagging errors in main-topic summaries, and most of the LLM evaluators scored below non-LLM factuality metrics, with GPT-4 the exception 4. Asking the tool to check its own notes recovers little.

Omission is the error class that a reader cannot see. Frederic Kirstein, Terry Ruas and Bela Gipp annotated 200 machine-generated meeting summaries built from AMI, ICSI and parliamentary transcripts: 169 labeled across nine error types, plus 31 error-free summaries as controls. Omission is one of the nine, defined as missing information from the meeting such as a decision or an action 5. A summary that drops a commitment still reads as complete.

Automatic quality scores will not flag it either. Kirstein, with Jan Philip Wahle, Ruas and Gipp, compared metrics against expert annotations on QMSum transcripts: structural disorganization gets penalized, hallucinations are sometimes rewarded, and correlations with human judgment stay weak to moderate 6.

The summarization benchmarks ran on 2023 models.

TofuEval tested Vicuna, three sizes of WizardLM and GPT-3.5-Turbo. No one has published the same expert sentence-level annotation on the models behind today's notetakers, so read 30.4% as a measurement of a previous generation rather than a current spec sheet. The shape of the failure carries forward: marginal topics break more often than main ones, a larger model is no safer than a small one, and the model that wrote the summary makes a weak judge of it. The transcription figures age better, since Whisper and its derivatives still sit under a large share of the market.

The protocol.

  1. Set transcript retention before the meeting, to 30 days. Every check below needs a source to check against. Most tools delete audio long before they delete the summary.
  2. Verify names, numbers and dates against the transcript. Word errors and hallucinations both land on those fields. Search the transcript for each digit and each proper noun that appears in the summary. Budget 5 minutes per hour of meeting.
  3. Ask for quotes, not conclusions. Prompt the tool to put one verbatim line of transcript under each decision and each action item. A claim with no quote under it is the first thing to check.
  4. Delete the sections covering side topics. Marginal topics carried a 43.6% error rate against 30.4% for main ones 4. An aside takes about as long to verify as the main thread and matters less.
  5. List the action items from memory, then open the summary. Comparing two lists exposes omissions 5; reading a single coherent list exposes none.
  6. Check the first 2 minutes, the last 2 minutes and any long pause. Non-speech audio is where Whisper writes "thanks for watching" 3.
  7. Forward only a version you have edited, with your initials and the date on it. Readers then know a human compared it against the source.

Pick one meeting this week and run steps 2 and 5 on it. Count the corrections. That number tells you how much of the review to keep.

Sources.

  1. Seyedi, S., Griner, E., Corbin, L., Jiang, Z., Roberts, K., Iacobelli, L., Milloy, A., Boazak, M., Bahrami Rad, A., Abbasi, A., Cotes, R. O., Clifford, G. D. (2023). Using HIPAA (Health Insurance Portability and Accountability Act)-Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison Study. JMIR Mental Health, 10, e48517. doi:10.2196/48517
  2. Koenecke, A., Choi, A. S. G., Mei, K. X., Schellmann, H., Sloane, M. (2024). Careless Whisper: Speech-to-Text Hallucination Harms. ACM Conference on Fairness, Accountability, and Transparency (FAccT '24). doi:10.1145/3630106.3658996
  3. Barański, M., Jasiński, J., Bartolewska, J., Kacprzak, S., Witkowski, M., Kowalczyk, K. (2025). Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio. ICASSP 2025. doi:10.1109/ICASSP49660.2025.10890105
  4. Tang, L., Shalyminov, I., Wong, A. W., Burnsky, J., Vincent, J. W., Yang, Y., Singh, S., Feng, S., Song, H., Su, H., Sun, L., Zhang, Y., Mansour, S., McKeown, K. (2024). TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization. NAACL 2024. aclanthology.org/2024.naacl-long.251
  5. Kirstein, F., Ruas, T., Gipp, B. (2025). What's Wrong? Refining Meeting Summaries with LLM Feedback. COLING 2025, 2100-2120. aclanthology.org/2025.coling-main.143
  6. Kirstein, F., Wahle, J. P., Ruas, T., Gipp, B. (2024). What's under the hood: Investigating Automatic Metrics on Meeting Summarization. Findings of EMNLP 2024, 6709-6723. aclanthology.org/2024.findings-emnlp.393

Keep reading