95.5% to 89.0%: stop asking a model to check its own work
Two rounds of self-review dropped GPT-4 from 95.5% to 89.0% on GSM8K. Eight studies on why models cannot grade their own reasoning, and the external checks that do work.
Read the article →Research-backed, actionable articles on AI at work, endurance training and how people learn. Every claim has a source.
Two rounds of self-review dropped GPT-4 from 95.5% to 89.0% on GSM8K. Eight studies on why models cannot grade their own reasoning, and the external checks that do work.
Read the article →Reasoning accuracy fell from 92% to 68% between 250 and 3,000 tokens. Four controlled studies on long-context failure, and the prompt layout I teach.
Read the article →Generative AI lifts what one person produces and pulls a team's ideas together. Five studies on idea homogenization, and a protocol for your next workshop.
Read the article →Models flip their answer in 58% of rebuttals and affirm users 49% more often than humans do. Five studies on AI sycophancy, and the prompts that hold the line.
Read the article →Nine sources on prompt injection: hijack rates measured on real agents, why filters fall to adaptive attacks, and the scoping rules to apply before shipping one.
Read the article →Nine studies on hallucinated references: the fabrication rates by model and field, why retrieval tools still miss 17 to 33%, and the check that catches them.
Read the article →Reordering two answers handed the win to the weaker model on 66 of 80 questions. Ten studies on LLM judges, their biases, and a validation protocol.
Read the article →Eight studies on the security of AI-written code: the failure rates by language, the confidence gap it creates, and the review routine that catches the weaknesses.
Read the article →Students predicted 33.9 days for a thesis and took 55.5. Ten sources on the planning fallacy, where it reverses, and a forecasting routine built on finished projects.
Read the article →Human and AI teams scored below the better of the two alone across 106 experiments. Ten studies on why we accept wrong answers, and a protocol that puts yours first.
Read the article →