All articles
5 min readai / productivity

14.2 points on symbolic tasks, 0.7 elsewhere: when to let a model think

In most of the AI trainings I run, someone asks whether to leave the reasoning mode on by default. Reasoning tokens cost money and add seconds to a call, so the question deserves a number. The largest audit of chain-of-thought so far puts the average gain at 14.2 points on symbolic tasks and 0.7 points on everything else.

What the research shows.

Zayne Sprague and colleagues at UT Austin ran a meta-analysis of more than 100 papers, then re-ran their own evaluations across 20 datasets and 14 models 1. Three task families carried the gains: symbolic reasoning at 14.2 points on average, math at 12.3, logical reasoning at 6.9. Across the remaining categories, accuracy with chain-of-thought averaged 56.8 against 56.1 without it.

The same paper hands you a test you can run on your own prompts. On MMLU, answering directly matched chain-of-thought accuracy unless the question or the model's own response contained an equals sign 1. An equation in the text predicted the benefit. Commonsense questions, knowledge lookups and reading comprehension gained close to nothing.

One more result from that paper changed how I wire systems. Most of chain-of-thought's advantage comes from the execution step, and a symbolic solver beats the model's own step-by-step arithmetic 1. When a task needs computation, give the model a calculator or a Python sandbox instead of more tokens to think with.

Where thinking makes the answer worse.

Ryan Liu and co-authors at Princeton took tasks from psychology where asking people to deliberate out loud lowers their scores, then gave the same tasks to models 2. Three of them broke: spotting patterns in strings produced by an artificial grammar, matching a target face against candidates, and classifying data whose rules carry exceptions. The drop reached 36.3 points of absolute accuracy for o1-preview compared with GPT-4o. On the two other tasks the effects went in both directions, so the result covers pattern tasks rather than reasoning in general.

Aryo Pradipta Gema and co-authors built tasks where a longer reasoning budget lowers accuracy outright 3. Claude Opus 4 answered a counting question near the ceiling with short reasoning and fell to roughly 80% once Python code distractors appeared and the budget grew. In a regression task, extended reasoning pulled models off sensible priors and onto plausible-looking spurious features. Few-shot examples repaired most of that one.

The reasoning trace is not an audit trail.

Miles Turpin and colleagues added biasing features to prompts, such as reordering the options so the right answer was always (A), and measured accuracy on BIG-Bench Hard falling by as much as 36% 4. The explanations stayed fluent and skipped the bias that had produced them.

Anthropic's alignment team ran the same experiment on reasoning models. Yanda Chen and co-authors slipped a hint into evaluation questions, confirmed the model had used it, then checked whether the chain of thought admitted it: 25% of the time for Claude 3.7 Sonnet, 39% for DeepSeek R1, and under 20% in most settings 5. Treat the trace as a draft and check the answer against the source.

The protocol.

  1. Sort the task before you pick a mode. Reasoning on when a correct answer needs arithmetic, unit conversion, constraint tracking or multi-step deduction. Reasoning off for retrieval, summarization, classification, extraction and rewriting.
  2. Run the equals-sign test on your own traffic. Pull 50 real prompts and count how many need an equation or a chain of constraints to answer. That share is your reasoning budget.
  3. Before you standardize, A/B the two modes on 30 to 50 labeled items and log three numbers: accuracy, cost per call, p95 latency. Keep reasoning on only where the accuracy gap clears your error bars.
  4. Hand computation to a tool. A calculator, a SQL query or a Python call beats the model's own arithmetic steps 1.
  5. Trim irrelevant context before you buy more reasoning tokens. A long budget amplifies distractors that a short one survives 3.
  6. On pattern tasks with exceptions, give 5 to 10 labeled examples instead of asking for step-by-step analysis 2 3.
  7. Cap the step length. Terse intermediate notes matched or beat full chain-of-thought accuracy on reasoning benchmarks while using as little as 7.6% of the tokens 6.
  8. Never ship the trace as justification to a user or an auditor. Validate the answer against the source document, the database row or a test case.

The evidence stops short of a universal rule. Sprague's categories come from benchmark datasets, and your inbox of support tickets or contract clauses will not map onto them cleanly. Liu's drop-offs appeared on three tasks out of five. Both findings point the same way for the work I see in the field: the reasoning switch pays on calculation and deduction, and it buys latency on everything else.

Pull 50 prompts from last week, mark the ones carrying an equation or a constraint chain, and set your default from that count.

Sources.

  1. Sprague, Z., Yin, F., Rodriguez, J. D., Jiang, D., Wadhwa, M., Singhal, P., Zhao, X., Ye, X., Mahowald, K., Durrett, G. (2025). To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning. ICLR 2025. arXiv:2409.12183
  2. Liu, R., Geng, J., Wu, A. J., Sucholutsky, I., Lombrozo, T., Griffiths, T. L. (2025). Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse. Proceedings of the 42nd International Conference on Machine Learning (ICML). proceedings.mlr.press
  3. Gema, A. P., Hagele, A., Chen, R., Arditi, A., Goldman-Wetzler, J., Fraser-Taliente, K., Sleight, H., Petrini, L., Michael, J., Alex, B., Minervini, P., Chen, Y., Benton, J., Perez, E. (2025). Inverse Scaling in Test-Time Compute. arXiv:2507.14417
  4. Turpin, M., Michael, J., Perez, E., Bowman, S. R. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). papers.neurips.cc
  5. Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., Mikulik, V., Bowman, S. R., Leike, J., Kaplan, J., Perez, E. (2025). Reasoning Models Don't Always Say What They Think. arXiv:2505.05410
  6. Xu, S., Xie, W., Zhao, L., He, P. (2025). Chain of Draft: Thinking Faster by Writing Less. arXiv:2502.18600

Keep reading