All articles
5 min readai / productivity

15 points of swing at temperature 0: when to run a prompt twice

In my trainings I ask people to send the same prompt three times and line up the answers. Most expect three copies of one response. They get two or three different ones, and the room goes quiet. A single response is one draw from a distribution, and the number on your screen says nothing about the spread around it.

A deterministic setting is not a repeatable one.

Berk Atıl and 12 co-authors ran 5 hosted LLMs with deterministic settings, temperature at 0 and a fixed seed, across 8 tasks, 10 runs per condition, in both zero-shot and few-shot form. Accuracy moved by up to 15 points between runs of the same configuration. The spread between the best and the worst run reached 70 points. None of the 5 models delivered repeatable accuracy across all tasks, let alone identical output text 1. The authors suspect co-mingled requests inside hosted inference buffers, without confirming the mechanism.

That variance is the raw material for a method.

Majority voting bought 17.9 points on a 2022 model.

Xuezhi Wang and colleagues at Google sampled several chain-of-thought paths for one question, then kept the final answer that appeared most often. They named it self-consistency, and their main configuration samples 40 paths. On GSM8K, PaLM-540B went from 56.5% with greedy decoding to 74.4%, a gain of 17.9 points. SVAMP gained 11.0 points, AQuA 12.2, StrategyQA 6.4, ARC-challenge 3.9 2.

The method needs an answer it can count. Arithmetic results, multiple-choice letters, classification labels and short extracted facts all tally cleanly. Prose does not.

The margin shrank as the models improved.

Chiyan Loo rechecked the idea on Gemini 2.5 with HotpotQA and MATH-500, sampling up to 20 reasoning paths. The gain across 20 samples came to 0.4% on HotpotQA and 1.6% on MATH-500. Accuracy flattened early and fell at high sample counts in some configurations, which he attributes to extra paths adding noise on problems the model already handles. Token cost grew close to linearly with the sample count 3.

One author, two benchmarks, one model family, and a preprint rather than a peer-reviewed paper. The benchmarks also differ from the 2022 set, so 17.9 and 1.6 are not two measurements of the same quantity. Read it as a direction: a model that reasons internally has already spent some of the compute that self-consistency used to add from outside. Loo's own recommendation is to reserve multi-path sampling for problems that exceed what the model handles in one pass.

Resampling pays where a test can run.

Bradley Brown and co-authors pushed sampling far past 20. On SWE-bench Lite, DeepSeek-Coder-V2-Instruct resolved 15.9% of issues with one sample and 56% with 250, above the 43% that stronger frontier models reached on a single attempt. On GSM8K and MATH, Llama-3 coverage passed 95% at 10,000 samples 4.

Coverage counts the problems where at least one sample is correct, which presumes you can find that sample. SWE-bench Lite comes with a way: the repository test suite. Without one, the selection step caps the gain, and Brown and colleagues report that majority voting and reward models plateau beyond several hundred samples while coverage keeps climbing.

Benedikt Stroebl, Sayash Kapoor and Arvind Narayanan put a ceiling on the trick. A verifier that passes wrong answers, such as the thin unit tests of HumanEval and MBPP, accumulates false positives as you draw more samples. They find a strong correlation between a model's single-sample accuracy and its false-positive rate, so the weaker model suffers more. Once you price the cost of shipping a false positive, the optimal number of attempts usually falls below 10, and 5 or fewer in one of their figures. No sample budget lets a weak model reach a strong model's single-sample accuracy 5.

The protocol.

  1. Send any prompt whose answer is a number, a label or a short extractable fact 3 times, in 3 separate chats. Compare the final answers and ignore the wording.
  2. Take the majority of the 3. On a 2 to 1 split, read the minority path before accepting the majority. It often names an ambiguity you left in the prompt.
  3. Stop once 3 samples agree. On a three-way split the model is guessing: rewrite the prompt or cut the task in half instead of drawing a fourth sample.
  4. Cap at 5 samples while nothing automatic can check the output. Past that you pay tokens for confidence you cannot verify.
  5. With a real checker in the loop, unit tests, a schema validator, a SQL query that has to execute, go to 10 or 20 samples and keep what passes. Spend budget on the checker before the sample count.
  6. Skip all of it on a reasoning model for tasks it already gets right in one pass. Reserve multi-path sampling for prompts you have watched it fail.
  7. For free-form output, a summary or an email or a spec, there is nothing to tally. Paste the 3 candidates into a fresh chat and ask the model which one agrees most with the other two. Xinyun Chen and colleagues tested that substitution and matched standard self-consistency on math, where both approaches can be scored 6.
  8. In any evaluation you report, state the run count. A single-run score separated by 5 points from another single-run score is noise.

I run step 1 on every figure I put in a client deck. Two models out of three agreeing on 56.5% is worth more than one model answering once.

Sources.

  1. Atıl, B. et al. (2025). Non-Determinism of "Deterministic" LLM System Settings in Hosted Environments. Proceedings of Eval4NLP 2025. aclanthology.org
  2. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023. arXiv:2203.11171
  3. Loo, C. (2025). Self-Consistency Is Losing Its Edge: Diminishing Returns and Rising Costs in Modern LLMs. Preprint. arXiv:2511.00751
  4. Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., & Mirhoseini, A. (2024). Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. Preprint. arXiv:2407.21787
  5. Stroebl, B., Kapoor, S., & Narayanan, A. (2024). The Limits of Inference Scaling Through Resampling. Preprint. arXiv:2411.17501
  6. Chen, X., Aksitov, R., Alon, U., Ren, J., Xiao, K., Yin, P., Prakash, S., Sutton, C., Wang, X., & Zhou, D. (2023). Universal Self-Consistency for Large Language Model Generation. Preprint. arXiv:2311.17311

Keep reading