Drip-fed instructions cost 39%: when to start a new chat
In the trainings I run, people treat the chat window like a colleague on Slack. Vague first ask, model guesses, user corrects, model fixes one thing and breaks another, fifteen turns later the output is worse than the first draft. Researchers have now run that exact pattern as a controlled experiment, and the cost of delivering a task in pieces is large enough to change how you open a chat.
Same information, worse answers.
Philippe Laban, Hiroaki Hayashi, Yingbo Zhou and Jennifer Neville ran the comparison that settles the question. They took tasks with complete written specifications, code, SQL, math problems and constrained summaries, then split each specification into shards and released one shard per turn. Both versions carry the same constraints by the end. Only the delivery changes. Across 15 models and more than 200,000 simulated conversations, performance on six generation tasks fell 39% on average in the sharded setting, and the drop appeared in every model they tested, open and closed weights alike 1.
The decomposition is the part I now teach first. Aptitude, measured as the best run of a conversation, fell about 16%. Unreliability, the spread between a model's best and worst run on the same task, rose 112% 1. A model that handles your task from a full brief turns erratic when you feed it the same task in pieces. Two colleagues can run the same drip-fed conversation and reach opposite conclusions about whether the tool works.
The mechanism, as the authors describe it, is premature commitment. Models make assumptions to fill the gaps in an underspecified first turn, write a full solution before the constraints arrive, then lean on that early attempt as the conversation continues 1.
Compliance falls by turn 3.
Yun He and 18 co-authors built Multi-IF to measure the same decay with checkable rules. The benchmark holds 4,501 conversations across 8 languages, three turns each, where every turn adds an instruction a script can verify, such as a word count or a forbidden character. Averaged over all languages, o1-preview went from 0.877 accuracy at turn 1 to 0.707 at turn 3, the strongest turn 3 score in the set 2.
One number in that paper changed a habit of mine. The authors measured an error correction rate: how often a model repairs an instruction it broke on the previous turn once the conversation continues. The o1 models reached around 25%. Most other models stayed below 10% 2. Pointing out the broken constraint and continuing leaves you with a 1 in 10 chance of getting it back.
Ved Sirdeshmukh and colleagues pushed on realistic conversations in Findings of ACL 2025. Their MultiChallenge benchmark uses conversation histories of up to 10 turns ending in a final user request, built around four failure modes including instruction retention, which tests whether a constraint from the first turn survives to the last. Every frontier model they tested scored below 50%, with Claude 3.5 Sonnet highest at 41.4% average accuracy 3.
The benchmark scores age, the gap does not.
Those MultiChallenge numbers come from models available in late 2024, and Scale's updated leaderboard now puts the best systems in the 60s. Take the absolute scores as a snapshot. The finding that survives model generations is the gap between one-shot and multi-turn delivery of identical information, which Laban's team measured across 15 models of varying size and vintage without finding an exception 1.
A second limit worth stating: the sharded conversations come from a simulated user that releases one piece of information per turn, more mechanically than a person would. A human who front-loads the constraints sits closer to the full-brief condition, which is the point of the protocol below.
Correcting in place feeds the error.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu and Noah Smith named the trap at ICML 2024. Once a model has stated a wrong answer, it generates supporting claims to justify it, a pattern they call hallucination snowballing. Presented with those same claims in isolation, GPT-3.5 identified 67% of them as false, GPT-4 87%, and LLaMA2-70B-chat 94% 4. The model can spot the error. It cannot spot it while the error sits above in its own transcript.
A wrong turn lives in the transcript. Starting a new chat removes it; arguing with it inside the same thread keeps it in the context the model reads on every later turn.
The protocol.
1. Write the whole brief before the first message. Task, inputs, constraints, output format, and the test you will apply to the answer. The sharded penalty comes from information arriving late, so anything you already know belongs in turn 1 1.
2. On an underspecified task, block the first solution. Open with "ask me up to 5 questions, then wait" or "list your assumptions, no draft yet". Models commit early and build on that commitment 1.
3. Set a turn budget of 5 and count. Compliance on verifiable instructions falls measurably by turn 3, and repair after a broken constraint lands under 10% of the time on most models 2. Past turn 5 on a constrained task, stop patching.
4. Use the consolidation move instead of another correction. Send: "List every requirement and constraint I have given you in this conversation as a numbered list. No solution." Read the list, fix what it got wrong, paste it into a fresh chat as a single brief. The authors give this as their own recommendation for a conversation that went badly, and their RECAP variant, which restates all shards in a final turn, recovered part of the lost performance 1.
5. Repeat the constraint in the turn where it has to hold. Instruction retention from turn 1 to turn 10 is the weakest category on MultiChallenge 3, so do not assume the format rule you set at the start still applies to the file you are asking for now.
6. Re-ask a suspicious claim in a clean chat, as a standalone question. Do not argue with it in place. Isolated from its own justification, the model catches most of these claims itself 4.
7. Measure the gap on your own task. Take a task your team runs weekly, run it 5 times as one full brief and 5 times as a drip-fed conversation, and count the outputs that pass your review. Compare the two counts before you decide how your team should open a chat.
Open the longest chat you have running right now. If it is past 5 turns and the output still misses a constraint, send the consolidation prompt, copy the corrected list into a new chat, and compare the two answers side by side.
Sources.
- Laban, P., Hayashi, H., Zhou, Y., Neville, J. (2025). LLMs Get Lost In Multi-Turn Conversation. ICLR 2026. arXiv:2505.06120
- He, Y., Jin, D., Wang, C., Bi, C., Mandyam, K., Zhang, H., Zhu, C., Li, N., Xu, T., Lv, H., Bhosale, S., Zhu, C., Sankararaman, K. A., Helenowski, E., Kambadur, M., Tayade, A., Ma, H., Fang, H., Wang, S. (2024). Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following. arXiv:2410.15553
- Sirdeshmukh, V., Deshpande, K., Mols, J., Jin, L., Hernández Cardona, E.-Y., Lee, D., Kritz, J., Primack, W. E., Yue, S., Xing, C. (2025). MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs. Findings of the Association for Computational Linguistics: ACL 2025. aclanthology.org/2025.findings-acl.958
- Zhang, M., Press, O., Merrill, W., Liu, A., Smith, N. A. (2024). How Language Model Hallucinations Can Snowball. ICML 2024. proceedings.mlr.press/v235/zhang24ay.html