All articles
5 min readai / productivity

92% to 68% at 3,000 tokens: what to cut from a long prompt

In my trainings I watch the same move a dozen times a day. Someone drops a 60 page report into the chat, types one question under it, and waits. The window swallows the whole file without complaint, so the paste feels free. Four controlled studies say it costs accuracy, and they agree on where the loss comes from: too much text, and the question in the wrong place.

What the research shows.

Nelson Liu and colleagues ran the cleanest test of position I know of. They gave models a set of documents for a question from NaturalQuestions-Open, kept the documents fixed, and moved only the one holding the answer. Accuracy traced a U across six model families, including GPT-3.5-Turbo, GPT-4, Claude 1.3, LongChat-13B, MPT-30B and Cohere Command: highest with the answer at the start or the end of the context, more than 30% lower with it in the middle 1.

One comparison in that paper reframed how I teach prompting. GPT-3.5-Turbo answered 56.1% of those questions with no documents at all, from memory. Give it 20 retrieved documents with the answer sitting mid-stack, and it scored under its own closed-book number 1.

The same paper kills a habit I see in most RAG pipelines: retrieve more, hope for the best. Liu's team pushed retrieval from 20 documents to 50, and retriever recall kept climbing while reader accuracy moved about 1.5% for GPT-3.5-Turbo and about 1% for Claude 1.3 1. The extra documents arrived in the context and went unread.

Mosh Levy, Alon Jacoby and Yoav Goldberg isolated length from position. Their FLenQA benchmark holds the reasoning task identical and pads it with text of varying length, type and placement, at 250, 500, 1,000, 2,000 and 3,000 tokens. Across GPT-4, GPT-3.5, Gemini Pro, Mistral Medium and Mixtral 8x7B, average accuracy fell from 0.92 on the 250 token version to 0.68 at 3,000 2. 3,000 tokens is about four pages, and the reasoning task itself never changed.

Freda Shi and co-authors measured the cost of a single wrong sentence. Their GSM-IC dataset takes grade school math problems and adds one irrelevant sentence. Chain-of-thought, zero-shot chain-of-thought, least-to-most and program prompting all lost accuracy against the clean problems, and fewer than 30% of the base problems stayed solved once a distractor was in the text 3. Adding an instruction to ignore irrelevant information recovered part of the loss under all four prompting styles 3.

The 128K label describes the input, not the working range.

Those models are from 2023 and 2024, and the fair objection is that the frontier has moved. It has, and the shape of the failure has not. Ali Modarressi and co-authors rebuilt the needle-in-a-haystack test so the question and the buried fact share almost no vocabulary, which stops a model from passing by string match and forces it to infer the link. Every model they benchmarked advertises at least 128K tokens of context. GPT-4o was among the strongest and still dropped from 99.3% at short length to 69.7% at 32K 4.

Two honest limits. Liu's fix of repeating the query before and after the context reached near-perfect scores on synthetic key-value retrieval and barely moved multi-document QA 1, so position work buys you lookup rather than reasoning. And these thresholds are task-specific: Liu tested 10, 20 and 30 documents and 75, 140 and 300 key-value pairs, and your own documents will break at their own numbers. Treat the figures below as a starting point you measure, which is what step 6 is for.

The protocol.

  1. Set your working budget at 3,000 tokens of context per question, roughly four pages, and treat the advertised window as a ceiling you never approach. FLenQA's decay is visible at that mark 2.
  2. Ask the question before the documents, then repeat it word for word after them. Both ends of the context get read 1.
  3. Rank what you paste. The document you expect to carry the answer goes at the top or the bottom of the stack, never in the middle third 1.
  4. Cap retrieval at 20 chunks. Past that you pay tokens and latency for about 1% of accuracy 1. Spend the effort on a better reranker.
  5. Delete the boilerplate before pasting: legal footers, navigation, changelogs, the four appendices nobody asked about. One stray sentence measurably costs accuracy 3, and a 60 page PDF carries hundreds.
  6. Run a position test on any prompt you plan to reuse. Same question, same documents, key document at top, then middle, then bottom. Different answers mean your context is too long for the task, and no rewording fixes that.
  7. Add one line to the system prompt: "Ignore any part of the context that does not bear on the question." It helped across all four prompting methods Shi's team tested 3.
  8. Split multi-hop questions into separate calls. Retrieve for step one, answer it, then pass that answer as a short input to step two. Two 800 token prompts beat one 4,000 token prompt on the same work.

Run step 6 first. Take the prompt your team fires most often, move the key document from top to middle to bottom, compare the three answers. If they disagree, start cutting at step 5.

Sources.

  1. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157-173. doi:10.1162/tacl_a_00638
  2. Levy, M., Jacoby, A., & Goldberg, Y. (2024). Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). doi:10.18653/v1/2024.acl-long.818
  3. Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Schärli, N., & Zhou, D. (2023). Large Language Models Can Be Easily Distracted by Irrelevant Context. Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202, 31210-31227. proceedings.mlr.press
  4. Modarressi, A., Deilamsalehy, H., Dernoncourt, F., Bui, T., Rossi, R. A., Yoon, S., & Schütze, H. (2025). NoLiMa: Long-Context Evaluation Beyond Literal Matching. Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR 267. proceedings.mlr.press

Keep reading