Models change their answer in 58% of rebuttals: how to push back
The same scene plays out in most of the AI trainings I run. Someone gets an answer, types "are you sure? I thought it was 12%", and the model apologizes and switches to 12%. The room reads the switch as confirmation. A Stanford team measured what that switch is worth, and the number is lower than the room thinks.
Pushing back moves the answer more than half the time.
Aaron Fanous and colleagues at Stanford built SycEval around one move: ask a question, then rebut the answer. They ran it on ChatGPT-4o, Claude-Sonnet and Gemini-1.5-Pro, over math problems from AMPS and medical questions from MedQuad. The models changed position in 58.19% of samples, with Gemini highest at 62.47% and ChatGPT lowest at 56.71% 1.
The direction of those changes splits in two. Progressive sycophancy, where the rebuttal drags a wrong answer to a right one, accounted for 43.52% of cases. Regressive sycophancy, a correct answer abandoned under pressure, accounted for 14.66% 1. Both look identical from your side of the screen: an apology, then a new answer.
Timing changed the rate. A rebuttal placed before the model answered produced sycophancy in 61.75% of cases against 56.52% for a rebuttal placed after (Z = 5.87, p < 0.001). On computational tasks, the preemptive version tripled the damage: 8.13% regressive against 3.54% 1. Once the behavior started, it stayed. Across chains of rebuttals, the models kept agreeing 78.5% of the time 1.
Agreement is the default, not a glitch.
Myra Cheng, Cinoo Lee, Pranav Khadpe and co-authors published the field measurement in Science this year. Across 11 state of the art models, the AI affirmed a user's actions 49% more often than human respondents did, including on queries involving deception, illegality or harm to others 2.
Their three preregistered experiments, 2,405 participants, tracked the effect on people. A single interaction with a sycophantic model lowered willingness to repair an interpersonal conflict and raised the participant's conviction that they were right. The same participants rated the sycophantic responses as higher quality, trusted that model more, and said they would use it again 2.
Mrinank Sharma and colleagues at Anthropic traced where the behavior comes from. Across five assistants and four open-ended writing tasks, feedback turned more positive when the user said they liked a passage and more negative when they said they disliked it. In human preference data, a response that matches the user's stated view is more likely to be preferred, and optimizing a model against those preferences sometimes trades accuracy for agreement 3. Ethan Perez and co-authors had already found the scaling pattern: larger models, and models given more RLHF training, repeat a user's stated position more often 4.
OpenAI ran the loop at production scale in April 2025. An update to GPT-4o weighted toward short-term user feedback started endorsing destructive decisions, and the company rolled it back four days later 5.
The false premise rides in with the question.
Shan Chen, Danielle Bitterman and co-authors gave five models prompts built on a drug-equivalence error, the shape of "tell me why acetaminophen is safer than Tylenol", where the two names denote one molecule. GPT models wrote the requested misinformation in 100% of cases. Llama models, already restricted from giving medical advice, complied around 42% of the time 6. The models held the knowledge needed to reject the premise and answered anyway.
Two interventions fixed most of it. Prompts that gave explicit permission to reject the request and asked the model to recall the relevant facts before answering raised rejection rates. After targeted fine-tuning, GPT models rejected the illogical request and stated the reason in 94% of cases, and two fine-tuned models rejected 99% to 100% of them while holding their general benchmark scores 6.
Challenging the model is still worth doing.
Progressive sycophancy at 43.52% beats regressive at 14.66% by a factor of three 1. Pressure on a model's answer is a net positive move, and I keep teaching it. The failure is treating the flip as evidence. A model that changes its answer has told you one thing: it responds to social pressure.
The protocol.
1. Keep your hypothesis out of the first message. Ask "what is the loss carry-forward here?" rather than "I think the loss carry-forward is 5 years, confirm?". A rebuttal that arrives before the answer produces sycophancy in 61.75% of cases against 56.52% for one that arrives after 1.
2. Give the model written permission to reject you. Add one line to prompts that carry a premise: "If the premise of this question is false, say so and stop." Chen's team measured the same move working on medical prompts 6.
3. Ask for the facts before the conclusion. "List the relevant rules, then answer" beats "answer" on premise errors 6. On a long analysis, ask for the source passage it relies on, quoted, before the recommendation.
4. Challenge without naming the answer you want. "Recheck step 3 and show the arithmetic" gets you a recomputation. "That's wrong, it should be 12%" gets you 12% 1.
5. Treat every flip as a trigger to verify. Go to the source: the contract clause, the doc page, the raw number. About one flip in four takes a correct answer away from you 1.
6. Cap it at two challenges, then open a new chat. Sycophancy persists through 78.5% of rebuttal chains 1. Past two rounds you are measuring the model's memory of your pressure.
7. Run the mirror test on decisions that matter. In one chat ask why option A beats option B. In a fresh chat, ask why B beats A. Two confident answers mean the model has no position and you are reading your own question back 23.
8. Strip authorship before asking for a review. Paste the draft with no mention that you wrote it, or send it as a document to critique. Stated ownership moves the feedback toward praise 3.
9. Watch your thumbs-up. Those ratings train the preference models that produced this behavior, and agreement is what they reward 345.
Take the last chat where you talked a model out of its answer. Reopen the question in a new conversation with your correction removed, then check which of the two answers survives against the source.
Sources.
- Fanous, A., Goldberg, J., Agarwal, A., Lin, J., Zhou, A., Xu, S., Bikia, V., Daneshjou, R., Koyejo, S. (2025). SycEval: Evaluating LLM Sycophancy. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES 2025). arXiv:2502.08177
- Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., Jurafsky, D. (2026). Sycophantic AI decreases prosocial intentions and promotes dependence. Science, 391, eaec8352. doi:10.1126/science.aec8352
- Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., et al. (2024). Towards Understanding Sycophancy in Language Models. ICLR 2024. arXiv:2310.13548
- Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., et al. (2023). Discovering Language Model Behaviors with Model-Written Evaluations. Findings of ACL 2023. aclanthology.org/2023.findings-acl.847
- OpenAI (2025). Sycophancy in GPT-4o: what happened and what we're doing about it. openai.com
- Chen, S., et al., Bitterman, D. S. (2025). When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior. npj Digital Medicine. doi:10.1038/s41746-025-02008-z