All articles
5 min readlearning / ai

Practice scores rose 48%, exam scores fell 17%: how to train with AI

In the workshops I run, the loudest hour is the one where everyone solves problems with a model open. People move fast, the chat logs look impressive, the room sounds like it is learning. Hamsa Bastani's team measured what came after that hour. Students who had practiced with GPT-4 scored 17% below classmates who never touched it, on an exam they sat without the tool.

A 48% gain during practice, a 17% loss on the exam.

Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı and Rei Mariman ran a field experiment with nearly 1,000 high school mathematics students at a school in Turkey, covering about 15% of the curriculum in each of three grades 1. The control arm practiced as usual. A second arm practiced with GPT Base, a chat interface close to ChatGPT. A third practiced with GPT Tutor, the same model wrapped in teacher-written hints and blocked from giving final answers.

During the practice sessions both AI arms pulled ahead of the control arm: 48% better for GPT Base, 127% better for GPT Tutor.

Then the researchers removed access and ran the exam. GPT Base students came in 17% below the control arm. GPT Tutor students landed level with the control arm, with no gain to show for the 127%.

The chat logs explain the collapse. Students treated GPT Base as an answer key. The authors sent it the most common student message, "What is the answer?", ten times for each of 57 practice problems. It returned a correct answer 51% of the time, with logical errors in 42% of the responses and arithmetic errors in 8%. Students were copying a coin flip and banking the result as practice.

The survey data is worse. GPT Base students who scored below the control arm did not report performing worse or learning less. GPT Tutor students, level with control on the exam, reported that they had done better.

The Harvard trial names the variable.

Gregory Kestin, Kelly Miller, Anna Klales, Timothy Milbourne and Gregorio Ponti ran a crossover randomized trial with 194 students in Harvard's introductory physics course for life sciences in autumn 2023 2. Each student learned one topic, surface tension or fluid flow, in a 60-minute active-learning class, and the other topic at home with a tutor the team built, PS2 Pal. Median learning gains more than doubled with the tutor, reached in a median of 49 minutes. Students rated their engagement 4.1 against 3.6 for the class.

Kestin built the tutor on the same lesson plan and loaded it with the feedback notes he would have given in person. It moved one step at a time and held back answers. That design sits on the GPT Tutor side of the Turkish experiment, which is the reason the two trials point in opposite directions. One limit worth keeping: each post-test followed its lesson immediately, so the trial says nothing about what survived a month later.

Take the assistant away and people fall back on weaker habits.

Ali Darvishi, Hassan Khosravi, Shazia Sadiq, Dragan Gašević and George Siemens followed 1,625 students across 10 courses writing peer reviews 3. For four weeks every student got AI prompts that flagged shallow feedback. The researchers then split them four ways: keep the prompts, drop them, replace them with a self-monitoring checklist, or run both. Students who lost the prompts wrote weaker reviews. The checklist recovered part of the gap and stayed behind the AI condition. Running both together beat neither. The authors summarize it as students relying on the assistance rather than learning from it.

The meta-analysis measures the easy half.

Across 37 experiments published between 2022 and 2025, Zhiwei Liu, Haode Zuo and Yongjing Lu put the pooled effect of ChatGPT on academic achievement at Hedges' g = 0.577, 95% confidence interval [0.395, 0.759] 4. That is a moderate win and a fair summary of the literature as it stands. Most of the included studies score performance with the tool in hand, or minutes after. The Turkish trial is the one that measures both moments in the same students, and the sign flips between them.

The operational version for anyone who trains people: a score earned with the model open describes the pair, and a score earned alone describes the person. Your client is paying for the second one.

The protocol.

  1. Write the closed-laptop test first. Before you design a session, write the 20-minute exercise people will do at the end with no model open. Score it. That number is your learning metric, and no practice score replaces it.
  2. Make people attempt before the model speaks. Ten minutes alone on the problem, pen and paper, then the tool. Pretesting raises retention even when the attempt fails.
  3. Ship a hints-only assistant for the practice block. One system prompt: give a single hint, ask a question back, never state the final answer or the full solution. That restriction is the whole distance between a 17% loss and parity in Turkey.
  4. Write the hints yourself. The Turkish guardrail used teacher-written hints; Kestin loaded his own feedback notes. Both teams put a human's worked understanding inside the tool before letting students near it.
  5. Measure your model on your own problem set. Take your 20 hardest exercises, ask the model each one 10 times, count correct answers. Bastani's team got 51% on high school mathematics. If yours lands near that, an answer-giving assistant is teaching errors at scale.
  6. Stop asking the satisfaction form about learning. Both AI arms in Turkey misjudged their own results, in both directions. Keep the form for logistics and let the closed-laptop test answer the learning question.
  7. Re-run the test 4 weeks later, cold. Harvard tested within the hour, so its doubled gains carry no retention claim. Email the same exercise a month out, no warning, no tools, and compare the two scores before you buy the platform.

Sources.

  1. Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26). doi.org/10.1073/pnas.2422633122
  2. Kestin, G., Miller, K., Klales, A., Milbourne, T., & Ponti, G. (2025). AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting. Scientific Reports, 15, 17458. doi.org/10.1038/s41598-025-97652-6
  3. Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024). Impact of AI assistance on student agency. Computers & Education, 210, 104967. doi.org/10.1016/j.compedu.2023.104967
  4. Liu, Z., Zuo, H., & Lu, Y. (2025). The impact of ChatGPT on students' academic achievement: a meta-analysis. Journal of Computer Assisted Learning, 41(4), e70096. doi.org/10.1111/jcal.70096

Keep reading