All articles
4 min readai / productivity

AI makes you faster, and sometimes wrong: five studies draw the line

I have trained more than 300 people to work with AI, from engineers to analysts to sales teams. The question I get most often is not "how do I write a prompt". It is "can I trust this thing". The honest answer: it depends on where you stand, and five controlled experiments now show where the line sits.

The studies first, then the rule I teach.

The headline numbers are real.

In 2023, researchers at Microsoft and MIT ran a randomized experiment with 95 professional developers. Each had to build an HTTP server in JavaScript. The group with GitHub Copilot finished 55.8% faster than the control group 1. A controlled task with a stopwatch rather than a survey.

Same year, in Science: 453 professionals did realistic writing tasks (press releases, reports, emails). With ChatGPT, average time dropped by 40% and output quality, graded blind by other professionals, rose by 18% 2. One detail stood out: the weakest writers gained the most. AI compressed the gap between the bottom and the top of the group.

A third study followed 5,179 customer support agents at a Fortune 500 software company. With an AI assistant, agents resolved 14% more issues per hour on average. For novice agents the gain was 34%, while experienced agents barely moved 3. The AI had bottled the experts' playbook and handed it to the juniors.

Pattern so far: AI is a skill equalizer. The less expertise you have on a task, the more it gives you.

The two studies that cut the other way.

I open my trainings with the fourth study. Researchers gave 758 BCG consultants a set of tasks. On tasks inside the frontier of what GPT-4 did well, consultants using AI completed 12.2% more tasks, 25.1% faster, at 40% higher quality. On a task chosen to sit outside that frontier, consultants using AI were 19 percentage points more likely to get the answer wrong than consultants without it 4.

The researchers called this the "jagged frontier". AI capability has the shape of a coastline: brilliant at one task, terrible at a neighboring task that looks almost identical. The consultants could not feel the difference. The AI's answer read as confident on both sides of the line.

In 2025, METR ran a randomized trial with something the earlier studies lacked: expert participants working on their own turf. Sixteen experienced open-source developers, working on repositories they had known for years, were randomly assigned to solve issues with or without AI tools. With AI they were 19% slower. They believed they had been about 20% faster 5.

Experts on familiar ground came out slower, and convinced of the opposite.

The rule I teach.

Put the five studies together and you get a rule you can apply tomorrow:

1. Ask "am I a novice here?" before each delegation. On tasks where you are junior, the studies back heavy use: gains of 30 to 55% are on the table 123. On tasks where you are the expert, the burden of proof flips. Default to doing it yourself and using AI only for the parts you do not know.

2. Map your own frontier with a log. The METR result 5 shows self-perception fails where it matters most. For 2 weeks, note each task you delegate to AI: task type, time spent, and whether the output survived your review without rework. After 20 entries you have your personal jagged frontier. Mine, for example: AI drafts my training materials (huge win) but loses me time on debugging code I wrote myself.

3. Verify in proportion to your expertise, inversely. On expert ground you can check AI output cheaply, so using AI is safe but often pointless. On novice ground AI helps most and you cannot catch its errors. For novice-territory tasks, build an external check: run the code, ask for sources and open two of them, or show the output to someone senior. Do not let "it reads well" be the test. The BCG study's wrong answers read well 4.

4. Keep the expensive judgment for yourself. In the BCG study, the consultants who did best treated the AI's output as raw material to interrogate rather than as an answer 4. The failure mode was pasting the answer through. The gains in the literature go to people who stayed in the loop.

The five studies converge on one claim: the return on AI depends on the distance between a task and your own expertise, and your intuition about that distance is unreliable. Measure it.

Sources.

  1. Peng, S., Kalliamvakou, E., Cihon, P., Demirer, M. (2023). The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv:2302.06590
  2. Noy, S., Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Science, 381(6654). doi:10.1126/science.adh2586
  3. Brynjolfsson, E., Li, D., Raymond, L. (2023). Generative AI at Work. NBER Working Paper 31161. nber.org/papers/w31161
  4. Dell'Acqua, F., et al. (2023). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. Harvard Business School Working Paper 24-013. hbs.edu
  5. METR (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. metr.org

Keep reading