All articles
5 min readai / productivity

40% of Copilot's code was vulnerable: a review protocol

In the AI trainings I run, the engineering tables move faster than any other room, and they read the generated code least. A function appears, the tests are green, the branch goes up for review. Security researchers have been measuring what sits inside those functions since 2021, and the failure rates land between 24% and 45% depending on the language and the task.

What the research shows.

Hammond Pearce's team at NYU wrote 89 scenarios around MITRE's Top 25 weaknesses and had GitHub Copilot complete each one. Of the 1,689 programs produced, around 40% contained a vulnerability 1. The scenarios were built to bait weaknesses, so read that as the ceiling on a hostile benchmark rather than a rate for daily work.

Yujia Fu and colleagues went to the code people had already merged. They collected 733 Copilot-generated snippets living in public GitHub projects and scanned them: 29.5% of the Python snippets and 24.2% of the JavaScript ones carried a security weakness, spread over 43 CWE types, 8 of them in the CWE Top 25 2. Same order of magnitude as the lab benchmark, in production repositories.

Veracode ran the widest sweep in 2025: 80 curated coding tasks across more than 100 models, in Java, JavaScript, Python and C#. 45% of the completions failed the security test attached to their task. Java failed 72% of the time. Cross-site scripting failed 86%, log injection 88%. Bigger and newer models did not score better than smaller, older ones 3. That one is a vendor report rather than peer-reviewed work, and it is worth reading with that in mind, though its method is documented and its direction matches the academic results.

The number that changed how I teach this comes from Stanford. Neil Perry, Megha Srivastava, Deepak Kumar and Dan Boneh gave 47 participants five security-relevant tasks across three languages. Half had an AI assistant. The assisted group wrote less secure solutions on four of the five tasks, and rated their own code as more secure than the unassisted group rated theirs 4. Confidence moved one way while the code moved the other.

Iteration does not rescue it. Shivani Shukla, Himanshu Joshi and Romilla Syed ran 400 code samples through 40 rounds of AI-driven improvement under four prompting strategies. Critical vulnerabilities rose 37.6% after five iterations, from 2.1 per sample early on to 6.2 by rounds 8 to 10, including in the runs where the prompt asked for more secure code 5. Asking a model to harden its own output, again and again, adds weaknesses.

The C study points the other way.

Gustavo Sandoval's team ran a similar experiment with 58 students implementing a singly linked shopping list in C. The assisted group produced severe CWEs at a rate no more than 10% above the control group, and on several measures wrote fewer bugs 6. One study on one low-level task, and it lands opposite to Perry's result.

Task type explains most of the distance between them. Memory-safety plumbing in C is dense, repetitive and well represented in training data. The tasks where the assisted groups slipped involve a library choice: which cipher mode, which escaping function, which deserialization call, which random source. A wrong pick there compiles, passes the happy-path test and reviews clean.

Practitioners sense the boundary. In 27 interviews with software professionals, Jan Klemmer's team found people using assistants for security-critical work including code generation, threat modeling and vulnerability detection, while rating those same assistants below static analysis tools at finding vulnerabilities 7. They keep the tool and distrust it, without changing the review that follows.

The protocol.

  1. Gate AI-written diffs behind a static analysis run, in CI, before human review. Fu's team resolved 55.5% of the weaknesses they found by feeding the static analysis warning back to Copilot Chat 2. The scanner names the bug; the model fixes what you can name.
  2. Name the CWE in your fix prompt. Pearce's repair study got 100% of synthetic and hand-crafted vulnerability scenarios repaired when the prompt identified the bug, while real-world historical bugs stayed hard 8. "Make this secure" is weaker than "this is CWE-89, parameterize the query".
  3. Stop at two refinement rounds, then re-scan 5. If round three feels necessary, write the code yourself.
  4. Review AI-written code by category rather than line by line: crypto calls, query construction, template rendering, path handling, deserialization, sources of randomness. Those five Stanford tasks and Veracode's XSS and log-injection rates mark the same short list.
  5. Add one test that sends the attack, not one that confirms the feature. A quote in a name field, a ../ in a path, a script tag in a comment box.
  6. Send any AI-written diff touching authentication, file upload, templating or SQL to a second reader. The Stanford confidence gap is measured on the author, so the author is the wrong person to clear it.
  7. Track which language you are in. Java and web-facing code start guilty 3; parsing and data plumbing carry the ordinary risk of any code review.

None of this slows a team much. The scanner runs in CI, the second reader is already reviewing, and the extra prompt costs one line. Start with one repository this week: turn the gate on for AI-written diffs and count how many findings the model resolves once you paste the warning back. Fu's 55.5% is the number to beat.

Sources.

  1. Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., Karri, R. (2022). Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions. 43rd IEEE Symposium on Security and Privacy. ieeexplore.ieee.org/document/9833571
  2. Fu, Y., Liang, P., Tahir, A., Li, Z., Shahin, M., Yu, J., Chen, J. (2025). Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study. ACM Transactions on Software Engineering and Methodology. doi:10.1145/3716848
  3. Veracode (2025). 2025 GenAI Code Security Report. veracode.com
  4. Perry, N., Srivastava, M., Kumar, D., Boneh, D. (2023). Do Users Write More Insecure Code with AI Assistants? ACM CCS 2023. doi:10.1145/3576915.3623157
  5. Shukla, S., Joshi, H., Syed, R. (2025). Security Degradation in Iterative AI Code Generation: A Systematic Analysis of the Paradox. IEEE ISTAS 2025. ieeexplore.ieee.org/document/11269659
  6. Sandoval, G., Pearce, H., Nys, T., Karri, R., Garg, S., Dolan-Gavitt, B. (2023). Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants. USENIX Security 2023. usenix.org
  7. Klemmer, J. H., Horstmann, S. A., Patnaik, N., Ludden, C., Burton, C., Powers, C., Massacci, F., Rahman, A., Votipka, D., Lipford, H. R., Rashid, A., Naiakshina, A., Fahl, S. (2024). Using AI Assistants in Software Development: A Qualitative Study on Security Practices and Concerns. ACM CCS 2024. doi:10.1145/3658644.3690283
  8. Pearce, H., Tan, B., Ahmad, B., Karri, R., Dolan-Gavitt, B. (2023). Examining Zero-Shot Vulnerability Repair with Large Language Models. 44th IEEE Symposium on Security and Privacy, 2339-2356. nyuscholars.nyu.edu

Keep reading