AI Models Are Getting Better at Coding, But Who Is Checking Their Work?

Something remarkable happened quietly in the spring of 2026.

For the first time in the history of software engineering, AI systems began resolving more than half of the real, world software bugs submitted to them on a standard industry benchmark, without any human assistance, without being told the solution, and without seeing the code before the task began. They read the bug report. They found the relevant code. They wrote the fix. They ran the tests. They iterated until the tests passed. And they did it in minutes.

To understand why this is significant, consider what benchmark tests. SWE, bench, the Software Engineering benchmark developed at Princeton, does not ask AI systems to write Hello World programmes or solve textbook coding puzzles. It takes real GitHub issues from real open, source projects, submitted by real developers who had genuine problems they needed fixed. The issues involve understanding large, unfamiliar codebases, reasoning about the intended behavior of complex systems, and making changes that are correct, minimal, and do not break anything else.

Six months earlier, the best systems were resolving around 30% of these issues. A year before that, 12%. The improvement is steep, consistent, and accelerating.

The software engineering community is paying attention. And it is asking a question that the benchmark scores do not answer - if AI systems are writing more of the world's code, who is responsible for making sure that code is safe, correct, and does what it is supposed to do?

This article takes that question seriously, examining what AI coding systems can now do, where they fall short, what the risks of getting this wrong look like in practice, and what responsible use of AI in software development requires.

What AI Coding Systems Can Do in 2026

The capabilities of AI coding assistants have advanced so rapidly that many developers' mental models of what these tools can do are already outdated. The chatbot that could autocomplete a function in 2023 has been replaced by something categorically more powerful.

Writing Complete Features from Natural Language

Modern AI coding systems, Claude Code, GitHub Copilot Workspace, Cursor, and others, can take a natural language description of a feature and produce a working implementation across multiple files, with appropriate error handling, tests, and documentation. Not a rough sketch. A working feature.

"Add user authentication with OAuth support, including token refresh logic and appropriate error handling for expired sessions", this kind of instruction, which would previously require several hours of focused developer work, can now produce a functional, reasonably complete implementation in minutes.

The quality varies. It depends on the specificity of the instruction, the complexity of the existing codebase, and the nature of the feature being implemented. For well understood patterns, authentication, CRUD operations, API integrations, AI implementations are often good enough to be used directly, with light review. For novel architectures or complex business logic, they frequently require significant revision.

But the direction is clear. The systems are moving from "helps you write code faster" to "write code that you review", a fundamental shift in the role of the human in the development process.

Debugging and Root Cause Analysis

Finding the cause of a bug in a complex system is often harder than writing the original code. It requires understanding the intended behavior of the system, tracing the actual behavior, identifying where they diverge, and determining why. This is exactly the kind of multi, step reasoning that AI systems have become significantly better at.

Current AI coding systems can read stack traces, understand error messages, navigate large codebases to find the relevant code paths, form hypotheses about the root cause, and propose fixes, often correctly, on the first attempt, for the class of bugs that have clear symptoms and traceable causes.

For the harder class of bugs, heisenbugs that appear only under specific race conditions, subtle logic errors in complex business rules, performance issues that only manifest at scale, AI systems are considerably less reliable. But for the routine debugging work that consumes a significant proportion of development time, the assistance is genuinely valuable.

Code Review and Security Analysis

AI systems are increasingly being used not just to write code but to review it, scanning security vulnerabilities, identifying code quality issues, suggesting improvements, and flagging potential bugs before code reaches production.

GitHub's Code Scanning, Snyk's AI, powered security analysis, and similar tools now integrate AI, based vulnerability detection into development workflows, identifying common security issues, SQL injection, cross, site scripting, insecure dependencies, hardcoded credentials, with a reliability that often exceeds human code reviewers scanning the same code under time pressure.

This is genuinely valuable. Security vulnerabilities in code are extraordinarily costly, the average cost of a data breach in 2024 was $4.88 million (IBM, 2024), and AI, assisted security review catches issues that human reviewers miss. But it also introduces a new risk - developers who treat AI security review as a comprehensive guarantee of security, when it is not.

Refactoring and Legacy Code Modernization

Legacy code, old systems written in outdated languages or with outdated patterns, often poorly documented and not fully understood by any current team member, is one of the most painful and highest, risk categories of software work. Refactoring legacy code requires understanding what the code does, preserving its behavior while improving its structure, and doing so without breaking anything.

AI systems are increasingly capable of assisting with legacy code modernization, reading and explaining old code, suggesting modernized equivalents, translating between languages (COBOL to Java, for example, or Python 2 to Python 3), and generating tests that capture existing behavior before refactoring begins.

This is an area where the quality and reliability of AI assistance varies enormously with the complexity and obscurity of the legacy codebase. But the potential value, given the vast quantities of critical legacy code that organizations are struggling to maintain and modernize, is substantial.

The Performance Numbers - How Good Is "Good Enough"?

The SWE, bench results that opened this article represent the headline performance metric for AI coding systems. But what do these numbers mean for real, world software development?

The top AI coding agents in July 2026 are resolving approximately 50 to 55% of SWE, bench tasks autonomously. This means they are failing on approximately 45 to 50% of tasks. On a benchmark of real, world software engineering issues, nearly half the problems the AI attempts to solve are either not solved or solved incorrectly (Jimenez et al., 2024).

This is not a reason to dismiss the technology. A system that correctly resolves half of the routine bug reports in a development workflow represents genuine value, the 50% it handles frees developer time for the 50% it cannot. But it is a clear statement that these systems are not reliable enough to be trusted without human review. The failure rate is too high, the errors too unpredictable, and the consequences of incorrect code in production too serious.

The more important question than the average performance is the error distribution - what kinds of mistakes do AI coding systems make, how detectable are those mistakes, and what are the consequences when they go undetected?

How AI Coding Systems Fail - A Taxonomy of Errors

Understanding the failure modes of AI coding systems is essential for using them safely. They do not fail randomly, they fail in characteristic patterns that informed users can learn to anticipate.

Plausible But Wrong - The Confident Incorrect Solution

The most dangerous failure mode of AI coding systems is producing code that looks correct, passes superficial review, compiles without errors, even passes some tests, but contains a subtle logical error that causes incorrect behavior under specific conditions.

Unlike a syntax error, which fails immediately and loudly, a logical error in code can remain undetected through code review and testing, only revealing itself when the right combination of inputs in production triggers the incorrect behavior.

AI systems are particularly prone to this failure mode because they are optimized to produce plausible, syntactically correct, well, structured code, code that looks right. The same properties that make their outputs useful (fluency, structural correctness, adherence to conventions) can make their logical errors harder to detect than the errors a less fluent human programmer might make.

Off, by, One and Boundary Condition Errors

A disproportionate number of AI coding errors involve boundary conditions, the behavior of code at the edges of its expected input range. Arrays are indexed from zero rather than one. Loops that run one iteration too many or too few. Conditions that handle the null case incorrectly. Date and time edge cases.

These errors are particularly common in AI, generated code, because boundary conditions are often not explicitly stated in the requirements and require the developer to reason carefully about edge cases. AI systems, which pattern, match common implementations, frequently produce code that handles common cases correctly but fails on the boundary cases that were not explicitly specified.

Security Vulnerabilities Introduced Silently

AI coding systems have been documented to introduce security vulnerabilities into the code they generate, not through any malicious intent, but because they pattern, match code patterns that contain vulnerabilities, or because they do not reason about the security implications of their implementations (Pearce et al., 2022).

Research by Pearce et al. found that approximately 40% of the code generated by GitHub Copilot (an earlier generation of AI coding assistant) contained security vulnerabilities when assessed against the MITRE CWE (Common Weakness Enumeration) framework. Subsequent generations of AI coding systems have improved in this dimension, but the security risk of AI generated code that has not been carefully reviewed has not been eliminated.

The vulnerabilities introduced are not always obvious ones. SQL injection vulnerabilities in code that appears to parameterize queries correctly. Authentication bypasses in code that appears to implement authentication properly. Cryptographic implementation errors that produce code that appears to encrypt data but do so insecurely.

Hallucinated APIs and Non, Existent Functions

A failure mode specific to AI coding systems is the generation of code that calls APIs, functions, or library methods that do not exist, hallucinated interfaces that look plausible but are fabricated.

A developer reviewing AI, generated code that calls pandas.DataFrame.smart_merge() might assume this is a real pandas function they are not familiar with, rather than recognizing it as a hallucination. The code will fail when executed, but the failure might be attributed to a version mismatch or a missing dependency rather than recognized as AI confabulation.

This failure mode has become less common as AI coding systems are increasingly given access to real, time documentation and code execution environments that allow them to verify the existence of APIs before using them. But it has not been eliminated, particularly for less common libraries or for APIs that have changed between the model's training cutoff and the current date.

Context Blindness - Correct Code in the Wrong Place

AI coding systems sometimes produce code that is correct in isolation, syntactically valid, logically sound for the specified task, but wrong in the context of the specific system it is being inserted into.

The code might conflict with existing code elsewhere in the system. It might violate architectural constraints or team conventions that are not visible in the immediate context provided to AI. It might use patterns that are appropriate for some contexts but inappropriate for the specific security or performance requirements of the system being modified. It might duplicate functionality that already exists elsewhere in the codebase.

This context blindness is a function of the AI system's limited visibility into the full system. Even with large context windows, an AI system reviewing a single file or a small set of files cannot fully understand the architectural decisions, conventions, and constraints of a large, complex codebase. The code it generates may be appropriate for the context it can see and wrong for the broader context it cannot.

The Human Review Problem

Given these failure modes, the obvious mitigation is human review, having experienced developers carefully examine AI, generated code before it is merged into production systems.

This is the right answer. It is also an answer that is becoming increasingly difficult to implement as AI coding systems become more productive.

The Volume Problem

The productivity gains from AI coding assistance are real and significant. Developers using AI coding tools consistently report producing more code in less time, studies have reported productivity improvements of 20 to 55% on coding tasks when using AI assistance (Peng et al., 2023).

But this productivity gain creates a new problem - the volume of code that needs to be reviewed is growing faster than the human capacity to review it. If a developer using an AI assistant produces twice as much code per day, the code review burden for the team potentially doubles, unless review is also automated, which introduces its own risks.

The danger is that AI coding assistance reduces the time cost of producing code without reducing the time cost of reviewing it, creating a growing backlog of insufficiently reviewed code accumulating in codebases.

Expertise Problem

Effective code review of AI, generated code requires understanding what the code is supposed to do, understanding what it actually does, and identifying the gaps between those two things. This requires genuine technical expertise.

But AI coding tools are increasingly being used by developers with less experience, as a way of extending their capabilities into areas they do not fully understand. A junior developer using an AI coding assistant to implement authentication logic they do not deeply understand will struggle to review that code effectively, they may not recognize the security vulnerabilities the AI has introduced precisely because they lack the expertise to identify them.

This creates a paradox - AI coding tools are most useful for developers with less expertise, but effective review of AI, generated code requires more expertise, not less. The people most likely to rely heavily on AI, generated code without modification, are also the people least equipped to catch its errors.

The Complacency Problem

There is substantial evidence from human factors research that people reviewing outputs from automated systems show a characteristic bias toward accepting those outputs, a phenomenon called automation bias (Parasuraman and Manzey, 2010). When a system appears authoritative and produces outputs that look correct, reviewers tend to trust those outputs and reduce the rigour of their scrutiny.

In code review, automation bias manifests as a tendency to accept AI, generated code that looks syntactically clean and well, structured, without the depth of logical analysis that would be applied to code written by a human colleague whose work the reviewer is explicitly checking.

This is not a character flaw. It is a well, documented human cognitive response to authority cues, and AI coding systems produce very strong authority cues - clean formatting, appropriate naming conventions, clear structure, inline comments. The code looks professional because it is trained to look professional. Its logical correctness is a separate question.

Who Is Actually Checking? The State of AI Code Oversight in 2026

Against this backdrop, what does the actual practice of AI code review look like in organizations deploying AI coding assistance at scale?

The honest answer, based on available evidence, is inconsistent, often inadequate, and rarely well designed.

A survey of software development organizations using AI coding tools found that fewer than 30% had established formal policies specifically governing the review of AI, generated code (GitLab, 2024). Most organizations were applying the same code review processes to AI, generated code that they applied to human, written code, processes designed for a different volume and a different failure mode distribution.

Organizations in regulated industries, financial services, healthcare, defense, showed significantly higher rates of formal AI code review policies, driven by regulatory requirements that force attention to the provenance and quality of software in critical systems. But even in these environments, the specific vulnerabilities introduced by AI, generated code, were not consistently addressed.

The gap between the deployment of AI coding tools and the governance of those tools is significant and growing.

What Responsible AI, Assisted Development Looks Like

There is a clearer picture of what responsible AI, assisted software development, looks like. The organizations getting this right share a set of practices worth examining.

Treat AI, Generated Code as Untrusted Until Reviewed

The foundational principle is simple - code generated by an AI system should be treated as untrusted until it has received the same level of scrutiny that code from an unknown external source would receive. This is a higher bar than the review applied to code from a trusted colleague, because a trusted colleague has domain expertise, contextual understanding, and professional accountability that an AI system lacks.

This principle sounds obvious. In practice, it requires organizational policies that make it concrete, explicit statements of what "adequate review" means for AI, generated code, checklists that address the specific failure modes of AI coding systems, and accountability structures that make individuals responsible for the code they accept regardless of how it was generated.

Expand Test Coverage, Not Replace It

The primary technical mitigation for AI coding errors is comprehensive testing, automated tests that exercise the code across a wide range of inputs, including boundary conditions and edge cases, and that provide rapid feedback when the code's behavior deviates from its specification.

AI, generated code should be accompanied by expanded test coverage, not just the tests that the AI itself generates (which may have the same blind spots as the implementation) but additional tests written by human developers or generated through fuzz testing, property, based testing, and other techniques that explore the input space more systematically.

The investment in testing infrastructure that makes AI, generated code safe to deploy is one of the most important and most underappreciated aspects of responsible AI, assisted development.

Specialized Security Review

Given the documented tendency of AI coding systems to introduce security vulnerabilities, security review of AI, generated code should not be treated as equivalent to security review of human, written code. It should be more thorough, more systematic, and more attentive to the specific vulnerability patterns that AI systems are known to produce.

This means integrating static analysis tools, dependency vulnerability scanners, and AI, powered security review tools into the CI/CD pipeline and treating their findings as mandatory review items rather than optional warnings.

It also means that developers responsible for AI, generated code, should have access to security expertise, either through dedicated security review processes or through security training that develops their ability to identify the specific vulnerability patterns that AI systems introduce.

Maintain Human Expertise, Not Just Oversight

One of the most significant long, term risks of widespread AI coding assistance is the atrophy of deep software engineering expertise, the possibility that a generation of developers trained primarily to direct and review AI, generated code develops shallower programming understanding than the generation that learned by writing code from scratch.

This risk is not hypothetical, it is the same concern that arises in any domain where automation takes over the routine work that previously served as the apprenticeship through which expertise was developed. The medical field has grappled with it in the context of robotic surgery. Aviation has grappled with it in the context of autopilot systems.

Organizations that maintain genuinely excellent software engineering, not just organizations that ship a high volume of AI, generated code, will be those that continue to invest in developing deep human expertise alongside AI capability. This means creating opportunities for developers to understand the code they review, not just approve it. It means training programmes that develop genuine software engineering understanding rather than just AI prompting skills. And it means recognizing and rewarding the development of expertise, not just the production of output.

Establish Clear Accountability

When AI generated code causes a production incident, a security breach, a service outage, a data corruption, who is accountable? The developer who reviewed and approved the code. The team that deployed it. The organization that established the development process.

Not the AI system that generated it.

This accountability structure, which is the only reasonable one, given that AI systems cannot bear legal or professional responsibility, needs to be made explicit in organizational policies and understood by every developer who uses AI coding assistance. The productivity gain of AI, generated code comes with the accountability of human, reviewed code. The developer who clicks "accept" on an AI, generated pull request is endorsing that code as their own professional output.

Making this accountability explicit changes the psychological posture with which developers engage with AI, generated code, from passive consumers of AI output to active professional endorsers of code they are responsible for.

The Regulatory Dimension

The question of who checks AI, generated code is not only an organizational governance question. It is becoming a regulatory one.

The EU AI Act's provisions on high, risk AI systems, which include AI used in critical infrastructure, healthcare, financial services, and other sensitive domains, implicitly encompass AI systems that generate code used in those contexts. The requirements for human oversight, documentation, and accountability that apply to AI systems in high, risk contexts extend to the software development processes that produce those systems.

In financial services, regulators including the Bank of England's Prudential Regulation Authority and the US Office of the Comptroller of the Currency have issued model risk management guidance that applies to software systems, guidance that requires validation, testing, and documentation of software in ways that AI, assisted development must accommodate.

In safety, critical software, aviation, medical devices, nuclear systems, formal safety standards including DO, 178C (aviation software), IEC 62304 (medical device software), and IEC 61508 (functional safety) require rigorous verification and validation of software that does not yet have established pathways for AI, generated code. Regulators in these domains are actively developing guidance on how AI, assisted development can be accommodated within existing safety frameworks, but the frameworks themselves were designed for human, written code.

The regulatory direction of travel is clear - the provenance of software, including whether and how AI was used in its development, will increasingly be subject to disclosure and governance requirements. Organizations that develop robust AI code governance processes now are building regulatory compliance as a byproduct of good engineering practice.

The Bottom Line

AI coding systems are getting better at coding faster than almost anyone predicted. The benchmark results are real. The productivity gains are real. The economic incentives driving adoption are powerful and will not be reversed.

But better code generation without better code review is not progress, it is risk accumulation at scale. Code that looks correct but contains subtle logical errors, security vulnerabilities, or context, inappropriate implementations does not become safe because it was generated quickly. It becomes dangerous because the speed of generation encourages insufficient review, and the volume of generation exceeds human capacity to review carefully.

The question "who is checking their work?" does not have a comfortable answer in most organizations deploying AI coding assistance today. The tools have outrun the governance. The productivity gains have been captured faster than the review processes have adapted to accommodate them.

Closing this gap, developing the review practices, testing infrastructure, security processes, accountability structures, and human expertise required to use AI coding assistance safely, is the most important engineering management challenge of the current moment.

The AI is writing the code. Humans are still responsible for it. Acting like that is true, rather than treating AI, generated code as somehow exempt from the accountability that attaches to all code in production, is the foundation of responsible AI, assisted software development.

Written for a global, general audience · July 2026 Topics - AI coding, software engineering, AI code review, GitHub Copilot, Claude Code, SWE, bench, code quality, AI safety in software This article is for informational and educational purposes only and does not constitute professional technical or legal advice.

References

Asare, O., Nagappan, M. and Berger, T. (2023) 'Is GitHub Copilot a substitute for human pair, programming? An empirical study', in Proceedings of the 45th International Conference on Software Engineering - Companion Proceedings (ICSE, Companion 2023), pp. 11–15. doi -10.1109/ICSE, Companion58688.2023.00011.

Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Petroni, F., Berger, A., Cai, T., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I. and Amodei, D. (2021) 'Evaluating large language models trained on code', arXiv preprint arXiv -2107.03374. Available at - https -//arxiv.org/abs/2107.03374 (Accessed - 22 June 2026).

European Parliament and Council of the European Union (2024) Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union. Available at - https -//eur, lex.europa.eu/legal, content/EN/TXT/?uri=OJ -L_202401689 (Accessed - 23 June 2026).

GitLab (2024) The state of AI in software development - 2024 developer survey. San Francisco, CA - GitLab Inc. Available at - https -//about.gitlab.com/developer, survey/ (Accessed - 23 June 2026).

IBM (2024) Cost of a data breach report 2024. Armonk, NY - IBM Corporation. Available at - https -//www.ibm.com/reports/data, breach (Accessed - 22 June 2026).

Jimenez, C.E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O. and Narasimhan, K. (2024) 'SWE, bench - can language models resolve real, world GitHub issues?', in Proceedings of the International Conference on Learning Representations (ICLR) 2024. Available at - https -//arxiv.org/abs/2310.06770 (Accessed - 22 June 2026).

Khlaaf, H., Mishkin, P., Achiam, J., Krueger, G. and Brundage, M. (2022) 'A hazard analysis framework for code synthesis large language models', arXiv preprint arXiv -2207.14157. Available at - https -//arxiv.org/abs/2207.14157 (Accessed - 22 June 2026).

Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J. and Becker, B.A. (2023) 'Using large language models to enhance programming error messages', in Proceedings of the 54th ACM Technical Symposium on Computer Science Education, Volume 1, pp. 563–569. doi -10.1145/3545945.3569815.

Liu, J., Xia, C.S., Wang, Y. and Zhang, L. (2024) 'Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation', in Advances in Neural Information Processing Systems, 36. Available at - https -//arxiv.org/abs/2305.01210 (Accessed - 22 June 2026).

Parasuraman, R. and Manzey, D.H. (2010) 'Complacency and bias in human use of automation - an attentional integration', Human Factors, 52(3), pp. 381–410. doi -10.1177/0018720810376055.

Pearce, H., Ahmad, B., Tan, B., Dolan, Gavitt, B. and Karri, R. (2022) 'Asleep at the keyboard? Assessing the security of GitHub Copilot's code contributions', in Proceedings of the 43rd IEEE Symposium on Security and Privacy, pp. 754–768. doi -10.1109/SP46214.2022.9833571.

Peng, S., Kalliamvakou, E., Cihon, P. and Demirer, M. (2023) 'The impact of AI on developer productivity - evidence from GitHub Copilot', arXiv preprint arXiv -2302.06590. Available at - https -//arxiv.org/abs/2302.06590 (Accessed - 21 June 2026).

Sobania, D., Briesch, M., Hanna, C. and Petke, J. (2023) 'An analysis of the automatic bug fixing performance of ChatGPT', in Proceedings of the 2023 IEEE/ACM International Workshop on Automated Program Repair (APR), pp. 23–30. doi -10.1109/APR59189.2023.00011.

Tian, H., Lu, W., Li, T.O., Tang, X., Cheung, S.C., Klein, J. and Bissyandé, T.F. (2023) 'Is ChatGPT the ultimate programming assistant, how far is it?', arXiv preprint arXiv -2304.11938. Available at - https -//arxiv.org/abs/2304.11938 (Accessed - 21 June 2026).

Weyssow, M., Khomh, F. and Huang, Y. (2022) 'Better modeling the programming world with code concept graphs, augmented multi, modal learning', in Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, pp. 1–12. doi -10.1145/3524610.3527906.

Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, Z., Wang, Z., Ma, M., Zhao, Y., Feng, F., Liu, Z. and Tang, J. (2024) 'CodeGeeX - a pre, trained model for code generation with multilingual benchmarking on HumanEval, X', in Proceedings of the 29th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 5673–5684. doi -10.1145/3580305.3599790.