TL;DR: Coding agents may handle vulnerabilities with established safe coding patterns, such as parameterised queries against SQL injection, better than vulnerabilities requiring contextual reasoning. For C/C++, existing studies do not yet establish a clear shift in vulnerability classes, and apparently safer patterns such as replacing strcpy with strncpy can still leave security defects.

In our August 2025 benchmark, the SecMate team found at least one security issue in 172 of 240 generated samples. We assessed six tasks in C, Java, Python and Rust using SecMate, our source-code security analysis tool, alongside manual review. We have not rerun that benchmark with newer agents.

A hypothesis: familiar defences are easier to apply

BaxBench, published at ICML 2025, reported SQL-injection occurrence between 0% and 21%, with a median of 0%, across eleven eligible scenarios. Cross-site scripting ranged from 66% to 99% across seven scenarios. These rounded rates apply to functionally correct backends generated without security instructions [1].

For SQL values, a parameterised query gives the model a recognisable defence to apply at the call site. Correct output encoding depends on where the application inserts a value: HTML text, an attribute and JavaScript do not all use the same escaping rules. Access control requires knowing which user may act on which resource. Choosing an API alone cannot supply that policy.

We suspect this difference helps explain why some classes yield better results than others. These BaxBench results concern language models generating backends, so extending that explanation to repository-level coding agents remains a hypothesis. The rates establish neither a decline in SQL injection over time nor the cause of the difference between classes.

In C/C++, safe-looking code can still violate security requirements

C contains bounded functions that leave part of the security requirement with the caller. Consider this copy:

1#include <string.h>
2
3void set_name(char *dst, size_t dst_size, const char *src) {
4    strncpy(dst, src, dst_size);   /* no NUL when strlen(src) >= dst_size */
5}

Replacing strcpy with strncpy does not guarantee a secure string copy. The caller must supply the correct destination capacity and ensure that later operations receive a terminated string when they require one.

Assuming valid, non-overlapping buffers and an accurate destination size, strncpy stays within that destination. It produces a terminated string only when the source is shorter than the bound. A later operation that expects a C string may read beyond the unterminated buffer (CWE-170) [2]. We documented this termination problem in Golioth’s CoAP path handling.

With a non-zero destination size, snprintf(dst, dst_size, "%s", src) produces a terminated result when conversion succeeds and reports truncation through its return value. The caller still needs to supply the correct capacity and decide whether truncation is acceptable.

The Surgical Repair preprint uses strcpy(dst, src) → strncpy(dst, src, n) as one of its safe-pattern substitutions for dataset construction and scoring. The authors acknowledge in Appendix D.1 that strncpy lacks a termination guarantee. Their experiments also show models recognising insecure patterns during review and generating them under prompts that favour other properties, including simplicity or formatting [3]. Learning or recognising a defensive pattern does not ensure that the generated implementation uses it correctly.

Bounds checks can require reasoning even when the code contains no suspicious call. This C++17 helper checks whether an object count can be converted to a byte count without overflow:

 1#include <cstddef>
 2#include <limits>
 3#include <optional>
 4
 5struct Object {
 6    unsigned char payload[48];
 7};
 8
 9std::optional<std::size_t> object_bytes(std::size_t count) {
10    constexpr std::size_t width = sizeof(Object);
11    if (count > std::numeric_limits<std::size_t>::max() / width) {
12        return std::nullopt;
13    }
14    return count * width;
15}

The required relation is count <= SIZE_MAX / sizeof(Object). The caller must handle rejection, allocate enough storage, and keep later accesses within it. Our Slim Bootloader disclosure describes the same class of allocation-size error: integer overflow left insufficient storage for the group descriptor table.

In a repository, input parsing, allocation and access may live in different functions. Ownership bugs unfold over time, races depend on possible interleavings, and protocol bugs depend on legal state transitions. Our Espressif USB analysis follows ownership and concurrent cleanup across functions.

These examples and disclosures describe security requirements that implementations must preserve. They do not establish how often agents violate those requirements or whether agents introduced the disclosed defects.

What C/C++ studies establish

Repository-level evaluations show that coding agents still produce memory-safety defects in C and C++ [4] [5]. SecRepoBench contains 318 code-completion tasks from 27 repositories, covering 15 weakness classes. It compiles the repository, runs developer tests, and reuses OSS-Fuzz inputs to check for the target vulnerability. Its evaluation of 29 standalone models and 15 agent configurations finds that agents outperform standalone models on the same tasks [4]. This establishes a benefit from the agent setup within that evaluation; it does not establish a change in the vulnerability distribution over time.

SecureVibeBench reconstructs 105 vulnerability-introducing tasks from 41 C/C++ projects. Its class-level results complicate our hypothesis: about 10% of generations for heap-based buffer-overflow tasks (CWE-122) are functionally correct but vulnerable, compared with 33% for classic buffer-overflow tasks (CWE-120). Both categories involve memory bounds. Task difficulty, small samples in some classes, and differences between agent frameworks limit what this ordering tells us [5].

Functional success also leaves security failures unresolved. On SusVibes, SWE-agent with Claude 4 Sonnet passes functionality tests on 57.0% of Python tasks and both functionality and security tests on 11.8%. This configuration has the highest functional success in the reported table; SWE-agent with Gemini 3 Pro has the highest joint success at 12.9% [6].

On SecureVibeBench, SWE-agent with Claude Sonnet 4.5 reaches 46.7% functional success and 23.8% in the benchmark’s correct-and-secure category. That category excludes both confirmed target vulnerabilities and additional static-analysis suspicions. The whole gap therefore cannot be counted as confirmed vulnerabilities [5].

Neither C/C++ benchmark establishes whether agents improve more on familiar API patterns than on checks that depend on context. Their aggregate scores and class results describe the evaluated tasks and configurations. Publication in 2026 also does not mean every evaluated model was released in 2026.

Where the evidence stops

Deployed applications contain familiar web weaknesses too. Deng, Fan and Meng manually validated 1,471 findings in 200 publicly deployed applications attributed to AI-assisted development. They found broken access control, cryptographic failures, injection and authentication failures [7]. Those observations describe what auditors found in that corpus; they cannot establish which classes agents introduce more or less often than human developers.

To test our hypothesis over time, we would need stable tasks and prompts across agent versions, enough opportunities for each weakness class, and the same functional and security checks for every version. Rerunning our 2025 benchmark and rescoring both sets of samples with those checks would provide a starting point. Broader C/C++ coverage would still be needed for ownership, concurrency and state-dependent failures.

What to check in generated code

When reviewing generated code, check the conditions around each defensive pattern:

• For SQL queries, verify that untrusted values use parameters and review any dynamically constructed query structure.
• For bounded string operations, check the actual capacity, termination and handling of truncation.
• For allocations, follow sizes from input through arithmetic to allocation and later access. Check how callers handle rejected sizes.
• For ownership and state, follow cleanup paths, concurrent access and legal transitions across functions.

Use security tests to exercise these requirements alongside functional tests. For strncpy, verify that every later use requiring a terminated string receives one.

SecMate uses program analysis to collect the calls, conditions and data relationships relevant to a suspected vulnerability. In our Espressif USB work, we used that evidence to investigate bounds, lifetime and concurrency defects. Researchers validated the findings before disclosure.

References

The SecMate Team