Trail of Bits audited 130+ of its own merged agent patches, found 21 regressions, and built a regression checker that has to prove each one live
Trail of Bits reviewed every Patch the Planet fix its engineers got merged upstream and confirmed 21 regressions in over 130 PRs, from 58 candidates (about 36% precision). Its new regression-hunt plugin replays every suspected regression on a running build in a fresh-context agent and produced 25% fewer false positives than a plain Codex review prompt, at the cost of leaving some real regressions unconfirmed.
Source: Your AI agent fixed the bug, but what else did it change?
Trail of Bits went back over its own agent-assisted security fixes and counted the damage. Reviewing every patch that a Trail of Bits engineer authored and maintainers merged upstream, it confirmed 21 regressions from over 130 PRs merged: 16 functional, one performance, and four build/test, and none broke a security protection. The checking itself was noisy: the 21 real regressions were identified among 58 candidates in a single run (~36% precision), and because agents vary from run to run, each run generates more candidates to manually sort through.
Two plugins, one rule: no live proof, no claim
Their response was to standardize both halves of the job, a standard way to write patches, with the same checks applied every time and a standard way to check for them, with written rules for what counts as a regression. Every step in the new plugins takes and returns a defined schema that includes the claim, the code revision, the required check, the raw evidence, and open questions, so that each step can verify the previous step's work rather than trusting its summary. Every conclusion needs a PoC run against a live deployment, and if a step can't produce a live PoC, its result is inconclusive.
The regression checker enumerates candidate regressions by tracing the diff through its callers, reproduces each one on a running build, replays the PoC on the live deployment on a fresh-context agent, and then has an adjudication step carry the burden of proof that the finding isn't a false positive, an incomplete fix, or a pre-existing bug. The patch-writing plugin adds a three-arm proof: the original code must fail, the patched code must pass, and a complete revert must fail again.
The tradeoff they report
Against the 21 known regressions, regression-hunt produced 25% fewer false-positive findings than the baseline, with a 95% confidence interval of 14% to 38%. The stricter evidence bar has a cost they state plainly: fewer false positives therefore came with a tradeoff in how many known regressions it could confidently report. The improvement loop is continuous rather than heroic. They use the tasks that an earlier version failed to benchmark for the next one, and maintainers tune it with a THREAT_MODEL.md and a PATCHING_GUIDELINES.md that regression-hunt passes to every step.
This is what a verifier looks like when the patch volume outruns human review: precision bought with live evidence, and an honest admission that some real problems end up marked inconclusive. Compare GetYourGuide's Automerge, where agents told to make CI pass deleted the failing test.