A flaky test let a coding agent fix the same nonexistent race three times

Chi Cong describes a week in which a coding agent fixed a lost-wakeup race in a Go connection pool four times. Its first hypothesis was good: it named a mechanism, tied it to the goroutine dump, and proposed the smallest change. Each time the integration test went red, the agent wrote a fluent fix that quoted line numbers, reran once, saw green, and the PR merged. By day four there were roughly 400 lines of locks, a retry wrapper and a channel-based rewrite of the pool. Then a reviewer asked: "What's the failing seed?" Nobody had one.

The race never existed. Ninety days of CI history showed the test failing about 1 run in 7 since long before any agent touched the repo, because its checkout deadline was too tight for loaded shared runners. At that pass rate, three "change, rerun, green" cycles in a row happen about 63% of the time with no real fix at all. Go's race detector stayed silent all week, and nobody weighed that. People on the team "just knew" to rerun that test, but the agent starts every session without that memory. To a human a flaky test is an annoyance; to an agent it is a lying oracle, and every lucky roll makes its story stronger.

The team's response has three parts:

• Write the folklore down. A nightly job regenerates a reliability file in the repo that puts every test in a tier (trusted, probation or quarantined) by its pass rate over the last 200 runs, with a 2% flake budget. The agent's instructions say a quarantined failure is not evidence and must never drive a production-code change, and that a single green rerun never confirms a fix.
• Retry before reasoning. For a probation or unlisted test, the agent reruns that one test 20 times in isolation on both its branch and the merge base, and compares failure rates instead of hunting for one green run. If the failure won't reproduce, it files a quarantine ticket with both logs and stops. "Not my bug" counts as a finished task.
• No reproduction, no concurrency PR. Reviewers accept a seeded test that fails on base and passes on the branch, a forced interleaving that fails every time before the fix, or a recorded race or execution trace. They check that the reproduction exists; they don't grade the argument.

They reverted the three merged PRs. The retry wrapper worried the author most, since it would have hidden real checkout failures in production while looking like defensive programming. The first nightly run quarantined 23 integration tests; the team had guessed three or four. A week later the agent hit a red billing test, measured 3 failures in 20 on both branch and main, and filed a ticket instead of a PR.

Why it matters for code review

Agents that run around the clock read CI more often and more literally than any person, so a flake rate people have learned to shrug off becomes a steady source of plausible, green, merged, wrong diffs. The reviewers here were checking the reasoning, which is the cheapest thing an agent produces. Continuous throughput only compounds when the verification signal is honest: keep test reliability data in the repo, give the agent a way to finish with "this failure isn't mine," and review for a reproduction rather than a convincing story. This is one team's postmortem, and the 63% figure is the author's arithmetic rather than a measured rate. Compare Chi Cong's earlier piece on capping open agent PRs per reviewer.