When the agent writes its own tests, a fresh-context verifier catches what they miss
A pre-registered study seeds 46 faults into agent-built services. An independent verifier in a fresh context beat the builder's own tests; swapping in a second provider's model did not add detection.
Source: Who Verifies the Builder? Toward Independent Assurance for Agent-Produced Software
Sion Israel Sion and Alain April (École de technologie supérieure, Montréal) start from a problem every unattended agent pipeline runs into: when the same agent writes the code and the tests, a green suite stops being independent evidence. Visible tests become a target, and code and tests can share one misreading of the spec.
Their proposal splits a build plane from an assurance plane. A verifier writes tests from the specification alone, without seeing the implementation, then strengthens them against the code with mutation analysis, property-based tests, and fuzzing. Failing tests go back to the builder as counterexamples, and the result is an assurance report that leaves the release call to a human.
In a pre-registered study on three stateful services with 46 seeded faults, the builder's programs were correct but its own tests missed subtle interactions. A verifier running the same model in a fresh context caught more of the seeded faults than self-validation (Holm-corrected p = 0.006), and the spec-first phase did most of the work. A verifier from another provider caught fewer faults, not more: 13 faults favored the same-model verifier and none favored the cross-provider one. The authors note this compares a single model pair, so provider and capability are confounded. Genuine false oracles stayed at 0.7 to 2.3% of tests.
Why it matters for code review
If agents run overnight and grade their own work, the morning reviewer inherits tests that share the author's blind spots. This paper puts a number on the cheap fix: separate the context, not necessarily the vendor. It also complicates the popular advice to put a second vendor's model on review duty. Here, independence came from what the verifier was allowed to see, not whose model it was.