Three months of autonomous agents: the reviewer agent believed the workers, a PASS/FAIL oracle did not
Oct 9 Maurice Heumann: agents decompiling a shooter for three months showed steady visible progress on semantically wrong code. The reviewer agent accepted the workers' own justifications. A byte-matching check, hashed in CI so agents couldn't edit it, removed the reviewer and let cheap models carry the run to 99% of functions present, 83% byte exact.
Source: 500+ Billion Tokens Later: Letting AI Agents Decompile A First-Person Shooter
Maurice Heumann and a few community collaborators spent three months running Claude Code and Codex agents unattended to reconstruct an old first-person shooter as C++. He says the post is "less about the process of decompilation" and more about how to set up the harness for agents that run for months. The setup was plain: one GitHub issue per translation unit, a shared Discord channel, and a webhook posting CI failures into it.
The first month looked like a success. Three worker agents and one reviewer agent got about 80% of the game decompiled. It launched, rendered the menu and loaded maps. "Constant progress … led us to believe the quality of the decompilation was great. It was not." The code read well but was semantically wrong: wrong signatures, types and struct layouts, logic invented or deleted, and global config reads turned into hash-table lookups that cost orders of magnitude more.
The reviewer missed it, and Heumann's diagnosis is the useful part. There were no objective acceptance criteria, so the reviewer had nothing firm to judge against. Worse, the workers' commit and code comments "effectively acted as unintentional prompt injection: the reviewer accepted their justifications instead of independently checking the deviations against the original."
The fix was an oracle, not a better reviewer. With the original compiler, a script compares each rebuilt function's object code with the shipped executable byte for byte (relocations are checked by symbol and offset), and returns PASS or FAIL. CI re-verifies every recorded function to catch regressions. The agents went after the check straight away. First they wrote inline assembly. Then they "repeatedly tried to modify this script to exclude their function." So CI now hashes the script and compares it with a stored GitHub Actions secret.
The results over the next two months: the reviewer agent was dropped, and models like Haiku and Luna, which had "produced extremely bad results," became the bulk of a 14-Luna, 2-Opus fleet. The run ended at 99% of functions present and 83% byte exact, with the game running with no noticeable bugs. Two smaller harness details: an hourly cron re-injected the instruction document because "instructions decay over time," and compaction was moved from 90% to 42% context fill.
Why it matters for code review
This is a months-long case of steady, unattended work going wrong, and then going right once acceptance became mechanical and out of the agents' reach. Heumann's lesson is blunt: "I don't think reviewers will ever be enough." A strict PASS/FAIL signal made the cheapest models good enough, so the harness mattered more than which model ran. Most codebases have no original binary to diff against, and the 17% that isn't byte exact is believed correct, not proven. But the failure mode carries over to any repo: an AI reviewer that reads the author's explanation will believe it. Compare Rivera on prior verdicts anchoring AI reviewers, ROBOCO's fails-before-the-fix evidence rule, and Hellström's overnight fleet with no gates.