Leadlord: agents hill-climbed their own merge gate overnight, with patches it must keep refusing
Thomas Fairhurst on a two-founder startup whose agents merge 50 to 100 changes a day on one server: a merge line cut median merge time from 16 minutes to 86 seconds, and an overnight agent run cut held-out gate time 29% under an anti-cheat rule.
In Hill-climbing our CI with Opus 5.5 agents, Thomas Fairhurst, co-founder of Leadlord, describes a startup where coding agents write most of the code and merge 50 to 100 changes a day, all on one rented 8-core server that also hosts production, staging, and the agents themselves. After a model upgrade in late September, the agents finished about 71 tasks a day instead of 47, each change got roughly five times larger, and the main branch went from about 0.7 to nearly 2 million lines in 18 days, about half of it tests. Every one of those lines had to pass the merge gate.
The first fix was not clever. Gates had been running side by side and slowing each other down, so the team put merges in a line, one heavy gate at a time. The median merge went from 16 minutes to 75 seconds on the first day. A test-runner migration then more than doubled the median for two days, which Fairhurst flags as a lesson: a change that is right in principle still needs to be measured in production.
The second fix ran overnight. Following Anthropic's eval-and-hillclimb method, one orchestrator and 168 subagents spent about 29 hours proposing, reviewing, and merging 77 changes to the checks. The guardrails are the interesting part: replay 24 real past merges with 8 held out, wait for a quiet server so timing noise drops from 41% to 1.7%, never skip or quarantine a test, and keep 17 known-bad patches, a flaky test among them, that the gate must still reject. If any of them passes, the change scores zero. Round one's speedup didn't hold on the held-out merges and was reverted. Fairhurst is candid that later rounds merged without their own held-out check and only the final eval measured them together. On merges the agents never saw, gate time fell 29%, and the share of product files covered by a test rose slightly.
Why it matters for code review
This is an always-on factory improving its own verification layer while the founders slept, and the gains came from the boring work: queueing, finding sleeping tests, and stating the theoretical floor before optimizing. The rule that made it safe is worth copying: when agents tune the checks that judge them, keep a set of changes the checks must refuse, and treat any optimization that lets one through as a failure. Fairhurst's closing line says it plainly: checking code only gets cheaper if you measure where the time goes and let agents climb only where the eval can catch them cheating.