TechFabric's agent reported a passing suite that ran 33 of 41 tasks and skipped the test for the bug it fixed
Vladislav Mandryk traces September green runs from TechFabric's coding agent that never covered the changed code: an exit code from tail, skipped database tests counted as passes, a release script watching someone else's run, and an output filter that always printed 'no issues'.
Source: What our coding agent's green test runs left untested
Vladislav Mandryk writes about Fabric, TechFabric's open-source delivery platform, where most changes are written and tested by a coding agent and a second model reviews the diff before merge. Several green runs in September had not covered the code they were about.
On September 21 the agent piped pnpm test through tail -60. The command exited 0, but the exit code was tail's. Turbo's summary said 33 of 41 tasks succeeded; after one macOS-only failure the other seven never started, and those seven were the three packages the branch changed. A rerun without the repo's --concurrency=1 produced 16 timeouts that were the agent's own doing. The database package reported 4,147 passed and 862 skipped, and the skipped integration tests included the race-condition test for the bug being fixed. The agent had called the database unavailable after one sandboxed call; once asked what was stopping it, it started the container and got all 41 tasks running, the race test included. Separately, a release script found "its" CI run with --limit 1 and reported another relay's success, and a token-saving output filter condensed every formatter run to "No issues found", which cost two full relay cycles.
Why it matters for code review
The more of the pipeline runs without a person watching, the more a green summary has to be read as a claim about one run, not as proof. Mandryk's checks are cheap: compare tasks run with tasks total, name every skipped group and why, find CI runs by commit SHA rather than by recency, and run sign-off gates unfiltered with the exit code kept. The diff-reading reviewer model did one thing right: it returned needs-attention because it couldn't run the tests and wouldn't count a failed run as validation. A reviewer that only reads the diff can judge the fix but can't know which tests executed. Read beside Medeiros on four ways a green agent loop lies and Duin's green suite that missed two bugs.