Joey Wang: test a software factory first on whether it stops, not how fast it runs
Oct 23: a design for an overnight, human-reviewed software factory where reviews bind to a commit SHA, failed results survive, and the first pilot deliberately feeds it broken checks, changed candidates, and duplicate events.
Source: Designing a software factory for verifiable delivery
Joey Wang's design for a software factory starts from a modest definition: automation that starts agent sessions in response to triggers. A scheduled job that handles one kind of small maintenance task overnight and leaves a pull request for a person to review in the morning already counts. That is different from a dark factory, where nobody reads the changes at all, and it is the version Wang wants to test.
The design hands reviewers evidence instead of a summary. One durable ticket links the request, the allowed scope, the plan, the consequential decisions, what actually changed, an audit against the original request, and any risk that was explicitly accepted. Review findings name the commit SHA they apply to, so an approval does not carry over when the code changes afterward: in the pilot, every new SHA gets a fresh review. Failed runs stay on the record too, so a final green run does not erase the failures that came before it.
Why it matters for code review
Most factory writing measures throughput. Wang's first pilot measures whether the system refuses to finish. Plant a known defect and see whether review catches it. Fail a required check and see whether the workflow will not complete. Change the candidate after approval and see whether the old approval stops counting. Deliver the same event twice and look for duplicate work. Leave out a needed detail and see whether the agent stops and asks a precise question.
That is the right order for anything meant to run unattended. Steady overnight progress is only worth having if a bad change halts at the gate instead of riding a stale approval into main. Wang is also careful about what a passing pilot would prove: only that those control paths worked under test conditions. Any claim of savings would need a comparable manual baseline and the full cost of review, repairs, and maintenance.