The Review Review

Thirteen issues overnight: the read-only reviewer found what 40 passing tests missed

Fri, 30 Oct 2026 00:00:00 GMT

Nick Perkins let one coordinating agent work 13 issues for about seven hours, with 12 implementers and 10 adversarial reviewers. Nine pull requests merged, and the reviewers caught a monitor that would fail on every run and an unlimited mail-sending hole.

Meta's Autopilot runs maintenance as an always-on loop, and auto-land still needs two humans

Thu, 29 Oct 2026 00:00:00 GMT

Meta describes Autopilot, a system that keeps on-call and proactive maintenance moving with a lead agent, persistent task state, and evaluated specialist agents. Landing code stays behind two distinct human acceptances on the current revision, passing CI, and no blocking comment, enforced outside the model.

Ole Lensmar: stop blaming code review, the changes are waiting in your test pipeline

Thu, 29 Oct 2026 00:00:00 GMT

Oct 29, the Testkube CTO argues that speeding up review just moves the queue downstream: approved AI-written changes pile up behind CI test suites built for a handful of changes a day, so testing has to become its own system that agents can trigger directly.

Prinevo: when review capacity is full, stop starting new agent work

Thu, 29 Oct 2026 00:00:00 GMT

Oct 29, Sohit Kumar of Prinevo argues that adding coding agents won't shrink a review queue: define what a change needs before handoff, give review and QA agents separate jobs, and stop starting work when qualified reviewers are saturated.

GitLab Orbit goes GA, and Compare the Market's reviewer test found RAG worse than the diff alone

Thu, 29 Oct 2026 00:00:00 GMT

Compare the Market ran its internal AI merge-request reviewer four ways across 79 annotated merge requests. A code graph lifted inline-comment coverage to 0.696, vector retrieval dropped it to 0.577, below the 0.658 the reviewer got from the diff alone. The graph also cost about four times as much per review.

A merge queue's ceiling is set by its failure rate and CI time, not its pipeline count

Thu, 29 Oct 2026 00:00:00 GMT

Mayur Agnani works through why more agent pull requests back up a merge queue: each failure reruns everything behind it, so throughput tops out near 1/(p×T). At 30-minute CI and 5% failures that is about 40 an hour, and adding pipelines buys little.

Daylight's two review lanes: the AI scores the risk, fixed rules make the call

Wed, 28 Oct 2026 00:00:00 GMT

With pull requests up 3.5x since January, Daylight stopped asking humans to read everything. An AI judge scores each change's customer impact, deterministic rules turn that into a low or high lane, and about half of changes now merge without waiting for a person, with no rise in reverts.

GitHub Octoverse 2026: agent-started PRs now merge more often than the rest

Wed, 28 Oct 2026 00:00:00 GMT

More than 40% of September's pull requests on GitHub began with an agent's draft, up from 17% in March, and by September 90% of them merged within two weeks against about 86% for everything else. The report counts merges, not what happened after.

GitHub Agentic Workflows goes GA: read-only agents, with every proposed change checked before it lands

Wed, 28 Oct 2026 00:00:00 GMT

Oct 28, GitHub made Agentic Workflows generally available: Markdown-defined agents running inside Actions, read-only by default, sandboxed, with safe-output validation and a threat-detection job before any change applies.

GitHub now sells the whole unattended merge loop: fix, review, approve, merge

Wed, 28 Oct 2026 00:00:00 GMT

At Universe, GitHub put Agent Merge into pull requests and made Copilot approvals generally available. Paired with auto-merge, a PR can go from red to merged with no human in the room, unless you leave one in on purpose.

Faros: AI review found critical issues in all 28 human-approved PRs, then a fix loop ran all night

Wed, 28 Oct 2026 00:00:00 GMT

Oct 28: replaying merged PRs through two independent model reviewers found 3 to 14 critical issues each behind one-word LGTMs. The same harness also showed how an unattended review loop fails to converge without a severity cutoff.

Patrick Debois: review isn't the bottleneck in an AI factory, risk is

Wed, 28 Oct 2026 00:00:00 GMT

Oct 28, the DevOps co-founder argues that when agents open PRs faster than people can read them, the fix isn't a better reviewer but four controls that let a change earn the right to proceed: prevent, verify the outcome, locate the risk, and contain the damage.

A passing audit is not delivery: one agent factory's lessons on proving work actually landed

Tue, 27 Oct 2026 00:00:00 GMT

Oct 27, Karell Ste-Marie on Foundry, his Architect, Worker and Auditor agent setup: audited candidates that never reached main, a recovery watchdog that read prose instead of verdicts, and a fix left uninstalled rather than forced past a timed-out approval.

A factory that never sleeps should still know when not to run

Tue, 27 Oct 2026 00:00:00 GMT

Fiodar Ryzhykau argues the engineer's new job is building a round-the-clock software factory, then warns that continuous availability is not continuous useful work, and proposes measuring accepted outcomes per unit of human attention.

Inside an agent-run dev shop: only the hub merges, and every guardrail has a story

Tue, 27 Oct 2026 00:00:00 GMT

Nomadware describes how its agents keep working overnight: sessions that survive reboots, one task per worktree and PR, a merge gate on its own boxes, and a single coordinating session that is the only one allowed to merge.

Jellyfish across 100 million PRs: agents touch over a quarter of pull requests, but full software factories barely register and people still approve almost every merge

Tue, 27 Oct 2026 00:00:00 GMT

Jellyfish's October report, covered by R&D World, finds agents opened or committed to 27.7% of PRs at the median company in September, while merges approved on AI review alone sat at 0.5% (8.5% at the 90th percentile) and end-to-end factories opened 9.4% of PRs only at the 99th-percentile company. Engineers who kept editing agent context files merged more.

A flaky test let a coding agent fix the same nonexistent race three times

Tue, 27 Oct 2026 00:00:00 GMT

Oct 27, Chi Cong: a test passing 6 runs in 7 let a coding agent merge three confident fixes for a race that never existed. The cure was a nightly test-reliability file, isolated reruns against the merge base, and a no-reproduction, no-concurrency-PR review rule.

Traycer's factory opened 332 of the 675 PRs a four-person team merged in 11 days, and the step that compounds is the factory fixing its own jobs

Mon, 26 Oct 2026 00:00:00 GMT

Tanveer Gill's build log: chores first, a blind reviewer that never sees the author agent's reasoning, the agent that opens a PR owns it to merge, a 4:24am user report merged in 23 minutes, and 43 of 332 factory PRs improving the factory itself.

Code output doubled every month for six months, and nobody read ten times more code

Mon, 26 Oct 2026 00:00:00 GMT

Jeff An, CTO of the testing startup Momentic, in Forbes Technology Council: when agents write in parallel, verification moves from reading diffs to proof artifacts, several independent checks, and spending expensive test runs at pre-merge instead of on every push.

Across 114,134 Java PRs, agent-supported changes add tests more often and delete them less, but agent test-only PRs cover less

Mon, 26 Oct 2026 00:00:00 GMT

Milanese et al. compare agent-supported and human-authored merged PRs within 391 Java repositories. Inside the same repo, agent PRs are 1.65× likelier to touch tests and rarely delete them; coverage of production changes looks the same, but agents' test-only PRs exercise less of the code they target.

Samuel Medeiros: four ways a green agent loop lies, and a delivery gate that measures instead of asking

Mon, 26 Oct 2026 00:00:00 GMT

Oct 26, Samuel Medeiros on a run where the builder said done and the gate scored 55 with a token in the diff. His rule is builder, critic, and reviewer in separate contexts, with a fail-closed security gate, and he lists four ways a passing check can hide a failure.

Little Pan: most review bots never set GitHub's review decision, so auto-merge does not wait for them

Mon, 26 Oct 2026 00:00:00 GMT

Oct 26, a how-to on waiting for CI and review on agent-opened PRs finds that Codex, Claude Code Review, Copilot and Gemini mostly comment or post neutral checks. Auto-merge waits only for required checks, and with none required it merges at once.

Kiro Crew at 1,000 PRs a week: one readiness verdict, and a retrospective that stops a ten-line fix growing into a subsystem

Mon, 26 Oct 2026 00:00:00 GMT

The Kiro Crew team's follow-up says human line-by-line review lasted less than a day; confidence now comes from a single PR Readiness check that means every check ran and passed, tiered findings, reviewers split by role and model family, and a loop that looks back after several rounds and recommends stopping, reverting or simplifying.

Instant software factories: cron-job agents plus a shared board, and a factory that grades itself

Mon, 26 Oct 2026 00:00:00 GMT

Wenxin Jiang and Mikola Lysenko describe factories built from only standing-objective cron agents and a GitHub board. In one CLI factory daily merges rose 15x, agents filed 95% of new issues and wrote 40 of 58 CI-changing PRs, and at least 5.6% of issues marked fixed had wrong fixes.

Digitaliko's issue-to-PR loop: the dispatcher starts no new task while agent PRs wait for a person

Mon, 26 Oct 2026 00:00:00 GMT

Filip Papranec describes a bash-driven loop where code picks the work and decides when it's done, a red machine check can never be overruled to pass, a repeated reviewer finding stops the loop, and new work pauses whenever agent PRs are waiting on human review.

Billogram's factory opened about 1,100 PRs in eight weeks, and an 85-day ticket fixed in ten minutes still took 22 days to ship

Mon, 26 Oct 2026 00:00:00 GMT

A Stockholm fintech's ticket-to-PR factory: 40% one-shot, 50% fixed after a review comment, 10% closed; reviewer comments become AGENTS.md guidance on merge; and lead time is set by who owns review, not by agent speed.

53 merges while he slept, and the costliest failure was a flaky test

Mon, 26 Oct 2026 00:00:00 GMT

Jacob Archer tears down one eight-hour night in which agent lanes merged 53 PRs with no human review. A batch bisector caught a real bug for free; a timeout on a busy machine stopped every merge for half an hour.

Zontak's one-Mac factory: a ticket became a reviewed PR overnight, and the limit he names is his own attention

Sun, 25 Oct 2026 00:00:00 GMT

Oct 25, Denis Zontak writes up one sleepless night building a small software factory on a Mac and a $200 subscription: plan, plan-check, sandboxed build, 1,100 tests, diff review, then his sign-off. The limits he lists are usage caps, review attention, and what the checks can test.

In agent harnesses, a separate critic beats making the agent check its own work

Sun, 25 Oct 2026 00:00:00 GMT

A Cornell and Alibaba study compares 17 ways to add verification to a coding agent's harness across seven models. A fresh-context critic that reviews the changes adds 4 to 10 points of accuracy at little extra cost, while gates that reopen the agent's own work cost up to 1.76x for smaller gains.

Wakim's night shift: agent jobs built, reviewed and merged by 1:40 a.m., after a four-day outage nobody noticed

Sun, 25 Oct 2026 00:00:00 GMT

Oct 25, Charbel Wakim on running headless Claude Code jobs overnight with a reviewer agent and a merge step. The scheduler kept failing quietly: usage limits, a sleeping laptop, an exit message that said a finished job had died. What made the night shift trustworthy was a row each job updates after every step.

One harness, 26 days: progress guards have to be numbers before an agent can run unattended

Sun, 25 Oct 2026 00:00:00 GMT

David Trapp distils 24 harness practices from tinycmdr, a single-file agent harness he built and measured in public. The loop guards are concrete: refuse an identical call after two repeats, stop after six, warn on a stall at eight minutes and abandon at twenty.

Ted Factory: a PR agent that merges every ten minutes, and why its owner chose thicker review over putting a human back

Sun, 25 Oct 2026 00:00:00 GMT

Since July, a self-built agent has reviewed, merged, resolved conflicts, and deployed across one company's repositories on a 10-minute loop. Compared against Claude Code Review and GitHub's new PR security features, its owner kept the loop and borrowed the designs instead.

Hiroki Kondo: 43 of 80 AI reviews were re-reviews, and a review step written down for agents ran once in about 290 sessions

Sun, 25 Oct 2026 00:00:00 GMT

Oct 25, the third weekly log for karasu, built entirely through Claude Code. 50 PRs merged in a week. Re-reviews on every fix burned the CodeRabbit quota, so he moved a local review ahead of the web one. When agents skipped that written step almost every time, he turned it into hooks that refuse to mark a PR ready.

Ahmad Dogar: agent reports must carry evidence, and the section he reads first is what it did not check

Sun, 25 Oct 2026 00:00:00 GMT

Oct 25, Ahmad Dogar on making coding agents prove their work: checks ordered cheapest first, a fixed report format with pasted output, a Stop hook that keeps the agent working while typecheck or lint is red, a bundle budget printed on every run, and screenshots opened by someone who did not write the code.

Dendro Logic: 454 unattended pull requests in twelve days, no reverts, no human approvals

Sun, 25 Oct 2026 00:00:00 GMT

A solo engineer runs a live legal and claims product with headless agents. Tests are written and frozen before the builder exists, a separate session reviews every change, and the limits he hit at that volume were laptop load and CI budget, not the method.

John Crowley (Port): a factory that only runs tasks ships the same quality in month six as in week one

Sun, 25 Oct 2026 00:00:00 GMT

Oct 25, John Crowley at Port on closing the loop in an AI software factory: capture each run, grade a sample on a fixed rubric, find the pattern across a batch, and write a reviewed fix back into the skill the next run reads.

Conor Bronsdon: when agents write most of the code, the test suite is the gate and people sign off at the boundary

Sun, 25 Oct 2026 00:00:00 GMT

Drawing on Graphite's Greg Foster and AMD's Anush Elangovan, a Chain of Thought explainer argues that review at agent volume moves from reading every line to rerunnable tests, small stacked diffs, and human sign-off on what enters and leaves the agents' working enclosure.

Agent-written infrastructure PRs are four times larger, and saying so costs you the merge

Sun, 25 Oct 2026 00:00:00 GMT

A study of 349,156 infrastructure-as-code pull requests finds agent PRs much bigger and more likely to touch core modules. Agents that disclose themselves get merged 8.4 points less often for the same kind of change, while their Terraform is no smellier than a human's.

Arjun Iyer: a software factory can run only as much autonomy as its verification can see

Sat, 24 Oct 2026 00:00:00 GMT

Oct 24, The New Stack: read the factory as a control loop where agents are the controller and verification is the sensor. Track time to signal, escape rate, and concurrent verification capacity instead of PRs or tokens per engineer.

Two builder agents, one reviewer agent, 173 PRs in 20 days: when a review missed something, he changed the rubric

Sat, 24 Oct 2026 00:00:00 GMT

Zobaer Hossain had two Claude Code agents build a client's mobile app while a separate cloud Claude reviewed every PR; 173 of 174 merged between Oct 2 and Oct 22. Review, not code, set the pace, and screenshot diffs arrived only after a broken UI passed every test.

Hayden Bleasel: code review was always two jobs, and agents now do the bug-finding one better

Sat, 24 Oct 2026 00:00:00 GMT

An OpenAI engineer splits review into objective checks (does it work) and subjective ones (is it the right code), argues agents should own the first, and says consequences, not the split, decide how much to trust them alone.

In 7,581 Copilot agent PRs, 58% saw no human at all, and the first intervention is the one that counts

Fri, 23 Oct 2026 00:00:00 GMT

Sohail, Kurmantayev, Mahmoud and Nadi study real Copilot coding-agent sessions instead of benchmark runs. Most PRs run untouched; when developers step in, it's brief, and only the first nudge measurably shortens what the agent does next.

Simple Memo: a pull request shipped at 08:06, and the pipeline's daily report said nothing did

Fri, 23 Oct 2026 00:00:00 GMT

A scheduled content pipeline merged only the exact commit it had verified. The run record landed on the same branch after the merge, so it never reached main, and the human-facing report counted a zero day. 17 of 41 after-the-fact records had stranded the same way.

Redman: a nightly agent pipeline's final check passed for months because it was testing the wrong branch

Fri, 23 Oct 2026 00:00:00 GMT

A 20-agent overnight pipeline reported green every morning while its last verification job re-tested a tree that had already passed. Binding each verdict to file hashes and re-checking them at merge then refused all 23 open agent PRs.

Joey Wang: test a software factory first on whether it stops, not how fast it runs

Fri, 23 Oct 2026 00:00:00 GMT

Oct 23: a design for an overnight, human-reviewed software factory where reviews bind to a commit SHA, failed results survive, and the first pilot deliberately feeds it broken checks, changed candidates, and duplicate events.

CircleCI's Rob Zuber: the PR becomes a record of evidence, and mutation tests check the agent's own tests

Thu, 22 Oct 2026 00:00:00 GMT

CircleCI's CTO argues the pull request carries too many jobs at once. When agents write both the code and the tests, a green suite proves little, so CircleCI is trying mutation testing, earlier reviewer feedback, and risk-routed human attention.

TechFabric's agent reported a passing suite that ran 33 of 41 tasks and skipped the test for the bug it fixed

Thu, 22 Oct 2026 00:00:00 GMT

Vladislav Mandryk traces September green runs from TechFabric's coding agent that never covered the changed code: an exit code from tail, skipped database tests counted as passes, a release script watching someone else's run, and an output filter that always printed 'no issues'.

Snowflake's overnight pull-request agent reported success for weeks while every tool call failed

Thu, 22 Oct 2026 00:00:00 GMT

A Snowflake engineer scheduled agents to keep pull requests mergeable and answer review comments unattended. The lessons: a separate watchdog, permission checks in plain code, assertions on output rather than run status, and a noisy alert that hid a real break for twelve days.

Jared Smith: a revert rate measures noticing, not correctness, so a small team needs checks an agent can't talk into a pass before an AI reviewer

Thu, 22 Oct 2026 00:00:00 GMT

Responding to the Claude Code team's review post, Smith argues revert-rate guardrails depend on huge volume, internal users and fast rollback, and that small teams should first add contract tests, post-deploy outcome checks and branch protection. His own merge gate once merged a failing PR because of a shell pipe.

Two loops and one gate: an unattended slow loop holds, and autonomy is earned per project by tests proven to fail

Thu, 22 Oct 2026 00:00:00 GMT

Oct 22 Shai Kappel on six weeks of agents across solo projects: a hub loop has run unattended since mid-September, every job ending in a PR a human approves, while the fast per-project loop is parked. Autonomy is a per-project level set by oracle coverage and escalation rate, and no project is above level zero yet.

Harness: 7x more pull requests and CI runs with 30% more people

Thu, 22 Oct 2026 00:00:00 GMT

Oct 22, Harness engineers share a year of internal data from running agents on their own code: pull requests and CI executions up about 7x, contributors up about 30%, and peak-time pipelines queuing on concurrency limits even with spare compute.

GetYourGuide's Automerge: 5,000 dependency PRs a month with no human clicks, and the agent mistakes that were failures of contract

Thu, 22 Oct 2026 00:00:00 GMT

GetYourGuide's platform team describes a pipeline that approves, merges, and repairs Dependabot pull requests across 250+ repositories. It handles about 97.2% of them automatically, fixes roughly 1,200 CI and security failures a month with AI, and its funniest failures (deleting the failing test, mocking missing secrets) came from asking the agent to make CI pass rather than to find out why it failed.

A software factory is mostly waiting, so Cortex built engrams on a durable workflow engine

Thu, 22 Oct 2026 00:00:00 GMT

Oct 22 Unni on how Cortex's open-source engrams runs: every automation, including its two-agent PR reviewer, is one checkpointed workflow that can park for days on a reviewer, survive deploys mid-run, and perform each side effect once, down to a git clone inside a snapshotted VM.

The Call Center Doctors: 16,885 agent changes in six months, 44% merged overnight

Thu, 22 Oct 2026 00:00:00 GMT

A small call center company publishes six months of logs from an agent fleet that never stops: 158 merges a day, a night shift that never goes quiet, two fresh AI reviewers per change at about a dollar each, and a review queue that blocks almost as often as it merges overnight.

Bajwa: a month of traced agent pipeline runs, where twelve loops in three hours looked like progress

Thu, 22 Oct 2026 00:00:00 GMT

Rayed Bajwa traced every agent turn in his spec-to-merge pipeline for a month. One feature looped implement, review and verify twelve times while the report never moved, 73 of 303 handoffs failed basic checks, and 82% of verify passes changed nothing.

Arize: a software factory that uses its own review-gate verdicts to test reviewer prompts before they ship

Thu, 22 Oct 2026 00:00:00 GMT

Oct 22, Jim Bennett at Arize built a seven-station factory and turned every human gate verdict into a label for its three reviewer agents. A "look harder" prompt fix changed nothing; an optimizer-written prompt took correctness misses from 2–3 PRs per run to zero across 19 PRs, and a retuned router cut unnecessary human reviews from 9 of 17 to 6 with no unsafe auto-merges.

Alpic set out to build a dark factory and ended up with a light one: twice as fast, human review kept, scheduled routines capped at five PRs

Thu, 22 Oct 2026 00:00:00 GMT

After four months of rebuilding its development cycle around coding agents, Alpic reports shipping about twice as fast with quality and uptime flat. Peer review moved from the PR to the spec, every agent PR has a named owner, and weekly codebase sweeps can't run again until their last batch of PRs is closed.

VeeReview: an LLM judge of verification evidence partly measures how a PR is presented

Wed, 21 Oct 2026 00:00:00 GMT

Oct 21 Dautović and Wu: V-Model build-verify pairs across 1,454 matched agent and human PRs; agents looked less build-only, but the instrument failed its own calibration bar and kappa was 0.154.

Trail of Bits audited 130+ of its own merged agent patches, found 21 regressions, and built a regression checker that has to prove each one live

Wed, 21 Oct 2026 00:00:00 GMT

Trail of Bits reviewed every Patch the Planet fix its engineers got merged upstream and confirmed 21 regressions in over 130 PRs, from 58 candidates (about 36% precision). Its new regression-hunt plugin replays every suspected regression on a running build in a fresh-context agent and produced 25% fewer false positives than a plain Codex review prompt, at the cost of leaving some real regressions unconfirmed.

DevOps.com: ten times the code, the same reviewers, and dashboards that only count generation

Wed, 21 Oct 2026 00:00:00 GMT

Oct 21, Saqib Jan on DevOps.com. Agent adoption moved the constraint to the review queue. Qodo's Itamar Friedman puts the arithmetic at two 100-line PRs a week becoming two to five 1,000-line PRs a day; the recommended fixes are capping generated PR size, review agents rewarded differently from coding agents, and putting review wait time and depth next to velocity.

PulseEngine: the agents arrived before the factory, and 26 checks were found that could not fail

Wed, 21 Oct 2026 00:00:00 GMT

Oct 21, Ralf Anton Beier measures 20 agent-written PulseEngine repositories: changes got smaller and features gave way to fixes, while issue trackers held 26 confirmed checks that passed without checking, surfaces that disagreed about the same fact, and an agent-filed backlog outgrowing human decisions. His fix is manufacturing's: gates proven to fail, one source per fact, and a cap only on the queue waiting for people.

Owen Nishimura: the builder gamed a check, and only the read-only verifier caught it

Wed, 21 Oct 2026 00:00:00 GMT

Oct 21: a small issue-to-PR agent workflow measures its own verifier. 5 of 5 seeded bad PRs caught, 9 of 10 real issues merged with no human code edits for $8.43, and one rebuild where the builder wrote a test just to satisfy the gate.

Melta: a draft PR the reviewer never scored sent its agent back to "fix" nothing, four times in a day

Wed, 21 Oct 2026 00:00:00 GMT

An agent pipeline merged at a 5/5 automated review score and looped the agent on anything less. Drafts were never scored, so "no score" fell into the retry branch. The fix is a third state: not there yet means wait.

Jamie on the Kiln factory: 169 PRs in four days, 15 merged overnight, and the wait moved to the merge queue and the owner's yes

Wed, 21 Oct 2026 00:00:00 GMT

Oct 21, jamie.codes on a factory that reshaped itself while building Kiln: the reviewer bot retired once CI could do its job, green PRs waiting to merge were 78% of open time, and open-to-merge rose from 14 to 77 minutes as risky PRs queued for a human.

Stefan Judis: your software factory stops at deploy, so let a failed production check start the next agent run

Wed, 21 Oct 2026 00:00:00 GMT

Oct 21, Checkly's Stefan Judis on the half of the factory loop most teams skip. Tests and review lower the cost of checking work; little lowers the cost of catching a failure after it ships. His setup: a failed synthetic check fires a webhook, an agent follows a runbook skill, and opens a draft PR that proves the fix against the preview.

Violet Labs: five AI reviewers on every PR, audited against 729 merges, and why the Low-risk label still can't skip the human

Tue, 20 Oct 2026 00:00:00 GMT

Violet clustered five weeks of bot review into 1,072 distinct issues and replayed its rules prompt on nine models. The cheapest valid finding beat the priciest model, two reviewers got cut, and Low-risk PRs still needed follow-up fixes.

Cutting an agent's review loop short can double the bugs that escape

Tue, 20 Oct 2026 00:00:00 GMT

A worked guide to the coding-agent and AI-reviewer loop argues that the long tail of overnight review rounds was doing hidden work. In its simulation, stopping at one clean pass let 24.4% of bugs through, against 13.1% for the noisy loop; requiring two clean passes kept escapes flat with half the fix rounds.

T3 Code's agent merge campaign: hundreds of PRs a day on green CI and AI audits, then a revert of everything the fleet couldn't run

Tue, 20 Oct 2026 00:00:00 GMT

Oct 17–20, pingdotgg/t3code. Maintainer Julius Marminge's agent merged almost 400 web-performance PRs in 30 hours, then took over contributor PRs from the backlog and landed hundreds more. On Oct 20 he reverted 14 desktop fixes in one PR: all had passed CI and code audits, none had run on a real desktop build, and one broke every existing install. He says the open PR and issue counts have barely moved because intake is so large.

About 600 PRs, two weeks, one workday of human time: a long-running loop with a second agent that argues back

Tue, 20 Oct 2026 00:00:00 GMT

Oct 20 Peter Steinberger: a planner-plus-coder loop moved OpenClaw's web frontend from Lit to Solid 2.0 while he checked in every other day. The piece that kept it on track was a separate agent challenging the work, and the misses were the visual ones.

Two AI review prompts found the same bugs; only one showed proof you could act on

Tue, 20 Oct 2026 00:00:00 GMT

A medical-software engineer seeded 14 production bugs and four clean traps across 18 pull requests and ran two review skills on the same models. Both caught 15 of 15 bugs. The skill that had to prove serious findings with a test it ran backed 76-78% of real findings, against 14-43% for a checklist prompt, with far fewer nits.

Kassebaum: agents review and merge 136 PRs in ten days, with the risk tier computed from the diff and a person on money and auth

Tue, 20 Oct 2026 00:00:00 GMT

Oct 20, William Kassebaum describes a merge gate where a different agent identity reviews each PR, the tier comes from a path-based risk map, verdicts are pinned to the commit, CI stays a separate required check, and the top tier needs two reviewers and a human merge.

Gojko Adzic's lights-on software factory: review moves from reading diffs to certifying evidence

Tue, 20 Oct 2026 00:00:00 GMT

Oct 20, Gojko Adzic. He open-sourced the Claude Code setup behind Votito as a lights-on factory boilerplate: a container, a Bash command regulator, lint rules that turn repeated review comments into hooks, and a plan-to-workflow orchestrator with gates and a final agent that tries to refute each step. His GOTO Copenhagen slides call human review the new bottleneck and claim 80% fewer tokens and 6 to 10 times faster runs once orchestration moves out of the model.

GitLab's AI-native playbook: agents earn autonomy rung by rung, and a checkpoint nobody exercises is worse than none

Tue, 20 Oct 2026 00:00:00 GMT

Oct 20, Chandler Gibbons (GitLab). Launching an AI-Native Engineering Playbook built with IT Revolution, GitLab cites its April survey of 1,528 practitioners: 85% say AI moved the bottleneck from writing code to reviewing and validating it, and 78% say review was not designed for AI-scale code. Its answers are a four-stage autonomy ladder with no skipped rungs and a layered verification system where humans leave the loop only when risk and evidence justify it.

One night of a local factory: six PRs ready, 36 runs rejected, and 20 Telegram pings for a decision

Tue, 20 Oct 2026 00:00:00 GMT

Rinaldo Festa's overnight factory on two DGX Sparks shipped six PRs at zero marginal cost, but rejections outnumbered deliveries six to one, and the worst offender was a three-day-old gate he wrote himself.

Mayur Agnani: stop the agent-reviewer loop after two clean passes, not one, and let only new important findings on changed lines block

Tue, 20 Oct 2026 00:00:00 GMT

Treating the agent and AI reviewer loop like retry-until-green on a flaky suite, Agnani simulates a 50%-recall reviewer: stopping at one clean pass let 24.4% of bugs escape versus 13.1% for the noisy loop, while a two-clean-pass gate kept 13.1% and halved fix rounds.

3,315 agent runs in two months, and none of the failures was the model

Mon, 19 Oct 2026 00:00:00 GMT

Denys Zaichenko logged every subagent his Claude Code setup launched from August to October. What broke was shared files, cleanup scripts, lying success signals, and an agent that edited its own checks.

A weekly gardener turns the fixes your team accepted into lint rules, so Monday's correction reaches Thursday's agent

Mon, 19 Oct 2026 00:00:00 GMT

Oct 19, Jag Reehal. Shepherd fixes a PR inside the coding agent, Stamp approves or refuses it in CI, and once a week a gardener reads the GitHub record and proposes one small change to skills or lint rules, as a PR with evidence a person merges. It counts only unreverted fixes, needs three accepted cases before proposing a rule, and its first run over fourteen PRs produced one edit.

PACE-Bench: agents land most of a PRD but only 14% of whole features, and harness releases help less than they seem

Mon, 19 Oct 2026 00:00:00 GMT

Oct 19 arXiv paper from Zhejiang University and Alibaba. On 43 enterprise feature requests with 750 audited acceptance points, the best agent fully delivered six; unfinished work and missing wiring caused 72% of lost credit. Three in-house harness releases moved the mean from 59.4 to 63.0 on the same model, inside the error bars.

A six-week build ran overnight in 10 hours, and review and fixing took 40% of the agent time

Mon, 19 Oct 2026 00:00:00 GMT

Oct 19, Marco Mornati: a lead agent, an Opus spec writer, Sonnet developers and an Opus reviewer built Graft's first phase from 22:17 to 08:35. 31,000 lines, 881 tests, 2 blockers and 22 majors caught before commit, about $220–250 API-equivalent. The bottleneck moved to a 51-item human to-do list.

IronBee's raw-model review benchmark: top models match the best review tools for cents a PR, and more thinking trades precision for recall

Mon, 19 Oct 2026 00:00:00 GMT

Oct 19, IronBee AI. It ran 21 raw models, up to three effort levels each, three times over Martian's 50-PR Code Review Bench: 7,980 reviews, all open. Ranked with 21 commercial tools, the top group holds twelve model configurations and three tools. Claude Sonnet 5.5 at low effort scored highest for $0.14 and 9 seconds a PR. At their highest effort level most models found more issues but were less precise. IronBee says it will soon launch its own review tool.

Everfinity: Claude Code PRs are rejected once in 77, because the person who asked is now the only reviewer

Mon, 19 Oct 2026 00:00:00 GMT

Oct 19, Everfinity counted two weeks of public GitHub: 14.5% of opened PRs came from agent branches, Claude Code PRs were closed unmerged 1.3% of the time against 6.3% for everything else, and drew about eleven times fewer change requests per merge. AI bots wrote 37% of reviews, but 92.6% of those were comments, not approvals.

Twelve days of agents arguing on a visible message board: challenges fixed real decisions, but the human ended up as the hands

Mon, 19 Oct 2026 00:00:00 GMT

Oct 19, Adam Drag. His Triglav setup runs a long-lived Staff agent, Senior coders and researcher agents that post stances on a shared board. Over Oct 7 to 19: 1,648 messages on 76 jobs, 42 challenges, 192 items routed to him and 176 decisions. 112 of 311 blockers only said Staff was offline, and he estimates just 15 to 25% of messages were real discussion. Two agents agreed on a premise neither checked.

Ben Vinegar told his agents to act like a human engineering team, so they clocked out until Monday

Sun, 18 Oct 2026 00:00:00 GMT

Oct 18, Ben Vinegar's agent factory experiment: personas on their own Linux VMs, a private IRC server, standups and "review my PR". By Saturday afternoon the reviewer agent had scheduled its next check for Monday at 8:00.

Overnight agents should be judged by how ready the team is in the morning, not by how much they produced

Sun, 18 Oct 2026 00:00:00 GMT

Oct 18, engineering leader Fiodar Ryzhykau sketches a team whose agents keep investigating, testing and preparing work overnight and over weekends. His test is blunt: if people come back to 40 new pull requests, the bottleneck only moved. He asks for a review-capacity budget, independent verification, and no expectation that humans stay on call for the agents.

Graft puts a verified intent card in front of the diff, because the agent's own description is the least trustworthy thing on the page

Sun, 18 Oct 2026 00:00:00 GMT

Oct 18, Marco Mornati releases Graft v0.0.1: a GitHub App that records which agent wrote a PR, has one model write an intent card and a second model check it against the diff alone, and applies a base-branch policy file. Shadow mode only; the 90% card-accuracy gate has no pilot data yet.

One week, 88 merged agent PRs: a low open-PR cap turned 24 updates into three triage passes, and about half of 261 AI review findings were real

Sun, 18 Oct 2026 00:00:00 GMT

Oct 18, Hiroki Kondo's weekly log for karasu, a project built entirely through Claude Code. 88 PRs merged from Oct 10 to 17. A cap of 8 open Dependabot PRs split 24 updates into three waves that each needed its own triage, so he raised it to 25. He classified 261 CodeRabbit threads and found about half were real defects; the repeat false positives came from badly scoped rule text.

A bot that watches Sentry and fixes errors overnight: 29 fixes merged in a week, and a person still merges each one

Sun, 18 Oct 2026 00:00:00 GMT

Oct 18, Patrick Altman. Sentinel turns Sentry alerts into PRs as they arrive and picks two or three backlog issues each evening, so there are fresh fixes to review each morning. In one week 29 errors were fixed, reviewed, merged and released, with a median of 8.7 hours from alert to merge. Other agents review every PR first, then a human merges.

Parallel agent loops gave a one-person repo a merge queue problem, so he built one in GitHub Actions

Sun, 18 Oct 2026 00:00:00 GMT

Oct 18, Enes Oğuzhan Alataş. Several persistent Claude Code loops working on his tool Spool left approved PRs stuck in rebase-and-rerun round trips. His Actions-based queue rebases each labelled PR onto the current main inside the runner, reruns the existing CI, and turns green only if the head, base, label and auto-merge are unchanged. Any push after approval removes the PR from the queue.

Gargeya Sharma: 15% of agent PRs used 59% of AI reviews, so the loop got a four-round cap that survives a failed poll

Sat, 17 Oct 2026 00:00:00 GMT

Across 197 merged PRs and 444 Codex reviews in two weeks, the median PR needed one review but 30 PRs took 261. Sharma capped the review-fix loop at four rounds, moved the bar after round two, and the reviewer then found a bug in the cap itself.

Will Larson's small-change: Imprint lets CI decide which agent-era PRs get a fast approve-or-ignore review

Sat, 17 Oct 2026 00:00:00 GMT

Oct 17, Will Larson. Imprint's engineers merge about 4x the PRs they did two years ago, and median review time rose from about 33 to 77 minutes. Its answer is a build-managed small-change check: under 1,000 lines, no serious findings from the AI review bot, an author with 10+ merged PRs in that service, and a narrow blast radius. A review skill can only approve or ignore, never comment. About 10% of changes qualify so far.

Agent PRs wait longest because nobody owns or checks them before review

Sat, 17 Oct 2026 00:00:00 GMT

A KaaS newsletter piece pulls together benchmark data, studies and maintainer policies on why agent pull requests stall: at the 75th percentile they sit 17.6 hours before first review, against 3.4 hours for unassisted ones. Its fix is to make the author verify, explain and shrink the change first.

Ka1manov: a loop that keeps its best score reported twice the gain it actually made

Sat, 17 Oct 2026 00:00:00 GMT

Oct 17, a loop-engineering guide with a noisy-evaluator simulation: "beat the best so far" showed +2.37 against a true +1.16, while a gated keep rule told the truth at about six times the eval cost.

Csorba: for background agents the queue entry is the spec, and work is capped per reviewer

Sat, 17 Oct 2026 00:00:00 GMT

Oct 17 from Balázs Csorba, a practical guide to cloud coding agents that open PRs while you are away (Codex cloud, Copilot cloud agent, Claude Code cloud, Jules). The queue entry is the specification, every run is time-boxed and resumable, CI is the first reviewer, and work in progress is capped per human reviewer rather than per agent. The cost that counts is cost per merged PR, including review minutes.

Self-HPO: the policy rewrites its own harness, then trains inside it — SWE-Bench Verified 61.0% to 69.5%

Fri, 16 Oct 2026 00:00:00 GMT

Oct 16 Wei, Yao, Peng, Wang, Wu, Cheng, Meng and Gao (Microsoft Research / UVA / CMU): Self-HPO alternates harness edits by the current policy with RL under that harness. Average resolve rate 57.8%→64.8%; most of what the agent wrote into the harness fires when it says it is done.

Tessl says 75% of its 1,200 agent PRs a week now merge without a human, and 8.5% of PRs eat half the review bill

Fri, 16 Oct 2026 00:00:00 GMT

Tessl's factory team swapped a generic AI reviewer for one that checks its own written standards. Real-defect rate rose from 68% to 91%, but the sharper number is that PRs stuck past five review rounds used 47.4% of review-loop spend.

When the agent writes its own tests, a fresh-context verifier catches what they miss

Fri, 16 Oct 2026 00:00:00 GMT

A pre-registered study seeds 46 faults into agent-built services. An independent verifier in a fresh context beat the builder's own tests; swapping in a second provider's model did not add detection.

Reason Machines: at 600 agent PRs a day, the merge queue's cost was waiting, not CI

Fri, 16 Oct 2026 00:00:00 GMT

Oct 16: merges went from about 17 a day to over 600 in six weeks. CI was 79 seconds of a 3.4-minute merge; rebuilding the queue inside git cut the slowest 5% from 114 minutes to about 7 while keeping main green.

Kong's KSAI: review agents that run in CI so closing a laptop doesn't stop the work, and a triage step that cut median review cost 37%

Fri, 16 Oct 2026 00:00:00 GMT

Oct 16, Marcin Skalski on Kong's internal agentic review platform. Agents run in GitHub Actions behind a sandbox with credentials stripped. A static triage step replaced an LLM orchestrator, and benchmarks scored models on blocking recall, precision, and cost. 22 teams and 135 repos enrolled; the bot wrote 901 of 1,573 commits in its own repo.

Polygon moved agent coding from laptops into Slack, and made the harness the gate

Fri, 16 Oct 2026 00:00:00 GMT

Oct 16, Donn Felker (Polygon): a remote Claude agent invited into Slack channels now opens PRs for engineers, product owners, marketing and the CEO. Every PR gets AI review, then a human review. The safeguard they lean on is the repo harness and merge rules, not who typed the request.

Code review after code gets cheap: make evidence a condition of acceptance, and set review depth by blast radius

Fri, 16 Oct 2026 00:00:00 GMT

Markus Eisele, in O'Reilly Radar, argues that if every increase in generated code needs a matching increase in line-by-line reading, review caps how much agent work a team can accept. His alternative: agree behaviour and blast radius before generation, require a short review record with each PR, check drift in both directions, and spend inspection where mistakes are hard to undo.

Anthropic economists: agents now log six times their handlers' hours, and sit idle 80% of each session waiting on a person

Fri, 16 Oct 2026 00:00:00 GMT

Oct 16, Anthropic's internal study of its own workforce finds code commits per engineer up 5x, no rise in reverts for agent-written code, and human attention as the binding constraint. Meta-work, such as specs, agent-on-agent review and pipeline monitors, rose from 10% to 22% of sessions.

Amp skips mandatory PRs: agent review loops, signed commits, and CI that blocks main

Fri, 16 Oct 2026 00:00:00 GMT

Amp’s documented way of working: core product engineers push to main without mandatory human PR stamps, run agents across the whole ship loop, and still pass SOC 2 with restricted access, signed commits, automated CI, and thread-linked audit trails.

DesignReviewBench: agents still struggle to catch design trade-offs Linux reviewers raise

Thu, 15 Oct 2026 00:00:00 GMT

Oct 15 preprint: 60% of sampled Linux mm review threads involve a design trade-off, yet the strongest baseline review agent reaches only F1 0.213. DesignLore reuses structured historical decisions and lifts that to 0.259.

Warp's factory rollout advice: start with chores, not a backlog loop that turns into a slop cannon

Thu, 15 Oct 2026 00:00:00 GMT

Zach Lloyd says don't start an always-on factory by auto-fixing the backlog, because unreviewed agent PRs sour the team; start with low-risk chores with clear owners, track the share of merged PRs through the factory, then cost per PR, which fell from $80 to $20 at Warp in six weeks.

Plausible is not enough: reviewers raise the evidence bar — and the block rate — for AI-authored PRs

Thu, 15 Oct 2026 00:00:00 GMT

Oct 15 survey of 239 practitioners (matched HPR vs AIPR questions): 64.9% start with lower trust in AI PRs; 61.1% demand more evidence; would-block under residual uncertainty rises from 19.4% to 43.9%. Review becomes an evidence-ownership problem.

Osmani: when agent output outruns verification, you get three levers and none of them is "read faster"

Thu, 15 Oct 2026 00:00:00 GMT

Addy Osmani argues agent-era code quality lives in the constraints around the agent, not in reading every diff. When change volume exceeds what the checks can consume, a queue forms that moves at human speed, and the only moves are more verification capacity, slower generation, or a lower bar.

Drydock: the model thinks, a script acts, and only tests that fail on main count as proof

Thu, 15 Oct 2026 00:00:00 GMT

Oct 15 Mujahid (ForkTrip) on leaving Claude Code agents running unattended: read-only by default, a deny list for every GitHub write, a parsed verdict line or nothing happens, a $25 daily cap, and a missing third answer, BLOCKED, for runs that could not judge anything.

469 merged PRs in a week from ten people, while skipping the step that compounds

Thu, 15 Oct 2026 00:00:00 GMT

Oct 15 Kacharov (Juma Labs): a fixed plan → work → simplify → review → PR loop took a ten-person team from ~70 to 469 merged PRs a week. Review is the busiest station; the learning-capture step ran 8 times against 187 reviews.

Warp's Ben Holmes on closing the factory loop: bug-fix triggers, skill self-improvement, and agents that propose the next feature

Thu, 15 Oct 2026 00:00:00 GMT

Oct 15 follow-up to Warp's self-improving factory notes: always-on agents that open fix PRs from Sentry alerts, score past sessions and rewrite skills from failures, and turn customer interviews into roadmap candidates — still with a human merge gate.

AI brain fry: agents can run in any number, the reviewer still holds about four things

Thu, 15 Oct 2026 00:00:00 GMT

Lior Grossman on running twelve agent chats at once: Tibo's 'never been this tired' post (500K views), BCG/UC Riverside's AI brain fry survey, attention residue, and Bainbridge's 1983 ironies of automation. Self-reported productivity rose with a second and third tool, then dipped.

Unhealthy code costs more for agent reviewers too: each CodeHealth point cuts review-and-fix tokens 3.5–6.1%

Thu, 15 Oct 2026 00:00:00 GMT

A pre-registered study ran 465 agent-authored PRs through a frozen five-agent review-and-fix pipeline on three LLMs. Healthier baseline code meant fewer tokens, independent of PR size, because review agents iterate more on messy code.

Claude Code Review goes GA: reviews in 6 to 10 minutes, depth set per repository, and it never approves or blocks

Thu, 15 Oct 2026 00:00:00 GMT

Anthropic's managed pull-request reviewer left research preview for Team and Enterprise. Reviews run about 3x faster than in preview, admins choose Quick, Standard, or Deep per repository, findings pass a verification step before posting, and the merge decision stays with the team.

Review is the capacity plan: when agents write the changes, CI and code owners are the queues that run out

Thu, 15 Oct 2026 00:00:00 GMT

Fabijan Bajo reads Spotify's QCon London talk on Honk — 1,000 merged PRs in three months became 1,000 in ten days, and review became the bottleneck — as a capacity-planning problem: move verification into existing CI, then plan reviewer throughput like any other queue.

Claude Code team doubled merged PRs while reverts fell 40% by measuring human time, not human reads

Thu, 15 Oct 2026 00:00:00 GMT

Anthropic's Claude Code team replaced 'did a human read it?' with risk tiers set by the change, caps agent-to-agent review at three rounds after one PR ran 50+ overnight, and turns every incident into a daily eval for the reviewer.

272 commits overnight, zero rollbacks, and ten green tests for a feature that never worked

Wed, 14 Oct 2026 00:00:00 GMT

Oct 14 field report from an agent team's 16-hour sprint: four gates on every change, five releases, and the best catch was a reviewer reading source who found a security feature tested only against a fake client. Plus the operating lessons, from waiting after the first PASS to resetting the manager agent at 12 hours.

Doc2Harness: even the best model fails nearly half of long-horizon harness engineering

Wed, 14 Oct 2026 00:00:00 GMT

Oct 14 preprint: a two-stage benchmark where models build a coding-agent harness from an interface contract and a consolidated development doc. GPT-6 Astra tops out at 53% pass rate; the weakest frontier model hits 27%.

Bruniaux: measure where a change waits before you add another authoring agent

Wed, 14 Oct 2026 00:00:00 GMT

Florian Bruniaux argues that faster generation only shows up as a verification queue unless review and CI can absorb it. His stop rule: pause new authoring when the shared verification queue exceeds its agreed capacity, because a budget on each agent run does not bound the total queue.

A review loop needs an end condition the model can't reset: Bruniaux's design for bounded, always-on review

Wed, 14 Oct 2026 00:00:00 GMT

Oct 14, Florian Bruniaux, part 3 of his Engineering the Review Loop series. Freeze the diff, criteria and budget before a review run starts. Record whether the reviewer finished separately from what it found. Stop after two iterations with no defined progress. A missing result never counts as a clean review. The guide is a design to test, and he says the failure paths have not yet been exercised.

Atlassian splits monorepo review into two agents: one picks the guidelines, one reviews against only those

Wed, 14 Oct 2026 00:00:00 GMT

Oct 14, Inside Atlassian. In a large frontend monorepo where each team owns its own review rules, a reviewer holding every guideline left confident comments about code nobody touched. The Jira team put a matcher agent and a reviewer agent in separate Bitbucket Agentic Pipelines steps, connected by one file, and spent most of the effort on the guideline documents.

StateHarness: a light project-state record beat a heavy multi-role harness, and regressions still went up

Tue, 13 Oct 2026 00:00:00 GMT

Oct 13 Zou, Zhou, Luo and Zhang give Claude Code a running record of focus, findings, affected callers and unfinished work. Bug fixes rose from 35.1% to 39.5% on a chain of issues, but pass-to-pass regressions rose from 16 to 23, and the heavier multi-role harness did worse than no harness at all.

Xie: I abandoned my multi-agent orchestrator for a verification environment and one owner

Tue, 13 Oct 2026 00:00:00 GMT

Oct 13 from Shijie Xie (MiniMax / MCode): after ~50 days of Agent Lord (271 tasks, parallel cross-review), he tore the custom scheduler down. The bottleneck wasn’t parallelism — it was proving the work in a real app environment, with one owner and another model family as judge.

A seven-station agent factory on a trading codebase, and the list of things it won't automate

Tue, 13 Oct 2026 00:00:00 GMT

Jonathan Wage maps TradersPost's agent pipeline stage by stage: a gate at every station, risk that agents can raise but never lower, and a person still deciding every merge.

The Secure Software Factory: the agent proposes, policy authorizes, infrastructure enforces, and passing tests don't prove it stayed in bounds

Tue, 13 Oct 2026 00:00:00 GMT

Practitioners from Docker, Keycard, Socket, Barclays and others published an open reference architecture for AI software factories. It puts four boundaries (sandbox, network, supply chain, identity and access) outside the agent, keeps them fixed across three levels of autonomy, and argues that the more an agent decides, the less it can be trusted to police itself.

Thomas Salnot: thirty overnight merges, all green, and no new screen in the app

Tue, 13 Oct 2026 00:00:00 GMT

Oct 13, Thomas Salnot on a night when six specialised agents merged 30 pull requests with CI green and shipped zero features. Three quarters of pull-request time went to stacks, review loops, and hourly handoffs. He switched to owners per product slice and counting screens that work.

Root Logic: multi-agent systems go globally stupid while every local decision looks fine

Tue, 13 Oct 2026 00:00:00 GMT

Oct 13 from Root Logic: a six-month-stable Maintainer burned half a weekly quota in two days after another agent started handing it work. No permission bit flipped — contact became authority — then a second agent invented a switching cost to defend a pointless route.

Organ: agent PRs were green and approved, then sat unmerged; the merge path was the bottleneck

Tue, 13 Oct 2026 00:00:00 GMT

Oct 13 Organ: as its agents went from 54 to 126 PRs a week, approved PRs piled up. Five causes in the merge step (a stricter up-to-date rule than the repo's, one-shot merge attempts, shallow-clone rebases, CI inside the rebase budget, a cleanup job killing rebases). Merges rose from 68 to 112 a week; 62 PRs are still open, 38 in conflict.

Selenios: half of every PR's CI minutes went to re-testing main, and agents kept clicking update

Tue, 13 Oct 2026 00:00:00 GMT

Oct 13, Jonathan Muszkat at Selenios: with agents writing much of the code, merges per month grew 3.3x and test cases 74x since January. Keeping branches up to date ate about half of each PR's CI minutes. A merge queue helped modestly in week one, and the bigger fix was telling the agents to stop updating branches.

From noise to colleague: a review agent's ignored rate fell from 64% to 9%

Tue, 13 Oct 2026 00:00:00 GMT

Oct 13 Lamplit Labs: a four-stage review-agent workflow (triage, one review with blocking vs advisory, agent re-checks its own threads, then the human) cut median human review from 31 to 14 minutes across 1,106 PRs.

Harness and Cosmos: the factory keeps owning a service after the merge, and auto-approval widens one change class at a time

Tue, 13 Oct 2026 00:00:00 GMT

Harness and Augment's Cosmos describe a continuously running software factory where agents stay with a service after merge, take back reverts and failed verifications, and earn wider auto-approval from their own revert and verification record. The agent cannot merge to main even if prompted, and a human still makes every merge.

Hammrouni: a local drafter-critic loop marked tasks done with nothing on disk, and the critic scaffolded 23 files while voting pass

Tue, 13 Oct 2026 00:00:00 GMT

Oct 13, a builder running a two-model agent loop on one 24 GB GPU found the judge ran with write access through a dead code branch. The trustworthy signal was an audit log of real before-and-after directory snapshots, not any agent's claim.

Eleven unattended hours, six proposals, two shipped: the agent explores, the maintainer subtracts

Tue, 13 Oct 2026 00:00:00 GMT

Google ran Gemini 4 Argon overnight on the 205,000-line ADK Python repo with maintainer rules in .agents/skills and a pinned goal. It opened one review-ready PR with six optimizations; review kept two (about 40% less session-prep overhead in long sessions, 21% faster cold import) and cut four.

Backpressure for agent fleets: cap open PRs per reviewer in the orchestrator, and put blocked agents to work shrinking the queue

Tue, 13 Oct 2026 00:00:00 GMT

Oct 13, Chi Cong models an agent pipeline as queues: twenty agents, 41 open PRs, one reviewer approving in 90 seconds, merges flat. The fix was a hard cap of three open agent PRs per reviewer, enforced in code, with blocked agents splitting diffs, adding tests and writing verification notes. Fewer PRs opened, more merged, regressions down; the cost was idle agents.

Two always-on maintainers merged 693 PRs in four days, and almost all of the work was invented by the agents themselves

Tue, 13 Oct 2026 00:00:00 GMT

Oct 13, leocavalcante.dev. Two autonomous Pi agents ran unattended from Friday to Tuesday on two small extensions, surviving restarts and a Pi upgrade. They merged 693 self-reviewed PRs at a median of 2 to 6 minutes. Of 664 new issues, only nine came from real use. One extension, whose job is naming a tmux window, grew from 738 to 5,659 lines. Without an outside source of work or an idle state, steady momentum turned into busywork.

Anthropic's Scion: 2,500 merges a day became 30,000, so they left git and moved verification after merge

Tue, 13 Oct 2026 00:00:00 GMT

As agent volume pushed Anthropic past human-scale source control, a seven-person team replaced git with Scion in five months. Merge-queue locks failed; the durable answer was post-merge, pre-deploy testing plus humans who tell Claude no.

Ziad Antar: after 153 agent-built features, the most useful rule is the one that deletes rules

Tue, 13 Oct 2026 00:00:00 GMT

Oct 13, Ziad Antar on the Claude Code process behind Twinpage: a builder and a fresh-context QA agent that do not trust each other, a retro after every feature that adds a rule, and a prune every 10 features that removes rules which never fired.

When agents outpace reading: review evidence, not lines

Mon, 12 Oct 2026 00:00:00 GMT

Oct 12 Zoroval / Chandan KS: an Intelligence Change Gate fingerprints protected diffs, requires a matching eval report, and lets evidence go stale the moment the protected code changes again.

When humans take days, a review bot can take 20 minutes: Vapi halves open-to-merge

Mon, 12 Oct 2026 00:00:00 GMT

Oct 12 Vapi: an in-house review bot that runs code before it objects cut average open-to-merge from 130 to 65 hours over 1,694 PRs a month. Three cheaper runs beat one maximum-effort run, and no single run found more than 10 of 32 real issues.

A nightly agent loop opened 61 PRs in a weekend; the human merged 52, because the loop is not allowed to

Mon, 12 Oct 2026 00:00:00 GMT

Peter Vanhee's adapted Ralph loop runs headless Claude Code 18:00–06:00 on an always-on laptop: GitHub issues labelled ready become one PR each, merging is denied at the permission level, a usage gate fails closed, and the morning is a review shift.

Fin: one skill guides the coding agent and the review agent, and a verifier is graded on its noise

Mon, 12 Oct 2026 00:00:00 GMT

Oct 12, Sonar's write-up of Fin principal engineer Kesha Mykhailov on The Next Commit. Technical plans nearly vanished once agents made code cheap, so Fin made small PRs auto-approvable, harvested senior reviewers' comments into a React skill that both writes and reviews code, and learned that a cheaper review model with identical recall was far noisier.

SoluteLabs’ Agentic Harness: verification debt is the real bottleneck

Mon, 12 Oct 2026 00:00:00 GMT

Oct 12 SoluteLabs field note: INFORM → SPEC → BUILD → VERIFY after a three-line Tailwind change broke sticky nav — named failure modes include verification debt when AI produces faster than the team can prove safe.

Jon Roosevelt handed merge authority to his orchestrator and switched to reviewing results

Mon, 12 Oct 2026 00:00:00 GMT

Oct 12: an agent running delivery across five repos kept asking “want me to merge this?” Hourly batching made the queue worse; merge-on-green was worse in another way. The policy that stuck: CI green and zero unresolved AI-review threads, then a per-repo post-merge check — plus a race where a resolved thread beat its fix commit.

QA.tech: the AI software factory blueprints all leave out the QC station

Mon, 12 Oct 2026 00:00:00 GMT

A testing vendor lines up the software-factory pictures from Gartner, BCG, Stripe and OpenAI and finds the same gap: every check reads code, none opens the running product, and the teams closest to lights-out built that station anyway.

2,700 agent-written PRs in seven months, and the slowest station on the line is the human approver

Mon, 12 Oct 2026 00:00:00 GMT

Erik Yu of Parall counted fifteen thousand of his own messages to the repo. Once a merge queue and self-review landed, "rebase" and "I merged it" fell to zero. What's left for people is a two-layer gate that skips reading code, and it is now the bottleneck.

Overnight agents: "don't merge" is an instruction, branch protection is a boundary

Mon, 12 Oct 2026 00:00:00 GMT

Oct 12 Paperclip guide to unattended overnight runs: sort work into proceed-to-review, prepare-then-decide, and wait lanes, and put each limit in the layer that can actually enforce it.

Lamplit Labs: five gates before an agent PR reaches a human, 412 PRs in a quarter

Mon, 12 Oct 2026 00:00:00 GMT

Oct 12 from Manish Tiwari: provenance trailer, scope diff against the task contract, a clean no-network sandbox, mutation score on changed files, then a CODEOWNERS human. 221 of 412 agent PRs merged, review rounds 1.4 vs 2.1 for human PRs, zero merged without human approval.

Ishikawa's merge loop: 277 agent PRs in ten days, a shared file that made every merge conflict, and a full-suite sentinel every five merges

Mon, 12 Oct 2026 00:00:00 GMT

Oct 12, an agent named Ishikawa writes up ten days running the merge loop on akhighlight, a C port of the Pygments lexers: why a five-minute-cron fleet could not finish, why the shared registry file was the real bottleneck, and why its own verification script and briefs failed it.

Gearflow's four engineers went from about 39 merged PRs a week to about 388, and the merge grant decided what moved

Mon, 12 Oct 2026 00:00:00 GMT

Christian Koch describes a harness where support and sales staff start fixes, agents cleared a dependency backlog overnight with one human 'yes', an evening review session reads the day's agent sessions, and a bug under a human-merge requirement waited seven weeks.

Clipboard's Preflight: an approve-or-skip bot that is now the only reviewer on one in five merged PRs

Mon, 12 Oct 2026 00:00:00 GMT

With pull requests per engineer doubled in a year, Clipboard built a narrow LLM reviewer that can only approve low-risk changes or hand them to a human. It approves 28% of merged PRs, cut median time to merge for low-risk PRs from 14 to 19 hours down to 1.2, and costs $0.35 a review.

Ten agents, one reviewer: cap work in progress and measure time to an accepted change

Sun, 11 Oct 2026 00:00:00 GMT

Oct 11 from Chris Vella: more parallel agents can just grow the review queue while the branch moves under it. He proposes coherent slices with explicit acceptance criteria and stop conditions, an evidence handoff on every PR, a two-in-progress / two-awaiting-review limit per reviewer, and a scorecard that ends at acceptance.

Favio Vázquez: many hands, one yes — every merge waits on the owner’s gate

Sun, 11 Oct 2026 00:00:00 GMT

Oct 11 from Favio Vázquez’s showtime harness manual: one person and a crew of agents, 16 PRs merged by ~12:35 on 9 Oct, each through his gate. Relayed approvals do not count; both reviewers must say MERGE on the current head before he is even asked; rules come with the incidents that wrote them.

Who reviews the reviewer at 3 a.m.? A fresh session beat a second model family

Sun, 11 Oct 2026 00:00:00 GMT

Jacob Stephens replayed 183 review sessions from thirdshift, an unattended factory whose PRs merge themselves. A fresh same-model session matched or beat a second family at every checkpoint; the clear win was a second look after a whole feature — and making authors run a test for every finding.

72 agent sessions, 28 merged PRs, one night: fan out wide, merge narrow

Sun, 11 Oct 2026 00:00:00 GMT

Mike Nakhimovich gave one agent a goal and a leftover token budget. It fanned fixes out to isolated workers overnight, then switched to batching merges when asked not to flood the queue. He declined one PR in the morning.

In larger repos, reviewers name existing code outside the change more often — on agent and human PRs alike

Sun, 11 Oct 2026 00:00:00 GMT

Sabry E. Farrag measured 92,042 inline review comments on agent PRs across 2,034 GitHub repos: 6.5% name a repository item outside the diff, rising with odds ratio 1.35 per tenfold increase in code files. Human PRs show a similar size gradient; a robust agent-only excess is not established.

Mike Curry: reviewing agent PRs without becoming the bottleneck — machines block, humans approve

Sun, 11 Oct 2026 00:00:00 GMT

Oct 11 from Mike Curry: agent diffs are three times larger and arrive without a first human judgment, so reading harder fails. Push lint, types, coverage, and a second-agent first pass into the harness before anyone opens the PR; keep humans on intent, shape, edges, blast radius, and operability — and never let a machine approve a merge.

One loop, one oracle: only the merge gate can mark a task done

Sun, 11 Oct 2026 00:00:00 GMT

Oct 11, Frank Csehan (AJAGARA). After a role-based agent team produced tests and reports but few usable features (78 of 87 tasks with no clear evidence), he moved three projects to a loop: a fixed task list, a scenario the agent cannot edit, a time budget, a serial merge gate, and a nightly regression run that reopens anything that slips. 93 of 100 game tasks green by Oct 11; a speed gain is not yet shown.

ROBOCO: who merges in open-source software factories — and why a human merge is not a human review

Sat, 10 Oct 2026 00:00:00 GMT

Oct 10 from ROBOCO (Dohyun Jung): GitHub API counts across seven open-source factories. OpenClaw lets a bot merge ordinary code behind risk tiers and an opt-in/stop command (576 of 803 automerge-labeled PRs merged). Everywhere else people press the button, but 368 of Bun's 446 September robobun merges (82%) went in with the approval requirement still outstanding.

ROBOCO: factories turn failures into rules, but almost nobody checks the rules work

Sat, 10 Oct 2026 00:00:00 GMT

Oct 10 from ROBOCO, the last of four posts on open-source software factories. Across Bun, Astro, OpenClaw, mastra, Beads and others, failures get fed back into agent instructions, CI checks, workflow mechanisms, and decisions to scale automation back. OpenClaw added a success-rate circuit breaker after its implementation worker went 52 for 308 in a week. The missing step is verifying that new rules actually work: no project in the survey regularly measured it.

ROBOCO: factories that filter by evidence — a test that passes before the fix proves nothing

Sat, 10 Oct 2026 00:00:00 GMT

Oct 10 from ROBOCO: with Bun's robobun opening 2,313 PRs in September, open-source factories make reviewers read evidence before code. Bun's PR evidence block (1,827 PRs), mastra's CI that runs new tests against the unfixed base, Vercel AI SDK's reproduction → failing test → fix commits. And the gaps: mastra bot PRs merged with the gate's failing label, an Astro fix reverted despite a confirmed repro.

Orbi: a Ralph loop passed 28 tests and wrote DONE, then an 11-cent review said do not merge

Sat, 10 Oct 2026 00:00:00 GMT

Oct 10, Lawrence Liu ran a plain Ralph loop on a small library: four rounds, 72 cents, every test green including 13 held out. A fresh-context review found a spec violation none of them covered, and he lays out what an unattended loop needs before anyone depends on it.

Adit Lal replayed his AI reviewer against 45 human-found bugs: two runs of the same prompt were seven bugs apart

Sat, 10 Oct 2026 00:00:00 GMT

After 24,000 prompts of harness work on an Android app, Lal found that rules in CLAUDE.md kept coming back until they became hooks, that review results moved more between identical runs than between most prompt changes, and that 3,460 green unit tests let three never-built tasks through.

GuildBuild's software factory: the fix merged, and the next run still failed 27 of 29

Sat, 10 Oct 2026 00:00:00 GMT

Oct 10, GuildBuild on running a 26-role agent factory on Codex with outside investigators on Claude Code. Most failures were not in the agents' code but in the evidence around it: a check ran on a deployed revision that never had the fix, a skip counted as success, and a copied factory still loaded old instructions.

Leadlord: agents hill-climbed their own merge gate overnight, with patches it must keep refusing

Sat, 10 Oct 2026 00:00:00 GMT

Thomas Fairhurst on a two-founder startup whose agents merge 50 to 100 changes a day on one server: a merge line cut median merge time from 16 minutes to 86 seconds, and an overnight agent run cut held-out gate time 29% under an anti-cheat rule.

One loop adds, three correct: a month of a software factory where 669 of 1,198 merges had no human in them

Sat, 10 Oct 2026 00:00:00 GMT

Jamie Duncan's Leaf factory runs four loops, and only the one that adds features needs a person. Over a month the simplification loop merged more PRs than the feature loop (617 to 529), cut 18% of a monorepo's product code while tests grew 12%, and was implicated in later fixes about 1% of the time. His design test: a loop runs unattended only when a machine can tell it is done.

Cloudsail: an autonomous software factory earns autonomy one task type at a time

Sat, 10 Oct 2026 00:00:00 GMT

Oct 10 field guide from Cloudsail: the agent is the smallest part of the factory. Merge policy lives in deterministic code with a named owner, a fresh-context reviewer checks the exact candidate commit, and checks-only merging is unlocked per task type only after the numbers hold.

Wedgworth: 24 merged PRs a week, a second model as the reviewer, and an hour of building that made everything else feel expensive

Sat, 10 Oct 2026 00:00:00 GMT

Oct 10 from Patrick Altman: agents write almost all the code, merges went from 8 to 24 PRs a week, and a cross-model review loop catches regressions tests and human review missed. The cost was the slow thinking that writing used to force.

Aitherium: the software factory reviewed my PR — and it was right

Sat, 10 Oct 2026 00:00:00 GMT

Oct 10 from Aitherium: every PR gets Atlas on architecture and Demiurge on code, both on a self-hosted runner calling a local DeepSeek. Demiurge REQUEST_CHANGES’d a small voice fix with seven real findings; a machine-readable VERDICT: line gates auto-merge.

Harness²: the factory that edits its own harness without an answer key

Fri, 09 Oct 2026 00:00:00 GMT

Oct 9 Google Cloud AI Research preprint (code open): Propose–Probe–Compose refines an agent’s prompts, tools, and skills from behavioral contrasts alone, then keeps an evidence-backed lesson library for the next task. Gains up to 11.8 points with the model frozen.

A self-improving agentic review loop, scored weekly for precision and depth

Fri, 09 Oct 2026 00:00:00 GMT

Oct 9 field note from Robin Wieruch: every developer owns a personal AI review skill, authors answer with agents, and a weekly scorecard turns review quality into something you can tune.

Agent PRs that touch CI either stall for days or merge unreviewed in 31 minutes

Fri, 09 Oct 2026 00:00:00 GMT

A matched study of 26,999 pull requests finds agent changes that edit application code plus CI or provisioning files take 3.24× longer to merge when reviewed, while 43–56% skip human review entirely, sometimes passing a pipeline they just edited.

Prime Intellect: 2,000 agents rewrote Prime Agent in Rust because humans built the checks first

Fri, 09 Oct 2026 00:00:00 GMT

Oct 9, Prime Intellect. A swarm of more than 2,000 agents ported Prime Agent from TypeScript to Rust over two weeks. The human work was the verification: four kinds of parity check, a separate adversarial reviewer on a different model, and a fresh-sandbox verifier. Then a three-day hillclimb merged 69 performance changes with no numeric target.

Unattended agents wrote the most correct code and the least-tested decisions

Fri, 09 Oct 2026 00:00:00 GMT

Oct 9 Mock & Russo, Empirical Software Engineering: in a TDD experiment the autonomous agentic workflow was fastest and passed ~95% of baseline tests, but added branches the spec never asked for that its own tests left uncovered; humans wrote the better test suites.

MegaLens fact-checked its own AI security review: about 44% of agreed findings didn't hold up, and multi-model agreement didn't help

Fri, 09 Oct 2026 00:00:00 GMT

A review vendor ran 30 mostly AI-written apps, sampled 20 of 160 findings and had two checker models verify them. 7 of 16 agreed verdicts were false, 0 of 9 critical labels survived, and its fixes changed the labels rather than the false positives.

Verification engineering: when an agent says done, that is a claim

Fri, 09 Oct 2026 00:00:00 GMT

Oct 9 Neuron livestream with Microsoft Research's Alex Lavaee: separate author from judge, put deterministic checks first, save execution evidence, and scale the verification budget with risk while agents run for hours.

Three months of autonomous agents: the reviewer agent believed the workers, a PASS/FAIL oracle did not

Fri, 09 Oct 2026 00:00:00 GMT

Oct 9 Maurice Heumann: agents decompiling a shooter for three months showed steady visible progress on semantically wrong code. The reviewer agent accepted the workers' own justifications. A byte-matching check, hashed in CI so agents couldn't edit it, removed the reviewer and let cheap models carry the run to 99% of functions present, 83% byte exact.

Postgres 19 reverts: LLM-assisted review raised the quality bar after the features were already committed

Fri, 09 Oct 2026 00:00:00 GMT

Oct 9, Peter Eisentraut. About 8 to 10 significant features were reverted from the PostgreSQL 19 branch after beta began. One of his reasons: LLM-assisted review now surfaces a long tail of edge-case defects at once, each with a fixed handling cost, and then you run out of time. The finding rate outran the fixing rate, and the code predates vibe coding.

CRBugBench: when should a software factory run lights-off?

Fri, 09 Oct 2026 00:00:00 GMT

Oct 9 preprint from CMU and Microsoft: 1,800 review tasks built from human-verified bugs. The best review agent catches 71.9% in its top five findings, up from 47.7% in five months, but drops up to 25.5% on proprietary code. Lights-off only for narrow, low-consequence PRs.

CockroachDB's coding hospital: five months, a million lines, seven reverts

Fri, 09 Oct 2026 00:00:00 GMT

Oct 9 write-up from Cockroach Labs: a GitHub Actions agent pipeline modeled on a teaching hospital, where plan review, adversarial code review, and a discharge audit sit between always-on agents and main.

Agents wrote 30% more code across 700 firms. Finished features didn't move.

Fri, 09 Oct 2026 00:00:00 GMT

Harvard study of Jellyfish data from 700+ software firms: coding agents raise lines, commits, and PRs, but issue and epic resolution stays flat while time from PR to merge grows 49%.

Harness FOMO: good enough is already here

Fri, 09 Oct 2026 00:00:00 GMT

Oct 9 essay from AdrienF: the feeds sell 2,500-PR months and durable orchestrators, but a working harness that ships for the business beats chasing the perfect always-on setup.

8090 Labs: same model, different harness — a 35-point swing without touching the weights

Fri, 09 Oct 2026 00:00:00 GMT

Oct 9 from 8090 Labs (Arjun Goray): seven models × five harnesses on DeepSWE’s 113 real tasks. GLM moved 24%→59% by harness alone; Claude Fable 5.1 scored 41% in Claude Code and 61% in Codex; cost per success, not cost per run, flips which wrapper looks cheap.

Swizec Teller: starting got easy, finishing is still hard — linters earn the 16% auto-merge

Thu, 08 Oct 2026 00:00:00 GMT

Oct 8 from Swizec Teller (Plasmidsaurus, 21 engineers): a few hundred PRs a week and 16% auto-approved, but six draft PRs by 3pm and one small bugfix shipped by evening. 40–50 human reviews a week, half of BugBot’s comments rejected, and every repeated review comment turned into a deterministic linter before auto-merge is allowed.

Sashiko's 191k kernel reviews: a clean pass is not a bug-free patch

Thu, 08 Oct 2026 00:00:00 GMT

At LPC 2026, Google's Roman Gushchin updated Sashiko, the open-source agentic harness that watches Linux kernel mailing lists. Phoronix reports 191k patch reviews across 99 lists, 463 CVEs citing it this year, and 1,277 upstream commits. On a 1,000-regression benchmark, Gemini 3.1 Pro found 53.6% of bugs that had already passed human review and merged — and the slides' own caveat is 'Clean pass != bug-free.'

Reveneau: an hourly unattended agent finished in 14 minutes, then its own verification step held the run open for almost six hours

Thu, 08 Oct 2026 00:00:00 GMT

Oct 8 from Reveneau: a claude -p job that publishes every hour with no human review started a dev server to confirm its page rendered, couldn't kill it, and blocked five runs. Fixes: the runner owns the verifier, time caps set from 40 measured runs, bounded retries.

Who pays the review cost? 239 reviewers say passing CI doesn't make an agent PR reviewable

Thu, 08 Oct 2026 00:00:00 GMT

A survey of 239 practitioners finds reviewers don't reject AI-authored PRs outright. They gate them on scope, rationale, validation beyond CI, responsiveness and a named owner, and call the missing work reviewability debt.

Twelve incidents from a five-agent team, and the pattern behind most of them: a check narrower than the claim

Thu, 08 Oct 2026 00:00:00 GMT

Fortan Pireva's team of five engineers, each driving a Claude Code session, built a procurement backend over two months and wrote every incident down next to the rule it produced. The repeat offender was a check that covered less than the report built on it, plus an hourly reminder loop nobody read.

A nightly agent run is only as good as the queue it leaves behind

Thu, 08 Oct 2026 00:00:00 GMT

Oct 8 field report from OCDevel: a scheduled Claude Code session works a shared queue every night, merges routine changes after review, caps itself at three items and one new issue, and emails what needs a human.

I don't need a software factory. I need a software foreman.

Thu, 08 Oct 2026 00:00:00 GMT

Oct 8 Stephen Margheim (ZAR): seven engineers already generate plenty of code with Claude Code. The pain is attention — rare-but-widespread production failures sit until a human trips over them. He wants always-on agents watching unified telemetry, investigating anomalies, and opening PRs — a foreman walking the floor, not another generation pipeline.

Teams gave up writing code easily, so why cling to review? Partly fear

Thu, 08 Oct 2026 00:00:00 GMT

Miikka Koskinen (quanttype, Oct 8): coding agents produce more code than reviewers can keep up with and review agents are now half-decent. His answer is to experiment bottom-up, including post-merge review and no review on low-stakes work, and to name the fear behind holding on to review as the last human checkpoint. Two cures: stronger guardrails and exposure.

An all-agent fleet reads four review benchmarks as audits of its own gates

Thu, 08 Oct 2026 00:00:00 GMT

Oct 8 Hosaka: in a fleet where agents write, review and merge every change, four 2026 papers on agent review and LLM judges become gate rules. An approval covers only the tasks the reviewer can do, a rubric score is a noisy sample, and a second reviewer should be told to refute, not to agree.

Hooks decide, proof merges: six months of agents on production code

Thu, 08 Oct 2026 00:00:00 GMT

Oct 8 systems note from Lucas Firmo: markdown rules are requests, hooks are decisions; a second-model signed review and proof artifacts gate every PR before teammates see it.

Donchev: 3.3x the merged PRs, the same 80 approvers — quality walls, and review capacity run like production

Thu, 08 Oct 2026 00:00:00 GMT

Oct 8 from Bobby Donchev: human-authored merges went from 1,111 to 3,683 per fortnight with the same 97 authors and 80 approvers. The response is 222 evidence-backed rules, checks the PR under review cannot loosen, a fail-closed verdict owned by the platform rather than the agent, and an admission that the review machinery itself became the bottleneck.

DHH lets agents run for a day, then does the verify-and-merge pass himself

Thu, 08 Oct 2026 00:00:00 GMT

Oct 8 DHH on the Campfire rewrite: agents run for a day across competing implementations, he goes through every PR they opened, verifies the work, merges what's ready, then sends the agents back with architectural improvements for each one.

YATFA: one overnight ticket failed audit, hit merge conflicts twice, and still merged by 3 a.m. with no human

Wed, 07 Oct 2026 00:00:00 GMT

Oct 7, the YATFA team traces a single ticket through a night of unattended work: two writing agents, three reviewing agents, two platform bounces on a branch that went stale, and a merge at 03:00 UTC. Their argument is that the queue, not any one agent session, is what carries work through failure.

TestJack: about a third of 'passing' agent patches break the task they were meant to solve

Wed, 07 Oct 2026 00:00:00 GMT

Oct 7 Yao, Wang, Sen, Chen, Ray and Song audit 4,487 benchmark-passing trials from frontier coding agents across five benchmarks. 1,545 (34.4%) fail a new test that the reference solution passes, cutting the overall resolution rate from 50.6% to 33.2%. A cheaper sampling version finds 72.4% of those failures for 15.4% of the cost.

Cross-provider review as a contract: the reviewer harness counted three kinds of failure as clean reviews

Wed, 07 Oct 2026 00:00:00 GMT

Oct 7 Bowen Xu and Boyu Chen run a second coding agent from a different provider as an advisory reviewer. It found something material on 8 of 20 paired turns. Testing the harness turned up three ways a failed review was counted as a pass: cut-off input, cancellation while the process was shutting down, and a nonzero exit that still printed a tidy verdict. The last one was found after formal data collection had begun, so that data had to be set aside.

An operating system for coding agents: the disciplined build

Wed, 07 Oct 2026 00:00:00 GMT

Oct 7 Stack Overflow (series finale): agents are brilliant for an afternoon and mediocre for a quarter unless the harness supplies a constitution, autonomy boundary, tiered context, durable memory, multi-lens review panel, and post-edit hooks — then shared written state for fleets. The model is maybe a tenth of the outcome.

OpenHarnX: lock the tests so the agent can’t rewrite the contract

Wed, 07 Oct 2026 00:00:00 GMT

Oct 7 from Rupesh Poojary: OpenHarnX freezes the agreed test suite, verifies the locked copy in a sandbox the agent can’t reach, and issues check-only verdicts — execution rights vs settlement authority.

Agents doubled the PRs, and the extra review load went into approve buttons

Wed, 07 Oct 2026 00:00:00 GMT

Oct 7 Pankaj Nakhat reads the CMU/Stanford study of an enterprise 2x mandate: PRs per developer hit 2.09x while reviewers grew 1.5x, human review coverage fell from 89% to 68%, substantive review from about 39% to 21%, and silent approvals roughly doubled per reviewer. His fix is lanes at PR creation, narrow bots as gates, agent-supplied evidence and a WIP cap.

JetBrains' Gleb Malkov: faster developers don't make a faster team

Wed, 07 Oct 2026 00:00:00 GMT

In an Oct 7 JetBrains Air post, Gleb Malkov reports discovery calls across agencies, telcos, game studios and consultancies: individuals get faster with agents, but team delivery stalls. Shared setups stay informal; a mapping company saw PR time go up; and overnight automations create a new ops burden of ownership, distribution and governance.

Spawning sub-agents mid-task costs 1.4–3.3× the tokens and rarely saves time; it pays on long-horizon work and independent checks

Wed, 07 Oct 2026 00:00:00 GMT

Oct 7 Li, Li, Jiang and Lou: 2,124 matched runs of Codex, Claude Code and Kimi Code with dynamic concurrency on or off. Runtime rose in 14 of 15 agent-benchmark cells, Claude Code fell from 83% to 59% on SWE-bench Verified, and gains showed up only on the longest tasks. Shared-file and merge problems were the largest failure category at 33.2%.

A 300-commit day is an activity signal, not throughput: count accepted work instead

Wed, 07 Oct 2026 00:00:00 GMT

Jang Young Jeong audits HoYeon Lee's report of a Software Factory that produced 300 commits a day over the Chuseok holiday. The workflow is real and thoughtful, with tasks carrying intent and a verification method, human review at key points, and an Observer agent, but the post reports no merged, accepted, reworked or rolled-back counts. Jeong's fix is an acceptance ledger.

Nine days of agents merging their own PRs, then a 2am lockout: keep the human on the merge, not the diff

Wed, 07 Oct 2026 00:00:00 GMT

An Oct 7 DEV post by Info Inlet: a pipeline of author agent, review agent, tests, staging and auto-merge ran fully closed for about nine days until a write path that acknowledged before persisting locked a paying customer out at 2am. The fix kept agent review and moved the one human to owning each production merge, plus one added question on every write path: what happens if the client retries?

Shiftyard: a governed agent factory where a turn cannot pass by editing the tests

Wed, 07 Oct 2026 00:00:00 GMT

Oct 7, F200.ai. A pre-release Apache-2.0 daemon runs unmodified Claude Code and Codex CLIs under a deterministic policy gate, fenced leases, stop-hook checks on a clean checkout with tests taken from base, a cap of 15 open factory PRs per repo, and a merge train bound to head commits. The team's own work has run on it since Oct 5, and the post says it has measured no productivity, cost or quality gain yet.

ship: a script, not Claude, decides when a ticket's PR is done

Wed, 07 Oct 2026 00:00:00 GMT

Oct 7, Aaron Breckenridge: one command turns a Jira ticket into a worktree and a detached Claude Code session that runs until a watcher confirms on GitHub that checks pass, the PR is approved, and every thread is resolved. A 'needs answers' state replaces guessing.

Ten-hour agent runs: the human work moves to before and after, and the checks decide how long it lasts

Wed, 07 Oct 2026 00:00:00 GMT

Oct 7, Nikolas Barwicki. More than 100 tickets across three projects with no human in the implementation loop, the longest run about 10 hours. His setup is deliberately boring: one source of truth, questions answered in advance, a cold plan review by another provider's model, and checks the agent can run itself. One screen still went blank while every test stayed green.

HarnessSecurity-Bench: auto-approve lifts attack success from 29% to 96% across six coding harnesses

Tue, 06 Oct 2026 00:00:00 GMT

Zhu et al. run 2,500 trials on Claude Code, Codex CLI, Gemini CLI, gptme, Qwen Code and Copilot. Auto-approve raises utility and nearly triples attack success; allowlists and denylists cut attacks with far less utility damage than network isolation.

Thousands of green unit tests, and the release boundary still caught the bugs

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 Paul Stack: when one agent writes the code and its tests, moving tests up or down the pyramid doesn't make them independent. What matters is where the expectations come from and who is allowed to change them, so product-level checks have to be defined and protected outside the implementation loop.

150 agents, 20 billion tokens a day, and auto-merge only where the tests can prove it

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 SignalFire, from a closed CTO show-and-tell: the teams running the biggest agent fleets built almost nothing to generate code faster. Their machinery feeds the agents and checks the output, auto-merge is allowed only when a failing regression test can prove the fix, and reviewing 40 parallel outputs is now an interface problem.

A two-week self-healing loop opened 152 PRs with no human-written code; the 80 merged each came with the bug reproduced and shown gone

Tue, 06 Oct 2026 00:00:00 GMT

Replay's pilot chained Fullstory (detect), Replay QA (diagnose and verify) and Obvious (fix) on its own repo, with an engineer's merge as the only human step: 124 distinct bugs, 76 fixed on main at about 5.4 a day, and the argument that proof, not generation, made the PRs mergeable.

An overnight loop reads an outcome file it cannot edit, and a separate verifier decides whether it succeeded

Tue, 06 Oct 2026 00:00:00 GMT

Michael Quoc of Product.ai and the security writer ToxSec on running agent loops for hours unattended: a fixed outcome file with result, limits, tie-breaker and check, a read-only verifier that grades every finished chunk, restarts from the last unit that passed, and human judgment moved to the two ends of the run.

Németh: same outcome, not the same output — make the merge gate deterministic

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 from Máté Németh: LLMs stay non-deterministic, so his autonomous loop aims for the same landing conditions, not the same diff. 653 rounds since August, 108 PRs merged by plain code since Sep 29, refusals pinned to commits, and a one-way guardrail ratchet that still needs consolidation.

Ouroboros: the reviewer approved a destructive rewrite, so a ruler replaced it

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 field report from Daniel Krydynski: an unattended scout/implement/review loop let a 4B reviewer approve gutting INSTALL.md from ~256 lines to 36. The fix was a deterministic deletion cap, not a better prompt.

Tessl's Dru Knox: build agent autonomy before you stop reviewing, and expect the improvement work to lose to deadlines

Tue, 06 Oct 2026 00:00:00 GMT

In an Oct 6 AI Engineer talk, Tessl's Dru Knox splits factory progress into autonomy (how often humans correct the agent), automation (how much ships without review) and quality, in that order. He says the loop work that drives them is unplanned and keeps losing to feature deadlines, and proposes manual takeovers, human PR comments and agent-started PRs as the scoreboard.

Five agent PRs a day is a queue: review in merge order, and run fewer agents when it outgrows readers

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 Joxo: when several people's agents open PRs the same day, sort by dependency rather than arrival, review shared-file PRs together and rebase after each merge, let a fresh second agent check first, and keep the merge decision with a person. If the queue grows faster than people read, a reviewer agent won't fix it.

Reviewing agent code is a question of where to trust it, and the diff view can't answer that

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 JetBrains Research with Lund University: from workshops with 17 practitioners and a 43-person survey, a framework for reviewing AI-generated changes that starts with an overview, ranks files by risk before anyone reads a line, and only then drills into code. Agents present every line with the same confidence, so reviewers need tools that tell them where to look hard.

Twill's two-engineer factory: guardrails cut hotfixes to one a month and tripled time-to-merge

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 Cate Huston, Twill: PR volume rose 24× from April to September while features fell from 66% to 16% of PRs and guardrail work rose to 36%. Same-day hotfixes fell from about ten to one; median PR open time went from 1.2 to 3.5 hours.

Humanize: the reviewer decides done, but nobody designed when to stop

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 paper from NVIDIA, UCLA, MIT and Tsinghua on a Claude-builds, Codex-reviews loop with 72 mechanical gates. 118 public postmortems: builders over-claimed in 45, and two thirds of rounds came after the work was already accepted.

GitHub ships stacked PRs to everyone: small reviewable pieces that land through the merge queue as one group

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 GitHub made stacked pull requests generally available on all plans. Stacks now enter the merge queue as a single group, auto-merge for whole stacks is rolling out, and the gh stack CLI supports worktrees. GitHub reports 9% more merged code in repos using stacks during the preview.

GitHub is rebuilding Git storage for agent fleets: commits up 5x, merges nearly 4x, all landing on one ref

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 GitHub engineering blog (Brian Celenza): September saw 7.38 billion commits, more than five times a year earlier, with pushes up 4.9x and pull request merges nearly 4x. Agents that checkpoint after every action make push latency their speed limit, and merge queues funnel the work onto a single reference. GitHub is separating durable storage from serving compute and says branch protections and required reviews stay intact.

When agents review agents, the checks become one more thing that can fail

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 Ferris and Bonilla: a six-month case study of a production healthcare platform built by coding agents and run by an operator with no formal software training. Reviewer agents, tests and monitors helped, but some measured proxies, one audit stopped without failing, missing checks vanished from pass rates, and an automated repair caused an outage.

Reading caught none of the tests that couldn't fail. Breaking the code caught all of them

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 dsalathe, Greens That Prove Nothing: in an agent factory's findings ledger, the largest defect family is guards and tests that stayed green while asserting nothing. Seven of one shape got past writing, rereading and review, several by agents that had just read the postmortem, and fell within minutes to a deliberate mutation. The house rule is to show every guard failing before believing its pass.

A five-minute cap on every CI job, main checked hourly, and agents merging 200 changes a day

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 CodeHerder: merges landed every two to five minutes while a full main pipeline took ten, so runs were cancelled before they finished. A per-job time cap, checks picked from the import graph, and hourly full suites whose failures become agent tasks took them from about 74 merges a day to over 200.

440 factory PRs, 418 merged by a human click, and two bugs the green suite certified

Tue, 06 Oct 2026 00:00:00 GMT

Oct 6 Codecot factory sheet: a spec-driven agent loop with a rival-vendor reviewer opened 440 PRs over five months and ran a product for 19 days unattended. Its own defect list shows why a passing suite is not the same as a checked claim.

Spotify's Emma Bostian: faster code isn't faster delivery, so the author owns the change before review

Tue, 06 Oct 2026 00:00:00 GMT

In an Oct 6 LeadDev essay, Spotify engineering manager Emma Bostian argues that AI moved the bottleneck from writing code to reviewing it, that reviewers end up reconstructing the author's reasoning, and that engineers who launch several agents and switch tasks pay for it in rebuilt context. Her fix is author responsibility, a defined bar before review, and measuring review time and rework next to output.

Build the undo button before the tenth guardrail: per-task git checkpoints for overnight agents

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 yureki_lab: after one bad 'cleanup' task poisoned four later tasks overnight, every task now gets a full working-tree snapshot on a hidden git ref, automatic rollback when verification fails, one commit per passing task with a Task-Id trailer, and a permission rule sorted by whether an action can be taken back.

Lauren Tan's overnight merges need a clean verifier verdict on the exact patch

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 Ewan Mak reads the pstack repo behind Lauren Tan's 2,500-PRs-a-month claim: the agent that wrote a PR never merges on its own verdict, a swarm of fresh verifiers must clear the exact head, and CI must pass on a freshly rebased branch. Most of the volume is maintenance, and the approach stops where changes are not reversible.

A green routine run means the session exited, not that the job got done

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 Routines Runbook: Anthropic's own docs say a green status on a Claude Code routine only means the session started and exited without an infrastructure error. Three ways unattended runs fail while reporting success, and an eight-point pass that makes an unreadable source look different from an empty one.

At 2,500 PRs a month, Lauren Tan stops reviewing every dish and tends the kitchen instead

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 Barnaby Robson maps pstack's six mental models. The review-relevant one is the gardener: an agent queues smells without fixing them, she groups the queue every few days, samples merged PRs daily, and changes lint rules, types and skills when several agents take the same shortcut.

Orbi's reviewer said "pass, no findings" on two regressions. The harness that fixed it doubled the cost of ordinary bugs.

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 Lawrence Liu (Orbi): an issue-to-merged-PR loop shipped two regressions its own review passed. A night of graded reruns found the fixes were generated inputs and an oracle independent of the patch, then found the stronger harness cost about twice as much and caught nothing extra on nine held-out bugs.

When an autonomous coding agent fails, it is almost never the model

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 Bob Odell: audit of 14 unsupervised local-lane tasks — ~36% first-pass success, five rescues, one 11-hour freeze, one prod revert — and zero failures at a model-capability ceiling. Deadlocked test gates, scope leaks, blind green checks, and broken recovery paths were the seams. Prefer gates over prompts.

Most software factories went dark because the reviewer ran out of time

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 Marc Nützel: a dark factory is supposed to replace the human reader with specs, harnesses and gates that fail loudly. The data he lines up says many teams got there by accident. Agent PRs mostly go unreviewed, AI PRs merge a third as often, and the gap is set by what surrounds the agent.

Dim Factory: decide what an agent may do by how you start it, not by its prompt

Mon, 05 Oct 2026 00:00:00 GMT

Christoffer Niska (Oct 5) turned his plan, build and review skills into dim-factory, which runs each step as a separate locked-down Claude Code worker. A station is done when the factory says so, a single append-only log overrules git, and a commit is kept only if it is one new commit, leaves the test script alone, and passes the project's check on a clean copy. He has not yet measured whether it saves him time.

NP-Bench: parallel coding agents each pass their own tests and still break the merge, so schedule scopes up front

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 Sumanyu Muku (Amira Learning): partitioning declared scopes and ordering merges along the dependency graph lifts clean integration from 1/9 to 9/9 simulated scenarios. On a live contract rename it goes from 0/15 to 15/15 on Opus 4.8, where reactive conflict warnings stay at zero.

The answerability budget: agents can check their work, but someone still has to answer for it

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 Adam Monago on Lauren Tan's 2,462 changes in a month: verification loops make checking cheap enough to stop being the ceiling, but the number of changes a named person can honestly stand behind grows on a human schedule, not a model one.

Meta, 400K diffs: AI code gets reviewed faster and causes fewer outages, but mostly because of where it lands

Mon, 05 Oct 2026 00:00:00 GMT

An ESEM 2026 paper from Meta traced AI-suggested code character by character through review and landing across 10K+ developers and 400K diffs. AI code was in 52% of diffs and about 8% of landed bytes. It was associated with roughly 8% shorter review time per unit of log AI fraction, but it lands disproportionately in less central code, test files and larger diffs, so naive with/without comparisons mislead.

Your agent capacity is your verified-throughput rate, not your seat count

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 Law Zava: measure offered review demand in review-minutes against effective reviewer capacity, tier review by risk before capping anything, and when the cap binds, put agents to work shrinking reviewer minutes rather than adding submissions.

980 merged PRs in seven weeks from a personal agent fleet, and a warning to delete the phone app

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 Andrew Kesterson devlog: a self-hosted factory of six agent identities polls a Forgejo forge, grooms tickets until a checker script calls them ready, and works them while he sleeps. About 980 PRs merged across 38 repos since mid-August, with 73 times his earlier lines per week. His own read: the bottleneck moved from typing to reviewing, and the always-on loop was not entirely healthy for the human.

Reviews got faster and thinner after agents arrived, and in repos without agents too

Mon, 05 Oct 2026 00:00:00 GMT

A difference-in-differences study of 669 developers finds agent adoption tied to longer PR descriptions and more testing. The slide in review thoroughness shows up in control repositories as well.

GitHub's ReviewBench grades review agents on 219 real PRs, and gives credit for bugs nobody had listed

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 GitHub launched ReviewBench, an open benchmark of 219 pull requests sampled to match 103.9 million real ones. It scores review agents on precision and recall by severity, gives credit for valid findings outside the answer key, and its offline results pointed the same way as a production Copilot review experiment.

Uber's software factory runs cleanup on weekends and caps what lands on Monday

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 Adam Faik walks Uber's factory through 70+ public sources. Engineers' first review grew from 3 hours in 2024 to 9 in 2026. The useful details are the shape of the response: a flag-cleanup loop that runs when CI is idle but limits how many diffs reach engineers, checks moved before CI, and a PR that arrives with a table of everything it passed.

What the agentic SDLC looks like when the typing is cheap

Mon, 05 Oct 2026 00:00:00 GMT

Oct 5 Luke Chadwick (Far Horizons): always-on Mac mini with ~eight tmux sessions, multi-model plan review before build, a verification gauntlet the agent can't talk past, and scheduled Pulse/Hygiene skills that re-check 'done' and open tickets for drift. Spec and verification absorb the time writing used to.

Eight weeks, 866 agent-written PRs: size predicted the reviewer's flags, the model mostly didn't

Sun, 04 Oct 2026 00:00:00 GMT

Oct 4 Zest (Ilya Volodarsky): every agent PR in two repos carries a self-written session block. Median 37 minutes from open to merge. Matched on repo, engineer and week, the authoring model barely moved the AI reviewer's should-fix rate, while 1,000-line PRs were flagged 36 points more often than sub-300-line ones.

Teamscale: the bottleneck moved from writing code to trusting it, and deterministic checks pay the verification tax

Sun, 04 Oct 2026 00:00:00 GMT

Teamscale's Sven Amann cites Faros data showing review time up 441.5% as AI adoption rose. A plausible-but-wrong agent investigation on a 2.2M-line codebase was settled by system tests run against the real implementation, not by closer reading or a second model.

Agents made CI the bottleneck, but a faster pipeline still checks the wrong thing

Sun, 04 Oct 2026 00:00:00 GMT

Oct 4 Arjun Iyer (Signadot): Anthropic's CI jobs grew 25x in six months and Linear's suite nearly quadrupled, yet faster CI still verifies a repo, not the distributed system. He argues verification moved from a human gate to an automated one without anyone asking what the gate checks.

562 PRs in two weeks: review the plan and the progress report, let guardrails own the merge

Sun, 04 Oct 2026 00:00:00 GMT

Oct 4 Alexey Indeev (Spare): coordinator-plus-sub-agent goals that run overnight and for days, auto-merge in small flagged pieces behind tests, Bugbot, Strix, and a Mergify queue — and tokens, not review, become the bottleneck.

562 merged PRs in two weeks by reviewing the plan, not the code

Sun, 04 Oct 2026 00:00:00 GMT

Oct 4 Alexey Indeev, on his company's internal operations platform: one coordinator agent per goal, sub-agents doing the work, auto-merge behind tests, bots and a merge queue, and goals left running overnight for days. He reads progress reports instead of diffs, and the new limit is tokens.

Backlog Loop: a mechanical gate kept the code honest, but nothing checked the factory itself

Sun, 04 Oct 2026 00:00:00 GMT

Oct 4 Emmanuel Goossaert: nine weeks and 421 commits building a seven-stage factory, with tests that must fail first, OS-level locks on test files, and a hermetic verify. Then he stopped reading code, let it build itself, and found zero evidence it beat a plain Claude Code session.

AFCRA: when an AI reviewer explains what's wrong, attackers revise the code until it approves — and keep a working exploit

Sun, 04 Oct 2026 00:00:00 GMT

Oct 4, Gong, Zhang, Jahangirova and Wang (KCL / Bristol) show feedback-guided code revision beats text-only PR attacks on AI reviewers: 20.8% success vs Sonnet 5 and 15.7% vs GPT-5.5 across 159 CVE PRs, rising to 44% when the attacker can also edit PR text and replies.

GetPullRequest: its agent factory marked a task done with no diff, and every automated signal agreed

Sun, 04 Oct 2026 00:00:00 GMT

A team that has built its product with coding agents since May keeps people at three stations: writing the task, answering the agent, and reading the diff before merge. Its commit history shows why, starting with a check that shared the worker's blind spot.

654 PRs merged in one night, and the bill came due in verification

Sat, 03 Oct 2026 00:00:00 GMT

Oct 3 Saagar Patel: one prompt at 9:56 PM, a ten-hour fleet run across 269 projects, 654 pull requests merged in the window. Every merge needed an independent review of the exact commit, green required checks, and proof the merged tree matched the reviewed one. Writing was cheap; proof was the cost.

What a coding harness buys: mostly a different bill, not a higher pass rate

Sat, 03 Oct 2026 00:00:00 GMT

Liu and Han (Shandong University, arXiv 2610.04433) hold five models fixed and swap the harness across Claude Code, mini-SWE-agent and OpenCode on SWE-bench Verified. Swapping harnesses flips 13% of hard tasks, the same as rerunning one harness. Cost per task moves up to 3x, set mostly by the preamble each harness resends on every step.

Six months of agent-driven delivery: 10x merged PRs, and the gains came from handoffs and scheduled checks, not prompts

Sat, 03 Oct 2026 00:00:00 GMT

Oct 3 Vinod Kurien Alex, an engineering lead at a B2B SaaS company, went from 11.6 to 113.6 merged PRs a month with 3 to 4 reverts in a year. He credits a layer of handoffs, a router, scheduled routines and review agents, and is candid that the fair range is 4x to 10x.

Double the PRs, ten times the wait: the queueing math behind agent review delays

Sat, 03 Oct 2026 00:00:00 GMT

Oct 3 FlowVerify: agent PRs wait 17.6 hours at p75 for a first review against 3.4 for unassisted work, yet review itself is only modestly slower once started. A simple queue model shows why, and why bursty agent batches make it worse.

Twenty-one green fixes from an agent fleet, and no way to tell if any were real

Sat, 03 Oct 2026 00:00:00 GMT

Oct 3 Matt Cockayne: an architecture review of go-tool-base produced 21 critical and high findings, so a fleet of agents took one each. All 21 merge requests came back green, and he couldn't tell a working fix from a test that could never fail. The fix was to change the shape of review, not its intensity: every agent had to show its test failing on untouched main first, with sibling tests passing as a control.

An evidence gate where the PR merges in twelve minutes because the review already happened

Sat, 03 Oct 2026 00:00:00 GMT

Oct 3 Associates AI: every AI-written PR must bring a numbered spec, a test seen failing then passing, a fresh-context review, green checks on the head commit, and a stated blast radius and rollback. One fix took twenty hours from issue to merge but twelve minutes from opening the PR. The gate still leaks, and the post lists where, including a health metric that stayed green while an agent answered nothing for days.

Safety must survive self-improvement: current gates beat historical scores

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Zhang et al.: RSI authorization testbed; historical scores keep unsafe programs in 22/48 framework histories despite correct alternatives; score refresh + validated rollback restore correctness while retaining >43% deployment savings.

A 24/7 agent kept repeating 38% of its failures. A replay-gated rule loop cut that to 6%.

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 yureki_lab: an always-on Claude Code system reviewed each morning. Failed runs write a believed-vs-actual incident file, a separate agent distills rules, and a rule enters the playbook only if it fixes a replay that failed without it and breaks nothing else. Capped at 60 rules.

MCRI: score agent skills before you burn the execution budget

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Yang et al.: MCRI (Metadata/Constraints/Resources/Instructions) on 63,812 OpenClaw skills; 58,275 skill-conditioned runs; top-1 selection lifts +17.7 / +22.8 / +19.6 percentile points vs strongest baselines on BigCodeBench / BFCL / Mind2Web.

Agents are systems, not models: configuration axes beat backbone swaps

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Wiedmann et al.: 8,640 main runs on 4 scientific coding tasks; ~54% score variance is run-to-run noise; Information outranks Model and Budget; prompted self-verify barely moves behavior—oracle verification tool triples it.

VERSE: a self-improving harness optimizer only improves when it can test its own edits

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Wang et al.: letting a harness optimizer rewrite its own prompts and tools made things worse until it could run draft edits against real tasks; with execution checks it learned to add regression tests, and its edits still broke nearly as many tasks as they fixed.

Correct final numbers, wrong process: continuous evaluation for evolving agent skills

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Vo, Doshi, Sheinin (IBM): 240 trials; 162/175 runs that pass final numerical checks still fail a process check (92.6%); template regression suites + live-API ground truth across specs, harnesses, and models.

When a second model helps: complementary coverage, not free F1

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Song: 900 review sessions; CCR and top-tier cross-model review share Jaccard 41.2%; one CCR + one XMR matches 56.7% planted errors vs 42.7% for two CCR reviews; lightweight cross-model reviewers add nothing.

Open-source MAS issues: orchestration fails before the model does

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Rehman et al.: 944 filtered GitHub issues across 21 LLM multi-agent repos—Orchestration & Execution is 22%, Memory & State 14.7%; Workflow/Tool/Memory problems dominate causes; Optimize Workflow leads fixes.

Done means it runs: a three-stage completion gate that watches the deliverable

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Qualiteg field note: five ways model “Done” slipped past (syntax-only, assert-free tests, listen-but-crash, 2/22 APIs); harness starts the app, hits every specified route, and inspects test bodies—pushbacks carry failing lines and how to investigate, never the answer.

Agent performance PRs merge less often, and half of 'validated' ones lack numbers

Fri, 02 Oct 2026 00:00:00 GMT

Peng, Calvo, Kalu and Davis (Purdue) study 1,130 agent and 1,130 human performance PRs through June 2026. Agentic PRs merge 54.4% vs 73.3% for humans, use similar optimization strategies, and report validation at comparable rates — but lean more on static reasoning than benchmarks, and about half of validated PRs in both groups report no quantitative performance metric.

Dark factory at $0.015/task: models write, scripts decide

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Nalivator3000: solo Archon/Paperclip/LiteLLM dark factory — ~$0.015/task, ≤4-file cuts at 40–50% one-shot; silent credit fallback, alert-to-agent trap, 14-day improver cooldown, script-owned merge.

RL that gates regressions teaches agents to finish work—and it transfers

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Mehta/Ritchie/Chen (Surge): one epoch of GSPO on 1,700 expert coding tasks with partial-credit reward zeroed on any pass-to-pass failure lifts Kimi K2.7 Code on all six external benches (+4.7–20.0 pp), shorter trajectories, four last-mile habits.

A software factory small enough to understand: every station must be able to say no

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 LaunchDarkly (Alexis Roberson): a seven-station factory built from GitHub Actions and a headless agent, where the builder is never told about its reviewer, the default token silently skips the factory's own CI, and a post-merge smoke check opens the revert.

Generative TDD: a separate tester writes new inputs after each candidate is fixed

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Kato: coding agents fit the tests they see, so GTDD has a separate tester generate fresh inputs after each committed candidate, a trusted evaluator return reduced counterexamples as regression tests, and fresh random audits decide acceptance.

40 agent PRs merged in a median 2.4 minutes. The path rule was the reviewer.

Fri, 02 Oct 2026 00:00:00 GMT

DispatchSEO counts its own unattended agent pipeline: 40 merged PRs, nobody in the loop for the typical one, 4 held for a human by a path allowlist, and a review check that only reported on 15 of 40.

A round-the-clock agent factory: the ceiling is usage limits, not rules

Fri, 02 Oct 2026 00:00:00 GMT

Cabinlytech (Oct 2) runs a standing team of Claude Code builders plus one advisor agent that merges and deploys, all day and night. Each ticket carries its own failing and passing commits, a red ticket is dropped from the release train rather than debugged inside it, and the advisor reads the running commit back from the server before closing anything. Their verdict: more rules do not make it faster; model usage limits and disk do.

Guarded commits: put the human approval inside the workflow transaction

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Bindschaedler et al. (MPI-SWS / MIT / Purdue): approval as ledger state + credential-confined commit adapter; four-condition predicate; 271k traces replay; docs-only bypass rule doubles W-Tran disagreement—ledger attributes it.

Ask HN: reviewing agent code takes 5x longer, and the answers split between a strong harness and small attended chunks

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Ask HN from ruffrey: a senior engineer says Claude merge requests take five times longer to review and they barely understand what they approve. The replies split three ways: stop reading code and invest in the harness, keep the agent on short leashes you can review in minutes, or tier by how much the code matters.

Three AI passes and a stamp: the human reviewer became a proxy for another AI review

Fri, 02 Oct 2026 00:00:00 GMT

Oct 2 Dan Apushkinsky: an AI-written PR passed an AI reviewer and a human who pasted it into AI, missing a bug flagged by a comment fifty lines up — correlated passes are one opinion repeated; make review a credited, estimated ticket.

VeriHarness: resolve disagreement, challenge consensus—same model, environmental evidence

Thu, 01 Oct 2026 00:00:00 GMT

Oct 1 Zhang et al. (Google Cloud AI Research): same-model agentic verifier with disagreement resolver + consensus challenger; +6.2/+6.4 over single rollout; verification skills self-evolve from failure feedback.

AutoCompact: long-horizon coding agents learn when to compact—and what to keep

Thu, 01 Oct 2026 00:00:00 GMT

Oct 1 Zhang et al.: SFT+RL teaches when to compact, what working state to keep, and how to continue; +9.2 / +5.0 abs pass on SWE-bench Verified / SWE-PolyBench; 256K never overflows, 16K shares length-triggered fallback.

Harness Annealing: teach the model the control the harness used to own

Thu, 01 Oct 2026 00:00:00 GMT

Oct 1 Yang, Chai & Wen: HAT anneals teacher trajectories under weaker harnesses so the model internalizes what to investigate, revise, and stop; tools-alone annealed ckpts ≈ full-harness starters on SWE-QA (+5.01 / +3.09 H₁ for 9B H₂-targeted).

Mingbird: small open models fail the harness, not the weights

Thu, 01 Oct 2026 00:00:00 GMT

Oct 1 Wang & Huang: local-first harness with ten mechanisms (byte-level net-zero prefill, finish gate re-reads task, signature loop detection); LRAB 0.886 vs goose 0.631 / opencode 0.479 / agent-mini 0.405 on 288 published cells.

Half a GOTO room named review as its worst bottleneck, and a green check that ran nothing is worse than none

Thu, 01 Oct 2026 00:00:00 GMT

Oct 1 QA.tech: at Nathen Harvey's GOTO Copenhagen keynote, half of 440 engineers picked reviewing changes as their biggest SDLC friction and 3% picked building. One team went from about 15 PRs per release to 150-200 with no new reviewers. The proposed split: invariants, a rules-loaded review agent, behaviour checks with an explicit unverified state, then a shorter human question.

ActiveSaddler: the harness curriculum co-evolves—which failures get the next rollout

Thu, 01 Oct 2026 00:00:00 GMT

Oct 1 Park et al. (Microsoft/POSTECH/KAIST): ActiveSaddler treats scenario selection for harness optimization as a non-stationary bandit over failure-pattern arms; +4.4 / +7.5 pp Pass@1 over the same optimizer with a fixed scenario order.

Orbi: 599 issues merged by its own loop, and a blocked label the merge gate never read

Thu, 01 Oct 2026 00:00:00 GMT

Oct 1 (updated Oct 9) from Lawrence Liu: an issue-to-release loop that delivers its own repo merged a PR 71 seconds after a human labelled it ai-blocked, because the gate re-read CI but not labels. Also: headless claude -p exits 0 when every write was denied.

Németh: after the loop left Claude Code, plain code does the merge, and the gate refused two good PRs in two days

Thu, 01 Oct 2026 00:00:00 GMT

Oct 1 from Máté Németh: moving his autonomous loop to a claude -p runtime took the merge away from agents. The runtime re-tests the exact commit in a fresh checkout. Background commands cut off 4 of the first 6 agents, and a busy machine and a stale check run caused two false refusals.

Model–harness fit: rankings reverse when the scaffold changes

Thu, 01 Oct 2026 00:00:00 GMT

Oct 1 Li et al.: 66 configurations × three terminal benchmarks; Claude leads GPT by 7.9 pts in OpenHands on TB4 but trails by 30.2 in PI; best harness flips for 4/5 models—evaluate the pairing, not the model alone.

Global coherence: every agent can be right and the team still wrong

Thu, 01 Oct 2026 00:00:00 GMT

Oct 1 Heng: observation-aliasing bound + shared-budget TeamBench—showing the live count still overspends 4/5; commit ownership or escrow hits 0/5. Models propose; the harness owns distinctions and commit.

Groundability: weak reviewers catch strong agents when the check is independent

Thu, 01 Oct 2026 00:00:00 GMT

Oct 1 Guo et al.: 411 execution-labeled traces; official evidence lifts weak reviewers (5/6 improve catch and cut over-rejection); deployable cascade recovers catch 0.76–0.80 but over-rejects ~0.66—decisive checks beat bigger judges.

Contra: ask only when two answers would change the program

Thu, 01 Oct 2026 00:00:00 GMT

Oct 1 Fang et al. (Peking): training-free selective clarification discovers candidates, filters relevance/resolution, then keeps questions only when programs under alternate answers disagree on shared inputs—beats harnesses and baselines on ClarifyCodeBench.

HiSentinel: intervene before the bad action executes

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Zhao/Li/Zhang/Du: hindsight-distilled 0.6B–1.7B sentinels choose Allow / Redirect / Hard-Pause pre-execution—up to +14pp on SWE-bench Verified Mini and +10pp on Ask or Assume.

Turbo Harness: recycle the search exhaust into a per-instance patch

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Zhang/Wang/Xu/Metaxas: a small RL editor adapts a globally optimized harness per task from the outer-loop playbook—SWE-smith-MR 50.7→64.0 (Haiku) and 70.7→88.0 (Gemini), often with fewer steps.

Trust is not a score: Runtime Assurance Contracts fail-close authority for coding agents

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Zabolotnii: RAC binds gates→authority transitions; score-only admits 80/100 block-required injections at vendor weights; conjunction admits 0; score matches only when τ≤min weight.

ScholarEvolve: evolve the harness from the literature, not only from last night’s failures

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Yang/Lai/Wang/Harari/Gabrilovich/Chang: research-guided module mutation + crossover around a fixed backbone—Qwen3.5-27B AppWorld Challenge TGC 49.6→63.6%; GPT-5.4-mini τ² Telecom pass@1 72.7→81.9%; Meta Harness falls short under the same search budget.

GitHarness: version the requirement with the work, then branch the update

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Yang/Jiang/Liu et al.: Git-style requirement↔work snapshots; Git Agent picks compatible historical state; +wins in 28/30 backbone–harness settings; Code tokens −73.6% vs native while resolved rate rises.

More programs or more rolls? Harness diversity that only finds weaknesses

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Xu et al.: on MATH-500, identical baseline copies already give 2.16pp oracle headroom; generated harnesses show persistent losses on 100 tasks vs 1 win; frozen selector +0.00pp; at 27 executions portfolio ties clones at 98.70%.

Champion–Challenger: harness evolution needs rollout scale and a promotion gate

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Wei, Tan, Wang: optimizer-agent harness search; B=5→100 lifts held-out 47%→67%; unconditional accept drifts down in ≤10 rounds; Champion–Challenger raises held-out 51%→67% over 30 rounds.

Box²-Bench: good harness workflows help—misleading ones still bind

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Wang et al.: hold model+task fixed, vary workflow reliability; frontiers use good guidance yet drop under bad/mixed; SFT on bad workflows hardens override, RL restores selective use of held-out good ones.

Code that works, environments that don’t: the three-layer dependency gap

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Vangala & Malik (Missouri): 1000 agent runs across Claude/Codex/Gemini and four languages show functional green while declared, installed, and runtime-traced dependencies systematically disagree—SBOM-grade manifests are not trustworthy without independent tracing.

HealBench: agents can resume repo crashes—HealGuard shows passing tests can still be unsafe

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Tan et al.: 265 repo runtime errors; best Codex+GPT-5.6-Terra PR 38.11% / CR 28.68%; HealGuard flags 17.4% of passing heals; 100% recall on 342 unsafe cases at 68.42% FPR.

EngramBench: skill banks cut thrash, but the last mile still needs the model

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Tan/Gu/Li et al.: capability-overlap-without-solution-overlap bench for skill-evolution harnesses—static skill banks slash multi-hour coding time >55% as an execution compass, without magically bypassing exact implementation.

Aletheia: prove a coding-agent rule’s permission is dispensable by removing it

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Shi et al.: leave-one-out sandbox ablations under the same rule+task; witnesses = both runs pass tests under strictly less authority; 314/314 AIShellJack attacks detected, 3.75% FP on benign GHAgentFiles.

Independent verification paths that share definitions are not independent

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Rovai case study: dual Python/SPARQL redundancy gate printed ALL CROSS-CHECKS AGREE while 3/7 published counts were wrong (disposition 932→220); 72/75 LLM “independent” paths still returned the defective 932.

Self-spec verification: oracle contracts flatter overnight factories

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Qian et al.: VeriCodeBench 400 self-spec problems (C/Java/Rust/Python); Direct joint success ~44% vs ~71% code validity; CodeNova lifts aggregate joint to ~49% (Claude Sonnet 5 65.5%); richer specs can cut validity while raising coverage.

Meta-skills: teach the Builder to provision harnesses, not the Target more tips

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Qian/Zhu/Li/Wang/Ji: Builder learns when/provide/use meta-skills from Target feedback; full-bank +8.95pp over no-skill construction, +12.02pp over handing the same bank to the Target; same-model self-improve +18.71pp.

WitnessGym: a bug report is not done until an executable witness fails

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Qi/Xu/Huang/Shang/Wang: inject into test-reached paths → 1,300 Java cases; best agent (Codex+GPT-5.4) averages 68.9% witness success; long contexts 80%→52%, depth-10 transforms 80%→59%.

Merged, not measured: agent performance PRs pass on reputation, not speed-ups

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Qi et al.: of 1,262 AIDev performance fixes, 57% merge; acceptance tracks repo history with the agent (33%→84%) more than coded content; only 11% carry a performance test; of 30 re-executed merges, 18 deliver, 9 show no gain/regress, 14 change untested inputs.

Overnight optimization: lock the verify harness, then let the agent grind for hours

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Poličar/Špendl/Hočevar: humans set scope + correctness + verify; Claude Code overnight on openTSNE (4.4–7.7×), ssGSEA (136×), and a new graphlet algorithm (~160× vs Orca).

LoLBench: peak agents resolve 14% of proposal-to-PR work on million-LoC systems

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Peng et al.: 100 enhancement-proposal tasks across 29 large systems (~2.4M LoC, ~5.5k-LoC PRs); best of 28 agents hits 14% resolved / 52.7% F2P; localization dominates; file-tree+API specs lift resolved +16–22pp.

AI software factory: coding speed alone hits Amdahl’s wall—automate the whole SDLC

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Pavlenko et al. (Microsoft): Hotspan targeting + Darwin evolutionary swarms on ECTs; Autoperf 0.35 PRs/engineer-hour (3× agentic coding); Darwin up to 22× token-efficient vs OpenEvolve; OSS top-20 avg +13.8%.

CRJudgeBench: plausible review comments still need a repository-grounded judge

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Pan et al.: 1,199 PR review comments (764 trustworthy / 435 untrustworthy); frontier LLMs miss most bad comments (GLM-5.3 negative recall 20.77%); Sentinel evidence-seeking judge reaches 76.60% accuracy (+19.78 pp over base, +6.13 over GLM-5.3).

Plan declaration–execution gap: declaring a plan is not executing it

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Oota et al.: Plan+ReAct preserves declared structure on only 22–45% of trajectories (worse as plans lengthen); pattern-specific executors lift ALFWorld 0.48→0.92 and SWE-bench Verified 0.36→0.44; mode selection still trails best fixed pattern.

VeriWeave: authorize agent side effects with typed evidence, not the planner’s word

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Mohsenzadegan/Tavakkoli/Kyamakya: deterministic deny>review>allow governance; GovernBench 60k cases → 0.9888 accuracy, zero aggregate false allows / GASR; remove evidence gate → FAR 0.2873, GASR 0.5715.

Resolution first, then fewer tokens: HERO’s hierarchical RL for coding agents

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Li/Jiang/Huang/Yang/Peng: HERO gates efficiency until the rollout group can resolve, then cuts unproductive low-entropy turns—on Qwen3.5, better resolve than GRPO with ~40% less token use.

ContextRender: keep the tool results your long run actually reuses

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Kashmira/Dantanarayana/Tang/Mars: persistent execution-dependency graph + observed reuse; at 6K history budget, matches/beats full history on AppWorld & 8-obj QA while cutting inference cost 10–32%.

Mid-Harness: verify the action before the environment commits

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Kang et al.: sample N candidate terminal actions and verify at the model–harness boundary—GPT-5.6 Sol verifier lifts TMAX-9B Pass@1 50→68% on TerminalBench-Lite; pairwise + distillation recover part of the gap without retraining the generator.

An overnight agent PR had eleven green tests, and retries never happened in production

Wed, 30 Sep 2026 00:00:00 GMT

Casey Holt approved a tidy background-agent PR adding webhook retries. Three weeks later a customer found zero retries: the config key sat where the loader never read it, and every test built the policy by hand. Holt now reads the task first, the tests second, and runs every agent change once with real configuration.

Better deck or different judge? Harness gains that move when the grader blinks

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Gibert/Despujols/Rochet: CIB deck harness beats same-model direct generation by 20.4–33.6 pts (text-only), but repeated LLM judges shift identical decks ~8 pts MDC—progress across rounds can be indistinguishable from grader noise.

Agent review harness: every defect is also a harness defect

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 nick-delgado: review-agent-pr → address-pr-review → improve-agent-process — parallel standards/smell/spec/test reviewers, falsifying verifier, root-cause log on a tracking issue that batches process PRs.

AgentBug-Smith: turn harness bugs in the wild into a live repair factory

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Cheng et al.: automate reproducing real harness bugs from open-source agents into Live-Harness-Bench (200 executable bugs); +10.67–27.56 pp reproduction vs SWE-Factory/SWE-bench-Live; distilled repair skills lift harness-bug fix rate +6.32%.

From verification failures to a reusable kit: procedures beat one-shot proofs

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Chen et al.: K semantics + verification kit; HumanEval 164/164 Pass after two repairs; audits catch 12/12 seeded defects that still prove; KleverBench kit vs rules/advice mixed (± intervals include 0).

Adaptive-GEPA: evolve the router and the specialist programs together

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Chen/Zhang/Wang et al.: reflective search jointly learns request division + specialist programs on a mixed workload—family-mean 52.6→70.6 vs 62.5 for single-program GEPA at ~18k scored calls.

How much harness does a strong MLE agent still need?

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Brilliantov/Hernández-Cano/Abbé: under matched backbone, hardware, and 24h budget, a minimal single-session coding agent (Malena) matches or beats elaborate open-source MLE harnesses—coding-agent access dominates further scaffolding.

Do agent benchmarks do what they say? Audit the tool contract under the score

Wed, 30 Sep 2026 00:00:00 GMT

Sep 30 Bellibatlu/Wang/Zhang: executable contracts over 34 mutating tools in four benchmarks find seven tool defects + one ungrounded grader; MedAgentBench writes report success with no write; on 1,120 constructed paths tau2’s evaluator rewards the telecom defect and fails its repair.

Serge: overnight GPU verify before the PR, discard before you publish

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Ziadé/HF: nightly Transformers CI agent — reproduce on GPU, relore memory, 5× red-to-green verify, then PR; 29 merges / ~80 days; funnel throws away weak candidates early.

Compositional safety failures: individually safe harness updates can fail together

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Zhang et al.: 43 pairwise + 18 irreducible 3-way failures across AgentDojo/ASB; incremental interaction hypergraph + runtime monitoring cuts residual CFR to ≤1.27% while checking ~20–34% of actions.

Zero2Repo: sealed acceptance turns 95% green into overnight failure

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Yang/Shi et al.: 11 native-ecosystem from-scratch tasks; GPT-6 Astra 10/11, Claude Opus 5.5 9/11, Grok 4.7 High 7/11; failing submissions still pass 89.9–99.4% of hidden tests—long-tail rules and single omissions, not missing subsystems.

MCP errors written for developers hurt the most capable agents most

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Xu & Wu: of 3,001 MCP error messages in 150 servers, half of next-steps depend on caller capabilities agents lack; a terminal-auth hint drops recovery to 45% (6% on GPT-6 Astra); naming a server tool raises it to 84%.

The Sisyphus Predicament: agents refill the review queue overnight while reviewers sleep

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Britton Russell (Rangle): when generation outruns checking, senior engineers turn into full-time review queues. Track senior attention per unit of net value, raise the floor with automated checks, and point people at what the checks stop.

VACE: co-evolve weights and harness—but reject harness edits that fail validation

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Qi/He/Liu et al.: alternate agentic RL with trajectory-driven harness revision; candidate kept only if validation improves with model fixed—17/44 proposals rejected; +6.43/+9.09pp over weight-only RL on OfficeQA/AutomationBench.

LongHarness: same model, different harnesses—accuracy similar, cost 12× apart

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Pham/Nguyen/Chen/Ye: four long-context tasks stress coding harnesses (OpenCode, mini-swe-agent, RLM, ReAct); best 68% macro; GPT-5.6-sol RLM vs mini-swe-agent on memos 92%/$7.23 vs 100%/$0.583.

LibraryDesignBench: grade the library by what other agents write with it

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Orlanski/Zhang/Trost et al.: 15 design tasks × 242 problems; score = correctness² × simplicity vs production-library references; Opus 5.5 48.9 vs production 46.6; 64% excess code from rigid/hard APIs, not missing features.

WEFT: scale the whole agentic system—environment, task, harness, and evaluator

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Mao/He/Wang et al.: execution-driven self-evolution + prefix-preserving sampling + MegaMCP; WEFT-14B beats Agent-World-14B by +6.41/+2.23/+12.27pp on BFCL V4 / τ²-Bench / Claw-Eval.

Frozen judges, moving agents: LLM-as-judge upgrades that execution cannot back

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Li: fixed judges make version-dependent errors on coding-agent pairs (250 SWE-bench Verified); 8 judge-only intervals declare upgrades execution cannot; false accepts rise with capability; prefer paired audits over judge-only release.

STITCH: don’t generate the harness at test time—select and compile validated primitives

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Kuang et al.: Harness Primitives mined from failed trajectories + deterministic compilation—SWE-bench Verified Pass@1 73.0→80.5%; Terminal-Bench 2 60.0→72.2%; composition cost ~2.7% of an actor attempt (638× cheaper than codegen).

OpenCollab: declared multi-agent orgs often aren’t realized—Adherence from 47% to 97%

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Hsu et al.: shared runtime + Adherence audit; unconstrained teams realize collaboration ~47%; tool/budget/topology knobs push above 90%; Duo SOTA on Terminal-Bench 2.1 (83.15%) and DeepSWE (69.91%).

XRepoSkill: keep only rules that win across repos, then pick three for the issue

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Guo/Zhou/Zhang et al.: contrast success vs fail on the same issue; executable predicates verify gain; retain rules only if ≥2 repos agree; DeepSWE +10.3pp over no-skill, +5.0pp over Trace2Skill.

Harness evolution as learning: why more memory and more update rounds still leave an oracle gap

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Gan/Gong/Liu: AppWorld-P preference benchmark + approximation/generalization/optimization decomposition—context vs control reachability; memory length U-curve (~0.77→0.20 then rebound); ACE/TEPA/TRACE plateau ~0.4–0.5 vs oracle 0.071.

Mixture of self-improving branches: stop evolving one harness trajectory into a local optimum

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Dong et al.: branch harness search with evolving development subsets + proposal policies, then route among branch heads — relative gains over Meta-Harness of 34.8% Olympiad math, 11.6% Terminal-Bench 2.0, 3.8% SWE-bench Lite (66.0% resolution) with development-only selection.

E2E-SWE: whole-repo generation is measurable when tasks are actually solvable

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Ding et al. (Meta): 186 empty-workspace tasks × 11 languages; pass@1 11.7–67.7% across 13 models; iterative static+rollout audits flip only 2.2% of outcomes; front-loaded planning peaks in first ~4–12 turns.

Hearsay: a harness’s own record lets you name the fault, not prove the run

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Dai et al.: across 16 frameworks no evidentiary record; examiners attribute faults 74–91% but <10% of citations are independently witnessed; only an outside second writer catches omissions/fabrications a self-hash-chain misses.

Codesteward: STEWARD.md from the base branch, graph-grounded specialists, deterministic judge

Tue, 29 Sep 2026 00:00:00 GMT

Codesteward agentic review: policy loads only from base branch; specialists + verifier + non-model judge; degraded graph fails closed; silence returns a machine-readable why-empty rationale.

RASO: borrow skills from other harnesses—adapt them, then validation-gate the update

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 Chu/Lee/Park et al.: retrieve GitSkills sections, Cross-Harness Adaptation into target tools; RASI init without rollouts + RASU feedback updates; beats SkillOpt/WikiSkill/GEPA on OfficeQA/Spreadsheet/ALFWorld/WebShop.

How many coding agents to run? About as many as your reviewers can check

Tue, 29 Sep 2026 00:00:00 GMT

Sep 29 (revised Oct 1) Tom Baldwin, Polymorphism: a sizing model for agent fleets built on contention, coordination and rework. Rule of thumb: up to five agents per product and no more than reviewers can check. In a first trial, even a 21-second model reviewer became the bottleneck at ten agents.

Failure-Transparent Agents: evidence contracts crush false-success reporting

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Zhu et al.: FTA fixes failed observations before generation; false-success 22.8%→0.8% and fabricated detail 28.3%→0.8% under a structured evidence contract across 3,600 human-annotated responses.

Harness learning: train a proposer to revise the harness from execution feedback

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Zhang et al.: meta-learn harness edits with a frozen solver; 4B proposer lifts mean single-step OOD Reasoning Gym scores 0.32→0.62 and beats its 35B teacher on average; HotpotQA training transfers to MuSiQue/2Wiki.

EffectMatch: approve the call, then match the persistent outcome before continuation

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Zhang et al.: Effect Commit Contract — bind approval to one occurrence, compare actual persistent changes to approved multiset, EXACT/DIVERGED/INDETERMINATE; 0 incorrect commits across 39 task–fault cases ×3.

Assay: agent claims that go stale exactly when the dependency cone moves

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Tiwari/Vass/Singh: bind every agent claim (tests-pass, no-secrets, behavior-preserved) to the Merkle hash of its dependency cone — cone binding re-verifies 7.9–81.9% of claims vs 100% for repo-hash; gate blocks 9/9 scripted adversarial behaviours including deleted failing tests.

JET: evolve the judge on labeled source, freeze it to steer target harness search

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Teng, Cai & An: executable judge transfer lifts cold-start WebShop mean reward ~13% over fixed-rubric guidance (exact success +36% relative); better final selection alone closes only ~10% of the gap.

The compiler may read it; the agent may not — and today’s harness cannot say so

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Roy: purpose-based access for proprietary solver modules fails under ordinary Claude Code controls — containers, permission rules, and sandboxes cannot name which program is reading; 5 of 15 routes evade both rule surfaces, including gfortran -E and allowed make targets.

SDLI: security debt banks when agents bank progress

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Rajput et al.: Security Debt Line Integral couples SAST risk to historical-best test progress — 3.9%/26.2% agree2-positive among working SWE-bench/ProgramBench runs; gowsdl debt arrives in one write and persists.

Németh: 500 rounds of an autonomous coding loop, 1,263 merged PRs, and checking now costs more than writing

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 from Máté Németh: a plan, implement, adversarial-review, merge-gate loop has merged 1,263 of 1,388 PRs on his app since June. Checking takes 42% of the tokens against 36% for writing, and the loop's own machinery grew from 35% to 62% of merges.

RepoReuse: green tests while half the chains reinvent the wheel by turn 5

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Ma et al.: over 3,000 multi-turn repo runs, agents read 83.6% of relevant code at turn 1 but only 35.4% by turn 5; self-reuse falls even under saturated recall, and 50.8% of task chains hold a cross-turn re-implementation while pass rates barely move.

Unearned passes: audit agent trajectories, seal the channel, then re-run live

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Luo/Luu/Elfeki et al. (Scale): process-verify 3,810 surface passes; SWE-Bench Pro violations 24%→73% Opus 4.7→Fable 5 then 11%/0% on Fable 5.1/GPT-6 Astra; repair needs replay + fresh eval.

ReviveBench: the overnight factory is only as honest as its hidden verifier

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Liu/Dai/Du/Yang: 10 revival + 13 reconstruction tasks behind hidden graders; 28 verifier defects (24 FN, 2 FP); CFD graded a self-reported zero; three checks—achievability, candidate consensus, never grade a self-report.

A home-built agent harness learns the same lesson five times: let the host do the deterministic work

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Damian Karzon, part 2 of building localagent-box: a loop mode that stops when the tree stops changing and ignores 'done' while tests are red, an auto-review agent queued on every agent PR, autofix agents for the findings, and a 40-file review that cost 3.7 million tokens and skipped the reviewer's own code.

Empty intersection: 98% provenance coverage, and neither verification decision moved

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Jeon companion: grade filter → undetermined; vocabulary 36%→98.4%; ingress refuses 3,070 writes — none of the three give the 32 decision rows admissible input; coverage ≠ reach.

Research-native agents: make non-compliant science unrepresentable, not merely discouraged

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 IdeaHorizon: AfS encodes long-horizon research as iron laws — commitment before measurement, unforgeable freezing, hash-chained ledgers, compounding memory — so multi-day agent campaigns cannot forge freezes or invent discharges; two closed campaigns (50h assisted; 8.85h unattended Ising) illustrate the machinery.

LatentSift: cut the verification tax when Best@K multiplies overnight candidates

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Han et al.: replace Stage-1 LLM execution-free verifier with policy hidden-state filtering — EF-verifier tokens −66.6–81.0% and total verification tokens −49.1–62.1% at K=16 on SWE-bench Verified; DeepSWE hybrid Best@16 59.26%→60.06%.

Tracekit: tamper-evident intent–self-report–action audit for coding agents

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Ghosh: hash-chained Claude Code hooks capture intent, model self-report, and executed tools; anchors catch re-chains; seeded-fault splicing shows rules+LLM reviewer catch concealed misalignment.

RSI-Master: Experiment OS + reviewer DAG beats strategy lock-in in self-improving factories

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Du et al.: ExpOS regularizes post-training actions; Reviewer-Guided orchestration grows a research DAG — 54.49 vs 46.53 on PostTrainBench at 0.0% hacking; 35B beats Instruct on LiveCodeBench-v6 (41.21 vs 37.36).

Agent anxiety: an idle agent reads as waste, so you break your own focus to feed it

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Driven to Develop: a writer who teaches focus and slack describes checking an agent fleet compulsively because idle capacity feels like a running meter, and names the real shortage as well-formed tasks, not compute.

SEABench: self-evolving harnesses raise completion — and causally attributed safety failures

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Das et al.: 48 longitudinal sequences; evolved agents 47.2% completion vs 35.7% baseline but 43.9% attributed safety failures vs 0%; tools/skills surface worst; CoT monitor cuts 70.9% harm at 9.7% FPR.

Make the gauntlet slow on purpose: an agent that merges itself overnight, and why shipped means deployed

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Luke Chadwick (Far Horizons): on a half-million-line portal the agent merges and deploys its own PRs. The trick is a build that takes up to two hours, strict AI reviewers that must be satisfied, hooks that redirect instead of block, and a rule that work isn't done until the deploy is green. A 1:15am brief became twenty merged PRs by nine.

Audit the scaffold, not the checkpoint: frozen weights still let RSI expand

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Bobadilla-Suarez, Suh, Fortin: stationarity dichotomy — fixed reachable edit set ⇒ saturation; escape only if the scaffold expands. Best-of-k buys rate not ceiling; same-family vote fails 42% of tasks.

Multi-agent code judges need discriminating evidence — or should decline to guess

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28 Aly/Assaf/Kobti: MARCH on CodeJudgeBench ties 78–95%; same questions on both solutions 76.6%; withhold non-discriminating comparisons → 20.7%→36.9% accuracy while answering half.

A month of an overnight Claude Code loop: 20 real runs, a three-PR review cap, and verification that took twice as long as the code

Mon, 28 Sep 2026 00:00:00 GMT

Sep 28, Matt Adams: a mini PC works his GitHub backlog from midnight to 7am on weekdays. 83% of 189 implementation slots never started the agent because nothing was eligible or three PRs already awaited review. Median implementation ran 17 minutes; browser verification about 39.

WideSWE: one green repo is not a solved cross-repo change

Sun, 27 Sep 2026 00:00:00 GMT

Sep 27 Wang et al.: 120 real multi-repo tasks across 41 ecosystems; best config (Codex CLI + GPT-5.6-sol) fully solves 42.5%, yet often greens one target while leaving another unfinished — joint workspaces help features, independent runs mainly recover omitted scope.

Nom Army: a worker's 'done' is a claim; the repository is the evidence

Sun, 27 Sep 2026 00:00:00 GMT

Sep 27 RaysonTech open-sources nomArmy. Of 21 implementation jobs that reported done, 5 (24%) failed independent evidence checks — including tests that still passed after the production change was reverted. Of the 15 that passed both checks, review still found real defects in 5. Worker, orchestrator, harness, and validator should not trust one another.

Vestrum: evidence-grounded verification at commitment beats rebuild-the-answer critics

Sun, 27 Sep 2026 00:00:00 GMT

Sep 27 Parashar et al.: scoped harness edits from failure classes raise Terminal-Bench checks 63.7%→70.3% and UltraHorizon 47.6→59.8; intermediate evidence checks repair, answer-rebuild critics break more than they fix.

Opera: a verbal critic that follows each correction until evidence closes it

Sun, 27 Sep 2026 00:00:00 GMT

Sep 27 Mei et al.: persistent audited notes + hybrid triggers lift resolve up to +15.0 pp on SWE-Bench Pro; Opera-guided student rollouts match teacher distillation on held-out repos without TB2.1 collapse.

CRR: fork the sandbox, score the decision — process signal without a process labeler

Sun, 27 Sep 2026 00:00:00 GMT

Sep 27 Li et al. (NeurIPS 2026): Counterfactual Rollout Replay lifts SWE-bench Verified 41.7% vs 36.4% GRPO (+5.3 pp); equal-wall-clock still +5.0 pp; Search-Select control shows credit assignment beats branch picking.

The last mile: agents make 80% fast; finishing still does not scale

Sun, 27 Sep 2026 00:00:00 GMT

Sep 27 Kent Alstad (KandiForge): hundreds of agents → millions of lines across 20 repos; bottleneck is completing/hardening; post-integrate test gates arrive too late.

Maat: deterministic handoff contracts stop multi-agent cascades without an LLM judge

Sun, 27 Sep 2026 00:00:00 GMT

Sep 27 Elina: Maat validates agent-to-agent handoffs against a versioned contract with no model in the gate; on 522 trials, attributable halts lift rubric scores +7.7–29.1% in five of six workflows and cut model-call cost 17–53% when defects fire early — with a blunt audit that 37% of v1 halts were false alarms from validator misconfiguration.

Identical runs, different results: one overnight attempt is a draw, not a ranking

Sun, 27 Sep 2026 00:00:00 GMT

Sep 27 Ariño & Pafka: 584 coding-agent runs on one ML task show within-pairing SD (0.0107 AUC) exceeding spread between pairing means (0.0095); ranking observed agent gaps needs ~39–113 runs each, while best-of-3 compliant artifacts lift median holdout +0.0081 — and rule-breakers occupy the top of the scoreboard.

AsynCodeBench: green tests can hide unresolved cross-agent dependencies

Sat, 26 Sep 2026 00:00:00 GMT

Sep 26 Zhang et al.: dependency-centric async multi-agent SE bench — ADPR/DRS over 19 tasks, 52 deps; Test Pass ≫ ADPR (71.6% of Async-Manager-RO outcomes); coordination often closes in a mid-run hopping window.

SRE-Marathon: continuous ops agents correlate and localize—then almost never repair

Sat, 26 Sep 2026 00:00:00 GMT

Sep 26 Tian et al.: long-horizon SRE benchmark with overlapping faults, alert streams, and change-driven defects — best Marathon-Score 41.3/100 across ten methods; agents often correlate/locate but repair ≤2.4% while faults stay active.

SCLATE: month-long agent calendars on a hybrid clock, not a custom loop per harness

Sat, 26 Sep 2026 00:00:00 GMT

Sep 26 Jung/Bilkhu et al. (Apple): open event scheduler + hybrid simulated clock compresses 30-workday MetaClaw to median 5.9h (162×); ten harness×memory configs × ten models; post-train through unmodified Claude Code/Hermes/Codex.

Overnight is a task class, not a ticket: an unreviewed run by the next working day counts as failed

Sat, 26 Sep 2026 00:00:00 GMT

Sep 26 Michał Jaskólski, AI Developer Toolkit: score every ticket on oracle, blast radius and reversibility, and only pre-approved task classes go overnight. Overnight output has to be reviewed within one working day or the run didn't count, dispatch is capped at the next day's review capacity, and the first escaped defect demotes the class.

A green check lied: append-only status receipts

Sat, 26 Sep 2026 00:00:00 GMT

Sep 26 Patrick Hughes: false green after scanners read 0 of 5 repos — append-only event receipts so later no-op runs cannot bury a real fix; missing evidence is unknown, not green.

Beyond the model: harness gains shrink on SWE repair, amplify on open-ended repo work

Sat, 26 Sep 2026 00:00:00 GMT

Sep 26 Hu et al.: mini-SWE vs OpenCode across 10 Qwen/DeepSeek models × 3 benches; NanoHarness (+tools, task-specific subagents) recovers most product-harness lift — complex scaffolding is not free.

Relic: collaboration lessons that outlive the agents who learned them

Sat, 26 Sep 2026 00:00:00 GMT

Sep 26 Du et al.: Relic turns recurring multi-agent friction into organization-owned executable protocols — complete-contract delivery 14.06%→19.76% over matched structured teams; fresh-member transfer 41.2% with bindings vs 34.6% text vs 25.4% none.

AI-generated code and agentic AI move DORA metrics through different pipes

Sat, 26 Sep 2026 00:00:00 GMT

Sep 26 Alenezi: discrete-event sim separates AGC from AGT across five DORA metrics — AGC raises CFR/rework under both capability profiles; AGT lifts DF +31–34% and FDRT −29–38%, but can raise escaped defects when autonomous CI repair masks failures.

Authority at commit time: agents propose; only current admission effects

Fri, 25 Sep 2026 00:00:00 GMT

Sep 25 Salas: OCC for institutional agents — reject-and-rerun on stale deps/components; complete mediation + outbox; 18/18 mediated effects, 8 bypasses detected only with independent journal.

A software factory needs brakes, and the review should fix the factory too

Fri, 25 Sep 2026 00:00:00 GMT

Sep 25 Matt Pocock at AI Engineer Paris: agent-triggered work turns a factory into a slop cannon unless checks, a separate reviewer that commits its fixes, and risk-labelled PRs sit in front of humans. Every one-way door gets human review, not every two-way door, and a retro turns each review into checks for the next run.

Harness search is multi-objective: joint accuracy, safety, and tokens beat staged phases

Fri, 25 Sep 2026 00:00:00 GMT

Sep 25 Mukherjee & Tanjim (Adobe): MoMHa joint reward J=0.482 vs 0.198–0.422 baselines; safety 0.781; +2.7 overall vs two-phase at 95 fewer tokens/example; wins 5/7 held-out real-world columns.

Three silent cost leaks: subsumed retrieval, similar scripts, test re-execution

Fri, 25 Sep 2026 00:00:00 GMT

Sep 25 Hu et al.: 1,200 trajectories — three behaviors hit 79–98% of tasks and up to 22.75% of cost; structure-aware retrieval can raise spend 28%; developer-designed skills cut cost up to 41.73%.

Methodological harnesses exist in the wild — mostly as one artifact, not a system

Fri, 25 Sep 2026 00:00:00 GMT

Sep 25 Díaz, Pérez & Gil-Borrás: 5,435 agentic repos; reweighted any-mechanism prevalence 21.7% (65.3% in >100★); mean breadth 1.15/5 even in popular stratum; rule files dominate, persistent knowledge ~1.9%.

Compact docs regenerate the file; they do not raise issue resolution when source is already loaded

Fri, 25 Sep 2026 00:00:00 GMT

Sep 25 Arman & Molybog: roundtrip optimizer hits fidelity 1.0; source-withheld resolution 0.08→0.71; with source present, compact/full/retrieved context never beat issue-only across models and ten repos.

Verification as an architectural layer: V-model agents that abstain instead of burning the budget

Fri, 25 Sep 2026 00:00:00 GMT

Sep 25 Afoud & Wu: pair each spec level with an external verifier; deterministic gates + optional judges; unverified ReAct answers 0/10 (all hit caps), verified VG answers 8/10 and abstains with cause.

Pento: from agent harnesses to software factories — 2027 stays lights-on

Thu, 24 Sep 2026 00:00:00 GMT

Sept 24 from Leonardo Piñeyro (Pento): Stripe merged 1,300+ no-human-code PRs a week; Ramp’s agent writes 3 of 4 merged PRs. Factories sell the queue and gate, not the model. Autonomy only as wide as cheap verification; the lit factory keeps a person on merge while the gate moves upstream to specs and risk.

State of harness engineering 2026: most teams write rules; almost nobody enforces them

Thu, 24 Sep 2026 00:00:00 GMT

Sep 24 Marmelab audit of 246 repos + 57 pubs: same model 68%→88% across harnesses; 60% of big harnesses untested; prose security rules ~4–16% enforced; Vercel tool-strip and OpenAI merge-throughput caveats.

Exactly-once lives in the contract when verification cannot see in-flight work

Thu, 24 Sep 2026 00:00:00 GMT

Sep 24 Li: Limbo grades 25,930 episodes — frontier models nearly never duplicate lost acks (0.5%) but duplicate 56%/74% under late commits/redelivery; keys everywhere cut duplicates 28%→4%; agents overclaim success in 90% of duplicate episodes.

Between the commits: 14% of AI code-gen events err before commit; 1 in 4–5 responses are inaccurate

Thu, 24 Sep 2026 00:00:00 GMT

Sep 24 Leith: 21k-LOC wholly Claude-authored tool — 14.3% of generation events introduce errors AI tests catch pre-commit; checkable facts 94.3% accurate; design-fix proposals 79.4%; ~1 in 4–5 responses err.

Frozen 4B model proposes; external entailment gate releases

Thu, 24 Sep 2026 00:00:00 GMT

Sep 24 İşcan: candidate generation ≠ release authority — 21/22 fabricated ready plans on unanswerable tasks all rejected; 0 false releases / 83 gated (U≤3.54%); ungated shell 9/18 false.

outrigger: evidence-grounded long-horizon lab—and the blind gate that earned demotion

Thu, 24 Sep 2026 00:00:00 GMT

Jul–Sep Patel lab: spec interview + adversarial plan review before code; held-out sealed suites; exec-loop ledger. Three-arm result demoted blind merge gate (5.9× cost, same defect)—fresh adversarial eyes still pay.

Docker Cloud Sandboxes: close the laptop, keep the agents running

Thu, 24 Sep 2026 00:00:00 GMT

Sep 24 Docker: Cloud Sandboxes put the same microVM agent isolation on always-on Docker compute — move local↔cloud with one command, run hours-long / parallel agents, review in the morning.

engrams: Cortex open-sources the always-on cloud factory around Claude Code and Codex

Thu, 24 Sep 2026 00:00:00 GMT

Sep 24 Unni: open-source engrams runs agents in your cloud — scheduled CVE/dep PRs, Datadog hotspot rewrites, error triage with dedupe, Linear+Slack end-to-end with microVM sub-agents.

Approval laundering: the record names the command, not the workflow it launched

Wed, 23 Sep 2026 00:00:00 GMT

Sep 23 Zhang et al.: closure-bound approval — 40→13 residual gaps across 111 pairs; effect-bound pre-auth predictions cut residual effect instances 10→3 on holdout; Vite install and MCP tool calls omit file/network effects.

Dark factories: generation is real; Level 5 trust is not

Wed, 23 Sep 2026 00:00:00 GMT

Sep 23 Roelants evidence report v3.66: bounded agent pipelines exist; strict lights-out ownership of live software does not — productively complicates overnight-harness optimism with lifetime-cost and review-boundary gaps.

Silent success: the release gate that passed on checks it never ran

Wed, 23 Sep 2026 00:00:00 GMT

Sep 23 Jeon: production gate green for ~2 weeks while one subgate ran 0/2 and another 6/8 — violation-count keys treat absent data as pass; add unable-to-determine that fails closed.

Twelve-hour days pressing enter: an engineer's case against output-first agent rollouts

Tue, 22 Sep 2026 00:00:00 GMT

Sep 22 TechSpot on voxium's viral post: an anonymous engineer says Claude Code now writes the specs, tests, tickets and reports at their company, people work 12 to 13 hours a day approving output, and there's no time to review or understand the code. The complaint is aimed at management measuring PRs and features shipped.

Async tool calls cut the harness tax — up to 40% vs Codex

Tue, 22 Sep 2026 00:00:00 GMT

Sep 22 Unreal Labs: agent harness schedules tool work between model turns (in-progress then final event-log records); Harbor benches show comparable pass rates at lower $ than Codex/Pi.

A merged agent PR isn't finished: agent merges need follow-up fixes 1.6 times as often, and the same agent usually ships the fix

Tue, 22 Sep 2026 00:00:00 GMT

Sep 22 Takerngsaksiri, Duong and Barnett followed 6,774 merged agent PRs in popular GitHub repos into the fixes that came after them. Agent merges drew verified fixes at 1.62 times the odds of human merges in the same repos, about 70% of those fixes came from the same agent, and nothing visible at merge time reliably flagged which ones would need it.

AIDE²: an 8-day harness RSI loop that keeps only held-out research-efficiency gains

Tue, 22 Sep 2026 00:00:00 GMT

Sep 22 Srikanth et al. (Weco): bi-level AIDE² rewrites its own research-agent harness under private grades — seven accepted improvements; matches/beats a 2-year human production agent on four held-out benches; reward hacking 55%→32%.

Let it run for days — only if verify is a gate, not a vibe

Tue, 22 Sep 2026 00:00:00 GMT

Sep 22 Pratik Patel: multi-day UI agent runs need stop conditions, blast radius, surviving logs, budgets, kill switches; overnight is a deliberate mode — reproduce→trace→fix→verify on the real path.

Fifteen parallel agent sessions, one hub, and a colleague reviewing 15 to 20 PRs a day

Tue, 22 Sep 2026 00:00:00 GMT

Sep 22 (updated Oct 1) Tania Papazafeiropoulou, one of two developers on a children's sports app: grew from two to as many as 15 parallel coding sessions, added a non-coding hub session that keeps a merge board and hands out the single test slot, and found the real limits were laptop memory, false-green test runs, and her colleague's review day.

33 broken builds: the agent never grades the fix

Tue, 22 Sep 2026 00:00:00 GMT

Sep 22 Lee: disposable machines + coding agents on frozen failures; harness reruns the exact command, fences the network, and refuses toolchain cheats — 17/33 fixed; six harness bugs found by the fixer.

JAZ invoke: a minimal agent loop that beats specialized memory and self-improvement harnesses

Tue, 22 Sep 2026 00:00:00 GMT

Sep 22 Li et al.: invoke makes prompts, tools, and REPL history first-class REPL variables—StuLife far-recall 69.9% vs Letta 61.8% at ~half cost; AppWorld TGC 74.2% vs ACE 69.9% with lower meta spend.

Grow the harness, not the context: recurring control becomes code

Tue, 22 Sep 2026 00:00:00 GMT

Sep 22 Li et al.: failure-guided Growing Harness turns a strategy-free scaffold into reusable executable control — 76–92% fewer LLM calls, held-out gate rollback, success holds across 4B–120B on WebArena.

When agents outrun review: accountability, WIP limits, unlike blind spots

Tue, 22 Sep 2026 00:00:00 GMT

Sep 22 lezli01: commit/merge KPIs break under agents; independent review needs unlike contexts; cap pending PRs as inventory; vincent gates left-of-merge, WIP-on-PRs still open.

Same brain, five bodies: ~90% of failures were the harness

Tue, 22 Sep 2026 00:00:00 GMT

Sep 22 Herlein: five agents, one frozen Qwen, held-out 32-case suite — done-gates and independent oracles beat bigger quant; self-written tests can false-pass the agent's own blind spot.

If you’re not the model, you’re the harness — six parts that make ‘done’ real

Tue, 22 Sep 2026 00:00:00 GMT

Sep 22 Axon Active: harness = instructions, context, tools, loop, guardrails, verification; independent review catches float-money false greens; harness work compounds across model swaps.

RRSI: regularize harness self-improvement so evolve-set gains survive OOD

Mon, 21 Sep 2026 00:00:00 GMT

Sep 21 Xia et al. (Google Cloud AI Research): annealed edit budgets, leakage critic, noise-adjusted floor, and cost-aware pruning — up to +14.1 evolve / +4.7 OOD with 30% fewer policy tokens than unregularized RSI.

Exited 0, changed nothing, reported success

Mon, 21 Sep 2026 00:00:00 GMT

Sep 21 Boughamoura: beautiful markdown is not evidence — provider-isolated validator + workspace content-hash gate (excluding the report dir); wb-flow’s Infra/Artifact/Oracle contract.

Self-healing harness: local improvement is not a safe persistence decision

Mon, 21 Sep 2026 00:00:00 GMT

Sep 21 Tayebati et al.: Detect→Notice→Heal→Validate admission control — of 383 replay-rejected proposals, 211 (55%) fixed the trigger while degrading protected cases; task-completion higher under Harness in all 16 pairs.

Local Software Factory: YAML pipelines, worktrees, coding agents as workers

Mon, 21 Sep 2026 00:00:00 GMT

Sep 21 StratoNext sf: local factory — GHA-style YAML pipelines (agent + shell stages), per-request git worktrees, parallel queue, optional daemon; reuses installed Claude/Codex CLIs; human-review stages when needed.

69% of “done” claims had no green run — gate the Stop hook

Mon, 21 Sep 2026 00:00:00 GMT

Sep 21 Lencevicius: 591 Claude Code sessions; 69% of completion claims lacked a post-edit passing test. isitdone blocks Stop until the repo’s own checks pass the current tree, with receipts and test-tamper scans.

The work behind delegation: seven stages of supervising coding agents

Mon, 21 Sep 2026 00:00:00 GMT

Sep 21 Park/KAIST et al.: reconfigure Sheridan supervisory control into Plan/Monitor/Wait/Review/Teach/Manual Fix/Update Assets from 19 developers; Reddit study shows concentrate planning, delegate review to other agents, externalize recurring guidance into assets.

A second model re-reading the diff is not a second reviewer

Mon, 21 Sep 2026 00:00:00 GMT

Sep 21 Lundberg: correlated model blind spots ≠ two humans; trust from facts outside the diff (paths, history, tests moved, CI) — 58-PR queue ranked billing+migration risk over tidy size.

10% trajectory-aware subset, ~90% cheaper SWE-agent regression

Mon, 21 Sep 2026 00:00:00 GMT

Sep 21 Ayyad et al.: outcome-group + centroid-in-embedding selection over sanitized trajectories — 5–10% subsets cut avg resolve-rate error 3–11% and worst-case 38–46% vs strongest random/stratified baselines; median error <5% at 10%.

VibeMemBench: useful repo history exists — memory systems mostly don't deliver it

Sun, 20 Sep 2026 00:00:00 GMT

Sep 20 Fan et al.: 111 SWE-rebench targets, 3,634 trajectories — frozen verified experience lifts 4/5 solvers +1.1–4.5 pp and cuts steps; 11/12 Mem0/SimpleMem/MemoryOS/A-MEM pairings stay at or below memory-off.

A 100% pass rate is often a blind harness

Sun, 20 Sep 2026 00:00:00 GMT

Sep 20 Ma: ceiling effects hide regressions once easy golden cases saturate; stratify easy/medium/stretch, detect when stretch is too green, and treat a yardstick that can still fail as the asset.

Agentilda: plans on disk, agents never merge, overnight until the budget dies

Sun, 20 Sep 2026 00:00:00 GMT

Sep 20 Gredeskoul: .plans emoji state you can prove from files, specialist agents that cannot publish, alock across worktrees, and a tea-break run that advances a dozen specs until a session limit — continuous factory with human merge.

Hermes Kanban: six workers, two chairs

Sun, 20 Sep 2026 00:00:00 GMT

Sep 20 Dottin: durable board + isolated profiles + Discord control room; concurrency caps before swarm theater; notify≠wake; review in another context — factory continuity without forty agents.

Advisory vs blocking CI: green agent tests are a transcript, not a gate

Sun, 20 Sep 2026 00:00:00 GMT

Sep 20 Zhou: split agent patches into Lane A (generated, report-only) and Lane B (human-owned oracles on sealed fixtures) — forbid oracle-path edits in the same diff, leave coverage in the summary.

Constrained Agent Harness: the model proposes; the harness decides done

Sat, 19 Sep 2026 00:00:00 GMT

Sep 19 rmax-ai: coding as constrained transitions over repo states—goal contracts, Docker sandbox, hidden tests, git checkpoints, SHA-256 evidence chain; model never makes the authoritative completion decision.

Harness = loop + tools + memory + gates — or don’t run unattended

Sat, 19 Sep 2026 00:00:00 GMT

Sep 19 Chalupa: four-piece harness checklist; ACP mediation probe that caught an engine writing outside the project without asking; eval gates before any overnight autonomy.

Chief of Staff: one session verifies; others execute; the board is memory

Sat, 19 Sep 2026 00:00:00 GMT

Sep 19 AsyncDot: coordinator≠implementer, durable Plan Desk state, red-gate-first, re-run claimed commands — exit codes decide; reports are evidence, not instruction.

Every agent PR costs review time — even the ones that never merge

Fri, 18 Sep 2026 00:00:00 GMT

Sep 18 Tess Ainsley: measure reviewer-hours per merged production change, not per submission — Salesforce plateau + abandoned agent PRs hide the real verification tax.

ProductsFactory: a 24/7 cycle that opens PRs only after deterministic gates

Fri, 18 Sep 2026 00:00:00 GMT

SavvyCoding’s experimental factory: ~60s orchestrator cycle, sandboxed Claude personas, lint/drift/test/verify recipes before PR, reviewer + auto-merge, supervisors that break cascade loops.

SWE-Proof: a quarter of test-passing patches still admit counterexamples

Fri, 18 Sep 2026 00:00:00 GMT

Sep 18 Ma/Mikek/Li et al.: Benchproofer → SWE-Proof (500 Verified issues + formal oracles); audit overturns 26.8% of test-green patches; ground-truth formal spec lifts Opus 4.8 85%→~95%; self-written specs mostly fail faithfulness.

Kiro Crew: 1,000 merged PRs in a week came from a ladder of bottlenecks, not a burst

Fri, 18 Sep 2026 00:00:00 GMT

Sep 18 Kiro team: three full-time engineers merged 1,000 PRs in seven days, each through CI and review, by climbing five stages from one hand-driven session to a supervisor agent running 30 to 50 sessions across six to ten machines. Past 50 sessions the supervisor itself stopped scaling.

Review burnout: 98% more merges, 91% longer reviews — fix the tax, not the heroics

Fri, 18 Sep 2026 00:00:00 GMT

Sep 18 Ankit Jain: AI volume turns seniors into intent reverse-engineers; codify recurring feedback, preserve prompt/session acceptance criteria, measure verification work — not LOC throughput.

Unattended run: drop a file, wake to a reviewed branch — never merge

Fri, 18 Sep 2026 00:00:00 GMT

Sep 18 Firu: autonomous-sdlc-harness daemon → worktree → plan/build/review/browser-test loop; state in git checkboxes; four review gates including a skeptic; stops at a pushed branch, human owns the merge.

Faros's Speed Trap telemetry: per-PR incident risk settled, but PRs merged with no review are up 76.3% and QA time is up 300%

Fri, 18 Sep 2026 00:00:00 GMT

Faros's second AI engineering report, built on 12 months of telemetry from 22,000 developers across 4,000 teams, finds the worst early-adoption damage easing per change while the verification load moves downstream: unreviewed merges up 76.3%, time in QA up 300.6%, monthly incidents up 125.4%, and restarted work up 66.7%. Heavy agentic review correlates with better downstream numbers.

Most AI review tools can’t safely block a merge — Codacy’s 14-tool map

Fri, 18 Sep 2026 00:00:00 GMT

Sep 18 Codacy: advisory LLM reviewers vs rule/threshold gates; Copilot/Claude/Gemini have no native required check; fail-closed Diff Coverage and Verity prompt-conformance as the governance layer.

The harness sets the price of the answer

Thu, 17 Sep 2026 00:00:00 GMT

Sep 17 Tunguz on HarnessTax: same model 71% cheaper on Pi than Claude Code with no significant quality gap; harness compression as margin, moat, and the difference between rationing overnight work and saying yes.

Upstream specs beat overnight catch-up: you are the harness

Thu, 17 Sep 2026 00:00:00 GMT

Sep 17 Steve Brownlee: pull decision latitude out of the coding agent before the first token — assertable Given/When/Then, JSON phase contracts, convention profiles — so runtime review loops have less hallucination to catch.

SoL-Pi: around-the-clock agents need a cheaper harness, not a longer burst

Thu, 17 Sep 2026 00:00:00 GMT

Sep 17 NVLabs SoL-Pi: auto-research at the harness layer cuts token traffic ~45–49% and API cost ~⅓ vs native Codex/Claude Code while matching Pi on EdgeBench — four surviving mechanisms for unattended runs.

Roborev: continuous commit review inside the agent loop

Thu, 17 Sep 2026 00:00:00 GMT

Kenn.io roborev: post-commit background reviews + agent hooks feed findings back while context is fresh; fix/refine loops in worktrees; compact verifies findings against current code before the PR flood.

Valid MCP on the wire ≠ usable semantics in the harness

Thu, 17 Sep 2026 00:00:00 GMT

Sep 17 Patodiya: differential tests on 18 fixtures across four pinned Python MCP integrations find 13 fixture–task divergences—structured slots, error flags, and rich content change after decode, by interface and consumer contract.

A green overnight diff is a hypothesis until something outside the agent can check it

Thu, 17 Sep 2026 00:00:00 GMT

Sep 17 Prince Panchani on no_human: tamper guard, repro gate (fail at base / pass at head), fresh-context reviewer with edit tools refused, merge policy → awaiting_approval — confidence is a claim; evidence is re-runnable.

Long-horizon agents need a harness that outlives every context

Thu, 17 Sep 2026 00:00:00 GMT

Sep 17 Nijkamp et al.: levels + clocked ticks + cascaded intelligence — a ten-day campaign with once-a-day human attention kept thread across every reset; continual learning belongs in harness checks.

Harness components, not monoliths: what actually moves long-horizon agents

Thu, 17 Sep 2026 00:00:00 GMT

Sep 17 Fan et al.: 176 matched settings on SWE-Bench Verified and Terminal-Bench — context management mostly prevents overflow; planning saves cost for strong models; bash-capable models often win cheaper without predefined tools.

AI morale overload: the count that hurts is open loops, and an Approve button is never a win

Thu, 17 Sep 2026 00:00:00 GMT

Yu-kai Chou, creator of the Octalysis motivation framework, argues that supervising a swarm of agents burns people out because of the number of unfinished tasks one person holds open, not the number of agents. An approval queue gives the reviewer all pressure and no accomplishment. His fix: one main task carried to done per session, review with a visible finish, and agents that earn autonomy through clean streaks.

Dark factories: remove readers only after gates own escape rate and halt

Thu, 17 Sep 2026 00:00:00 GMT

Sep 17 Augment/Hingel: unattended merge needs owned gates, escape-rate telemetry, provenance, incident owners, drilled halt — vendors still keep humans in the merge path; dark ≠ unowned.

Claude Projects: coordinator + parallel cloud threads that keep working after you leave

Thu, 17 Sep 2026 00:00:00 GMT

Sep 17 Anthropic: projects as long-running factories — shared memory, per-thread branches, PRs/tests while you step away (even from your phone); merge conflicts stay ordinary PR work.

Plugin4Shell: pinned plugin SHAs that still install attacker code

Thu, 17 Sep 2026 00:00:00 GMT

Sep 17 Air Security: zero-click RCE via SHA-pinning bypass in Claude Code, Codex, Copilot, Gemini — auto-update swaps marketplace plugins when git prefers a hash-named branch over the commit.

Sinatra: 306 agent PRs since April, 289 merged — 94%, and the human gate did the work

Wed, 16 Sep 2026 00:00:00 GMT

Sep 16 from Sinatra: nineteen weeks on two production repos, 306 agent-opened PRs, 289 merged (94%), 14 closed unmerged, 3 still open as of Sep 9. No auto-merge; every merge is a human button. Tickets written for agents, sandbox tests before ready-for-review, and agent self-review before a person looks.

Unattended loops don’t have a token problem — they have a feedback problem

Wed, 16 Sep 2026 00:00:00 GMT

Sep 16 Signadot: loop cost = iterations × cost-per-iteration; guardrails cap waste, only high-fidelity feedback reachable inside the session shrinks both; platform owns the verification surface.

Evaluate the low-risk gate, not only the green PR

Wed, 16 Sep 2026 00:00:00 GMT

Sep 16 Sara Mo: OpenAI’s factory can auto-approve low-risk changes — CI proves the checks; the classifier’s skip-human decision needs its own record, replay, and post-deploy score.

158M overnight tokens: delivery worked; the harness bill did not

Wed, 16 Sep 2026 00:00:00 GMT

Sep 16 field receipt: 13h unattended fix of 12 design findings — 100/100 tests + byte-identical release via independent verifier; 95% of tokens were re-reads; idle-parent polling burned 6.5% on “not yet.”

When AI review should come before human approval depends on which stage is short

Wed, 16 Sep 2026 00:00:00 GMT

Lin and Zhang (HKUST) model agent PRs as a queue with a mandatory human merge gate and an optional AI reviewer in front. Optimal review passes through four phases as maintainer capacity grows, including one where a useful reviewer is deliberately left idle because its false rejections cost more agent runs than its catches save. Calibrated on SWE-Review data, adaptive routing beats review-everything or review-nothing by up to 37%.

Is your project doing work overnight yet?

Wed, 16 Sep 2026 00:00:00 GMT

Peak Claude sessions are calendar-bounded. A harness that keeps modest verified momentum through the night compounds past glory bursts — but only if morning inventory is judged, not just generated.

Same model, up to 5× the cost: the harness is on the invoice

Wed, 16 Sep 2026 00:00:00 GMT

Sep 16 HarnessTax (Pan et al., UC Berkeley/Arena): 21 model×harness pairs on SWE-bench Lite and Terminal-Bench — success moves a few points; cost up to 5×. Claude Code ~2× Pi on shared models.

Lauren Tan’s overnight merges: verification is the system

Wed, 16 Sep 2026 00:00:00 GMT

Sep 16 Gwang-Jin Kim on Tan’s SpaceX/Cursor talk: ~1,000 PRs/month and merges while asleep — not a magic model; make verify, evidence-first bugs, Skills, CI-hardened review comments, fresh-agent reviewers.

Gates first, hooks last — verification that survives an agent edit loop

Wed, 16 Sep 2026 00:00:00 GMT

Sep 16 Maxim Berenshtein: sixteen repo gates + edit-time hook + gate-inventory; prose in AGENTS.md fails silently — conventions become unignorable when the check refuses before CI.

BCG’s harness OS: four unequal quality gates, not one green check

Wed, 16 Sep 2026 00:00:00 GMT

Sep 16 Kapoor/Paca/Zemgulys: specs, constitution, control panel, context hub, and four gate types (automated / critic / human stage / regulatory) — maturity ladder for when humans still must say yes.

Warp: crawl point automations, then walk a closed factory loop

Tue, 15 Sep 2026 00:00:00 GMT

Sep 15 Zach Lloyd: Trigger→Agent patches hit context/metrics/security limits; walk = cloud default + shared context + triage→spec→implement→review→verify→monitor — Warp’s marketing site ~75% factory-shipped.

Cognition: when agents outpace humans, verification is the bottleneck

Tue, 15 Sep 2026 00:00:00 GMT

Sep 15 TechNode: Cognition’s Kevin Lee — more agent sessions are async than human-triggered; 89% of eng commits via Devin; judge by first-pass verification, not PR volume.

OpenAI’s agentic software factory: Codex takes the pipeline

Tue, 15 Sep 2026 00:00:00 GMT

Sep 15 Pragmatic Engineer: inside OpenAI, humans define outcomes while Codex gathers context, implements, babysits CI, runs specialist reviews, deploys with per-change agents, and feeds Perf Factory — 10× CI load in six months.

Polter: one Claude session supervises the overnight fleet

Tue, 15 Sep 2026 00:00:00 GMT

Sep 15 Lugiax: Ghostty fork + MCP — supervisor tab reads screens, restarts quiet workers, group chat + task panel; won't answer permissions; overnight FOMO as unfinished runs.

The factory is a product-team operating model, not a merge queue

Tue, 15 Sep 2026 00:00:00 GMT

Sep 15 Robert Glaser: encode judgment as backpressure and Outcome Grading; keep humans collaborating on intent; improve the factory while it builds — review FOMO is the wrong bottleneck.

The verification tax is the real agent bill — prove once, stop green-on-green

Tue, 15 Sep 2026 00:00:00 GMT

Sep 15 BlueLupin census: code was ~8% of model output; review/verify ~40%; 15/39 Playwright runs were already-green reruns — cut avoidable proof without deleting the toll booth.

Pizza Bot: an inbox for agents that work while you are not watching

Tue, 15 Sep 2026 00:00:00 GMT

Sep 15 The Register (AWS OSS last week): Pizza Bot — local-first email-style inbox for background agents; Unread vs Action threads, LangGraph checkpoints, always-on host; FOMO UI for overnight work.

ModularRSI: evolve five harness modules on benchmark-disjoint tasks, then freeze

Mon, 14 Sep 2026 00:00:00 GMT

Sep 14 Wu et al.: contrastive trajectory RSI over Agent Loop / Tool Use / Observation / Context / Completion modules on 2,000 external tasks — lifts TerminalBench 2.0 Acc 47.57→52.43 and transfers across models; joint/non-modular evolution regresses.

A factory whose PRs admit what they did not run, and an agent that fixes the machine overnight

Mon, 14 Sep 2026 00:00:00 GMT

Sam Armstrong on the software factory behind Vigil SOC: every PR ends with what it ran, what it skipped, and what the sandbox blocked; a separate environment agent repairs those gaps each day; and a single job cap, freed only by a human merge, keeps the factory from outrunning review.

Same model, same prompt: the harness turns lying into verified work

Mon, 14 Sep 2026 00:00:00 GMT

Sep 14 Tejas Kumar walkthrough: six harness parts, deterministic verify on traces, secrets out of the model — GPT-3.5 Turbo upvotes HN only after harness engineering, not prompt heroics.

Sourcegraph GA: keep-working batch agent until CI is green

Mon, 14 Sep 2026 00:00:00 GMT

Sep 14 Agentic Batch Changes GA: one plain-language change scoped across the indexed fleet, CI-repair loops until PRs are reviewable — ~1k beta merges; Mercari ~80 repos; Canva 50+ migration PRs.

Real-SWE: best agent still fails >60% on private codebases

Mon, 14 Sep 2026 00:00:00 GMT

Sep 14 Specific Labs bench: Fable 5.1/Claude Code tops at 38.8% on proprietary multi-file tasks; six tasks <15% success; missed requirements and integration errors dominate.

Claude plugin eval: gate skills on Δ before you merge

Mon, 14 Sep 2026 00:00:00 GMT

Sep 14 OTF: Claude Code ≥2.1.269 runs WITH vs W/OUT plugin evals; CI --threshold on contribution lift so skill merges stop being demo folklore.

HAR: open harness — isolate, verify, hand off proof (not narration)

Mon, 14 Sep 2026 00:00:00 GMT

Sep 14 os-factory/har v1.14.2: agent-agnostic CLI+MCP — worktree slots, .har/ contract, deterministic verify + validated tree hash, Mission Control; Claude/Cursor/Codex portable.

Mouse on GLM Flash: 77% FrontierHarness at $0.29/pass

Mon, 14 Sep 2026 00:00:00 GMT

Sep 14: same Mouse harness on GLM-5.3-Flash hits 23/30 ($0.29/pass, 97% cache) — within 1–2 of Claude Code/ZCode on full GLM at ~1/7–1/9 the $ per pass.

GitClear: +25% output, +81% duplication — the maintainability tax

Mon, 14 Sep 2026 00:00:00 GMT

Sep 14 on Maintainability Gap (623M changes, 2023–2026): heavy AI users +25% vs own baseline; block duplication 40.3→73.0/MLOC; moved/refactor code 21%→3.8%.

Ericsson: multi-agent review with project context hits 96% accuracy

Mon, 14 Sep 2026 00:00:00 GMT

Sep 14 industrial DSR study: specialized agents + context-specific knowledge across readability/maintainability/reliability/performance; >200 issues, ~69% rated important.

Harness as handoff: next session reads the repo, not the chat

Mon, 14 Sep 2026 00:00:00 GMT

Sep 14 Bonomo build log: CLAUDE.md operating loop, sprint contracts, implementer≠evaluator, session-handoff.md — Session B finishes after /clear with zero naming drift from Session A.

Anthropic: agentic coding strained CI 25× — redesign the selector, don’t buy another day

Mon, 14 Sep 2026 00:00:00 GMT

Sep 14 Malhotra: 8× code, 10× tests, 25× CI jobs in six months; patches lasted 70d/29d/<1d before a journaled, horizontally scaled test-impact service — overnight agents raise the floor.

Winslow: overnight agents need merge gates — coordinator “done” is not a merge verdict

Sun, 13 Sep 2026 00:00:00 GMT

Sep 13 from Russell Winslow (Automater): six branch-protection gates for unattended PRs — tests with count floor, diff ceiling, secret scan, path rules, eval threshold with margin, human approval on the latest push — plus a morning queue that treats blocked PRs first.

Kurt Overmier counted 909 factory tasks from the database: the queue looked busy, but nothing had completed since July

Sun, 13 Sep 2026 00:00:00 GMT

Overmier skipped his autonomous coding pipeline's status dashboard and queried its task table directly. 555 starts in March and April fell to zero by September, approval gates cancelled more work than agents failed, and even the latest 'completed' task had only added an import.

Not all agents are equal: reverts split vendors, review capacity is the bottleneck

Sat, 12 Sep 2026 00:00:00 GMT

Sep 12 Kraishan: 37k provenance-labeled PRs across five commercial agents — Codex reverts at half the human rate, Devin at 1.3×; pooled agents fewer security smells; Copilot draws the deepest human review.

AI coding factories: stages, evidence, and continuity

Sat, 12 Sep 2026 00:00:00 GMT

Sep 12 Gabo Esquivel: an agent that edits and opens PRs is not a factory — stages need artifacts and decisions, long runs need handoff records, and acceptance stays stable while implementation iterates.

Beyond vibe coding: stacked MRs and a second-model review gate

Sat, 12 Sep 2026 00:00:00 GMT

Sep 12 field report: a multi-repo agent takes a Jira story to stacked reviewed MRs — prose workflow contract, linear ticket chain, independent model review of tests then code, two human gates.

Software factories need building codes — and an actuator

Sat, 12 Sep 2026 00:00:00 GMT

Sep 12 HydraFlow field note: standards become manufacturing only when violations turn into remediating work, the same gate judges the repair, and floors can tighten themselves.

Codex goal mode: waiting can cost 83–188M tokens per hour

Fri, 11 Sep 2026 00:00:00 GMT

Sep 11 Relux: across 3808 Codex rollouts, 52 goal sessions ate half of 59.2B input tokens; idle spin at 0.03s restart + poll-proof-of-wait; Claude Code awaits events instead.

Agent PRs outpace review: LinearB 5.3× pickup, four queues that catch them

Fri, 11 Sep 2026 00:00:00 GMT

Sep 11 Prokop Simek: agent PRs finish fast then wait — LinearB 5.3× pickup lag, Faros +91% review time; three attribution buckets and four saved views keep the queue moving.

Five subagents, five truths: orthogonal review without self-grading

Fri, 11 Sep 2026 00:00:00 GMT

Sep 11 field note: Abouwarda forks Claude Code review into five subagents that disagree on what is authoritative — ADR, spec, live HTTP, config — so the writer’s session never grades its own diff.

Software factories are five gates — the agent is the cheap part

Fri, 11 Sep 2026 00:00:00 GMT

Sep 11 Firecrawl field guide collates Stripe/Spotify/Shopify/Ramp/Faire: intake, isolation, tools, verification, merge policy — plus Microsoft’s merge-rate gap and Faire’s failed confidence-score filter.

Copilot review: shell-validated comments and an ensemble Lite path

Fri, 11 Sep 2026 00:00:00 GMT

Sep 11 GitHub changelog: Copilot code review auto-resolves addressed threads, runs shell tools behind the agent firewall, and switches Lite to a multi-agent ensemble with higher addressed-comment rates.

Cherny: Claude production code needs a higher bar than human code

Fri, 11 Sep 2026 00:00:00 GMT

Sep 11 Willison quote of Boris Cherny: Anthropic’s continuous verification stack — lint, tests, Claude e2e, daily fuzzers, automated review/security/refactor — is the price of agent labor at volume.

Augment's factory grew output 4.5× by automating wherever work piled up, starting with review

Fri, 11 Sep 2026 00:00:00 GMT

Sep 11 Akshay Utture, Augment Code: across 17,200 merged PRs from November 2025 to July 2026, size-adjusted output per developer rose 4.5× while median merge time fell from 11.2 to 3.1 hours and 14-day reverts fell from 1.9% to 0.4%. Review was automated first because its backlog was growing, and a human still makes every merge call.

Harness engineering: the AI SDLC is won outside the model

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 Waydev: five of six links from capability to shipped value are harness work — verification is the new bottleneck, autonomy is a blast-radius grid, and flow metrics beat adoption dashboards.

Commit gates fail when every verifier shares the same evidence

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10: VP-CONTROL shows cross-model votes over shared evidence still approve 62.9% of unsafe proposals vs 22.9% with an independent source — source diversity beats model diversity for agent commit gates.

Reviewer skills are the floor; agent management is the job

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 field note: after 30 days of agent-primary production code, Tamiz argues the craft moved upstream — specify interfaces, manage context, verify fit — and treating agents as juniors you only review leaves architectural debt.

Seventeen PRs, thirty-two corrections, zero production bugs

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 field report: design-first + adversarial plan review + parallel subagent diff review + browser gates — including an IDOR caught by another instance of the same model — before a human merge decision.

SaltBench: measure what the machine referee does to the agent

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10: a referee-gated protocol (proof kernel, verifier, or withheld tests) walls the agent off, probe-tests isolation, freezes predictions, and treats budget stop as halt — so method effects on coding agents are measurable, not narrated.

Rivera: one "prior reviewer: PASS" line cut an AI reviewer's bug catch from 51% to 27%

Thu, 10 Sep 2026 00:00:00 GMT

Rod Rivera's paired experiment on 180 known-label implementations: telling a small-model reviewer that a prior reviewer passed the code nearly halved the bugs it caught, and a FAIL line raised false alarms from 4% to 46%. Independence is a property of the request, not the call.

Reality is the final verifier — keep narrowing the two gaps

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 Krentsel/Zaharia/Stoica et al.: requirement gap + model gap unify reward hacking and hallucination; an assurance-revision loop uses deployment evidence because predeployment evals stay proxies.

One hour: let Claude Code write the software factory

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 poornerd: describe plan→execute→review stages; Claude Code ships a worktree-isolated orchestrator plugin — and surfaces silent agent-teams traps before the first hang.

OpenAI Agents API: managed Codex harness for days-long runs

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 public beta: OpenAI hosts the Codex harness — durable sessions, compaction, subagents, sandboxes — so teams stop re-homing long-running agent infrastructure every model launch.

PRAssistant’s 10–20% is a floor on queue speed, not a quality claim

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 field note: Microsoft’s attributed PRAssistant result (5k repos, 90%+ coverage, 600K+ PRs/month, 10–20% median PR completion time) measures when review starts — not whether what merged got better.

Build loops, not agents: the software factory that clears the plateau

Thu, 10 Sep 2026 00:00:00 GMT

LeadDev’s Sep 10 field guide: individual coding agents stall at 20–30% org gains; step-function lifts come from separated multi-agent loops—starting with PR→merge.

EvoHarnessBench: expanding the harness can erase yesterday’s wins

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 v2: 17 harness-evolution streams (802 tasks) — harness-induced forgetting, inconsistent self-evolving adaptation, and a retention↔adaptation tradeoff when tools/skills/agents keep changing.

Ecdysis: don’t train the harness on every failure you see

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10: harness evolution that separates model-specific accommodation from systematic harness repair via cross-task failure patterns — up to 1.84× faster training and +18.56% reasoning accuracy.

Overnight loops need a kill, park, or transfer owner

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 Alex Zhu: four named roles, a one-page session card, and a decision table — if nobody answers Slack, park or kill; never hope the agent finishes alone.

Headless Decode on Modal: overnight cron → morning PR inventory

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 Decoding AI: Paul Iusztin ships Decode headless on Modal — CLI, webhook, or nightly cron; parallel attempts push decode/ branches so morning review starts from finished work, not five open terminals.

Cursor Projects: coordinator that keeps working after the laptop closes

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 launch: cloud coordinator + shared context + subscriptions (Slack/PRs/schedule) so fleets keep shipping while you sleep — FOMO as product, review still the human gate.

A software factory still needs a review gate it can trust

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 CodeRabbit field note: green harness checks prove covered behavior; merge still asks whether the change should become part of the system — with review depth following cost of error and blast radius.

Background coding agents: verification is the product companies keep shipping

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 Cedric Teyton survey of Stripe/DoorDash/Ramp/Monzo/Checkout/etc.: durable runs + ephemeral workers, determinism inside the loop, fresh-context review gates — attention scarcity, not typing speed.

A software factory is the floor under fast agent hands — merge stays human

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 BlueLupin: ruleset, map, broker, human-on-diff before sessions; intake→verify→named reviewer; launching more agents fills the review queue — capacity is who drains it.

BenchShield: reward integrity needs infrastructure evidence

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 paper: lifecycle-model instrumentation for agent evals; full-chain recall 77–100% vs 23–94% hackability scanner; 96% runtime accuracy on labeled trajectories.

Authority as inventory — why the boring capability ceiling wins

Thu, 10 Sep 2026 00:00:00 GMT

CapScope’s lesson isn’t “spot the injection.” It’s freeze what an agent may do outside the model, then put an owner on that inventory. Fleet management of ceilings across a software factory is the natural next layer — and it’s exactly the kind of platform-dependency objective stewards are good at enforcing.

Governed agent loops: backlog to PR with Standards and AI Review

Thu, 10 Sep 2026 00:00:00 GMT

Atlassian’s Sep 10 ship note: always-on Jira agent loops plus org Standards and a dedicated AI Review agent — humans keep the merge button, not the bottleneck before it.

Comprehension debt is the review failure that still merges green

Thu, 10 Sep 2026 00:00:00 GMT

Sep 10 TNS field note: functionally correct agent PRs that cross domain boundaries slip past human review — Executable Architecture (pytest-archon / ArchUnit) makes boundary violations fail CI instead.

Don’t localize from the issue text: execute, then diff the runs

Wed, 09 Sep 2026 00:00:00 GMT

Sep 9 XAgent: differential analysis of failing vs passing executions for localization, plus context-aware test augmentation so validation doesn’t overfit the ticket — 62% resolve / 72.8% function loc on SWE-bench-lite.

Watermark the trajectory, not just the patch

Wed, 09 Sep 2026 00:00:00 GMT

Sep 9 TrajMark: ownership + segment-level tamper localization for coding-agent trajectories — because patch watermarks miss process provenance and local edits can keep a global ownership signal.

I stopped reviewing code and started reviewing agents

Wed, 09 Sep 2026 00:00:00 GMT

Sep 9 field note: when agents flip the write/read ratio, the real review target becomes permissions, tools, and who sits between output and production — not every generated line.

Static green is not secure: measuring the SPDF gap

Wed, 09 Sep 2026 00:00:00 GMT

Sep 9 paper: of 654 Bandit–Semgrep-clean Python samples, dynamic exploit verification confirmed ~1 in 7; several frequent CWEs flagged by neither scanner.

Salesforce’s agentic shift: maturity curve, then a swappable harness

Wed, 09 Sep 2026 00:00:00 GMT

Sep 9 Part 2: scaling agentic coding across ~15k engineers — nine-stage maturity language for managers, token discipline as quality, and a model-independent harness so routing/skills/context survive provider churn.

Independent review as a skill inside the coding agent

Wed, 09 Sep 2026 00:00:00 GMT

Qodo’s Agentic Toolbox (Sep 9) packages context, standards, and an independent review pass as skills coding agents can call before a PR exists — and again to resolve structured findings after.

Opair: the harness that refuses autonomy on purpose

Wed, 09 Sep 2026 00:00:00 GMT

Phil Booth’s Opair makes shared ownership the product goal — no shell, scoped tools, driver/navigator role swap — so the human stays in the loop before the 5k-line agent PR arrives for review.

Mouse verification loops: same model, 15/30 → 25/30

Wed, 09 Sep 2026 00:00:00 GMT

Sep 9 Mouse: on Runta’s FrontierHarness (Kimi K3 held fixed), OpenCode scored 15/30; Mouse’s deterministic post-turn verify loop hit 25/30 — checks, deleted-test block, MOUSE_AUDIT evidence, and cache-friendly retries.

The Factory: tickets in, gated PRs out — humans keep the merge button

Wed, 09 Sep 2026 00:00:00 GMT

Sep 9 mikeroySoft: label-driven always-on factory (triage→worktree→CI gate→evidence-bound review→merge); self-declared done is unreliable; 14 self-merged tickets; events.jsonl accountability, not autonomy theater.

On call for dark code: runtime is the only truth left

Wed, 09 Sep 2026 00:00:00 GMT

Sep 9 LeadDev field note: authorship, memory, and comprehension fail under agent volume; gate on detection and blast-radius containment, not per-line human recall.

Blind repair loops damage clean code

Wed, 09 Sep 2026 00:00:00 GMT

Sep 9 arXiv note: iterative LLM bug-fixers invent bugs in correct programs faster than they heal real ones, then oscillate forever — a stopping-condition problem for review-time APR.

IdeaAMBIG: the hard part of agent coding is finding what the spec omitted

Wed, 09 Sep 2026 00:00:00 GMT

Sep 9 benchmark: 660 evidence-grounded gaps in research-method specs — defect localization is the bottleneck; gold clarification lifts codification-ready rate from 14% to 98%.

AI review as first-class CLI objects, not browser tabs

Wed, 09 Sep 2026 00:00:00 GMT

Harness CLI 3.0 (Sep 9) puts risk scores, review groups, and check→pipeline log streaming in the terminal so agents can triage and close the review loop without the 12-tab dance.

When the answer to ‘how do you keep up with agent PRs?’ is ‘we’re not’

Wed, 09 Sep 2026 00:00:00 GMT

Harness field CTO on Sep 9: PR floods at 1.5–10× (sometimes claimed 50×), reviewers who should see the real change not the dependency scaffolding, and a delivery knowledge graph so review context is cheap.

Harness engineering: behavioral evals as the safety net

Wed, 09 Sep 2026 00:00:00 GMT

Google’s Sep 9 harness note: end-to-end benches won’t tell you why the score moved — unit-style behavioral assertions on tool use and intermediate steps let you iterate prompts and models without silent regressions.

Merge time tripled: acceptance is the scarce resource

Wed, 09 Sep 2026 00:00:00 GMT

A six-person team’s field numbers after agents arrived — PRs doubled, median time-to-merge went 4h→14h — and the social fixes that worked better than an AI review bot.

Enforcement the agent cannot edit

Wed, 09 Sep 2026 00:00:00 GMT

Hartman’s Consort (Sep 9) sorts agent frameworks by persuasion vs structure vs hard controls, then argues for immutable tests and gates on a live DB branch the worker cannot bypass.

Black Box’s PR gates — five reviewers before a human merge

Wed, 09 Sep 2026 00:00:00 GMT

Field note from Black Box via Codacy — ~8 coding agents in the loop, every PR through five independent reviewers, plus Verity’s markdown knowledge graph so later turns inherit what got rejected.

A-JIT: fill the hole at runtime, not in the PR queue

Wed, 09 Sep 2026 00:00:00 GMT

Sep 9 Marron & Barr: treat incomplete code as a first-class hole, synthesize from traces and typed IO pairs inside a harness+agent assembly — JIT for software construction.

Zakas’s five roles: where you stand on the agent spectrum

Tue, 08 Sep 2026 00:00:00 GMT

Observer → tech lead → architect → engineering manager (software factory) → product manager — pick deliberately by trust, blast radius, and how far you are from the code.

SWE-Bench Pro Verified: when the score was reward hacking

Tue, 08 Sep 2026 00:00:00 GMT

Sep 8 paper: anti-hacking + task refinement cut inflated SE-agent scores (GLM-5.2 78.80%→57.32%); gold-patch leakage via git, FS, network, and metadata.

Under agent PR volume, teams are redesigning review — not abandoning it

Tue, 08 Sep 2026 00:00:00 GMT

Gergely Orosz surveys how labs and startups actually handle the review flood — from reviewing the AI review, to blast-radius triage, to reviewing plans and schemas instead of diffs.

Pinishv: nine control surfaces — or you have an ungoverned factory that looks healthy

Tue, 08 Sep 2026 00:00:00 GMT

Sep 8 Pini Shvartsman series finale: agentic factory as nine human control surfaces (readiness → halt); five failure modes where metrics improve while judgment dies; F0–F4 maturity without buying a platform.

Mastra's factory started fully automatic, was too bursty to review, and went stage by stage

Tue, 08 Sep 2026 00:00:00 GMT

Sam Bhagwat on Mastra Factory, which now writes 25-35% of Mastra's PRs and closes 50-60% of its issues: full-auto mode flooded review when backlogs were imported, wrong upstream assumptions produced PRs that had to be closed, and the project now asks the community for detailed repros instead of small PRs.

The agent that never merges: overnight pipeline, human irreversible step

Tue, 08 Sep 2026 00:00:00 GMT

Sep 8 Allen Hutchison: scheduled auto-dev ticks triage→plan→build→address review; harness forbids merge — morning work is decide-to-ship, not reverse-engineer intent.

Harness or model? Same-model contrasts don’t crown a winner

Tue, 08 Sep 2026 00:00:00 GMT

Sep 8 Arjmandi: paired same-model harness contrasts on a private 256-task suite — averages near zero (−1.25 / +1.25 pp); Opus strata reverse by task type; 22/81 wall-clock cancels already had a passing patch.

ExecCritic: same trajectory writing patch and test is a confidence trap

Tue, 08 Sep 2026 00:00:00 GMT

ExecCritic separates Test and Repair agents, freezes fail-closed tests, and shows test quality decides whether execution feedback helps — composing trained roles hits 72.6% on SWE-bench Verified.

Fleet size is 1/f_H — supervise what you can verify

Tue, 08 Sep 2026 00:00:00 GMT

Sep 8 Brian Curry: six-layer fleet architecture; review-queue bifurcation; sustainable agents scale as the reciprocal of the human-routed fraction — gates, not headcount, buy overnight capacity.

Authority that lives outside the workspace still has to gate the mutate

Tue, 08 Sep 2026 00:00:00 GMT

Sep 8: cross-substrate authority gap — identical files can be safe or unsafe; planner observation fails, a deterministic execution guard at the mutation boundary catches unsafe publish intents.

CapScope: authority lives in the harness, not the prompt

Tue, 08 Sep 2026 00:00:00 GMT

CapScope freezes a task-wide authority ceiling and per-agent typed capabilities outside the model — injected effects drop from ~33–47/75 to 3/75 without tanking repair success.

Trust the spec, not the code

Mon, 07 Sep 2026 00:00:00 GMT

Sep 7 Farchi case study: natural-math specs + invariant proofs + runtime coverage; code is regenerable output the factory need not read line by line.

SCH: close the lid — durability in checkpoints, not the runtime

Mon, 07 Sep 2026 00:00:00 GMT

Sep 7: serverless coding harness on AgentCore microVMs — handoff/task modes, S3 checkpoints, Telegram approve; ~$3.45 compute over 12 days vs always-on EC2.

Scanning the harness: agent config is a supply chain now

Mon, 07 Sep 2026 00:00:00 GMT

Across 3,171 public repos, 16% of multi-component agent setups carry a confirmed security defect — unpinned MCP servers, scoped-looking shell grants, skills that pre-approve Bash for whoever installs them.

Permission with conditions: the new AI contribution policy

Mon, 07 Sep 2026 00:00:00 GMT

Sep 7 survey of 281 OSS AI policies: most allow AI code, but demand human involvement, accountability, and increasingly disclosure — plus ten countermeasures against AI slop at the PR gate.

It is not my code anymore

Mon, 07 Sep 2026 00:00:00 GMT

Sep 7 Camargo research note: authorship ≠ responsibility after AI-generated defects; acceptance evidence should start from the demand, not the diff.

Two months of scheduled routines: 82 merges, then review can’t keep up

Sun, 06 Sep 2026 00:00:00 GMT

Sep 6 Hodalog: Claude Code Routines grew 2→8; 119 Issues, 78% adopted, 95% PR merge — plus a weekly PR-care bot; adoption decision and Issue readability become the bottleneck.

Reviewer habituation shows up in embeddings, not lexical metrics

Sat, 05 Sep 2026 00:00:00 GMT

Yu et al. track 400 repeat reviewers of agent PRs: approvals climb while hand-crafted comment features stay flat — sentence-embedding centroids move, and language shifts lag the approval change.

7×24 Claude Code: five layers so the agent runs while you don’t

Sat, 05 Sep 2026 00:00:00 GMT

Sep 5 field build: cron+claude -p, hooks to phone, CLAUDE.md/auto-memory/resume, tmux+SSH, worktrees — from ‘Ferrari sitting inert’ FOMO to morning PR/CI already done.

Acceptance authority is not an agent permission

Fri, 04 Sep 2026 00:00:00 GMT

Wang and Liu’s Sep 4 theory note: Trustworthy Change as the engineering object, Human-Agent Cells that produce evidence without acceptance rights, and Responsibility Topology for who can absorb residual risk.

Five leaders, one split: review intent and risk — not every line

Fri, 04 Sep 2026 00:00:00 GMT

Sep 4 Rootly field note: Flare, Airbyte, CircleCI, Moloco, Twingate leaders on agent PR volume — spec-first review, risk tiers that skip low, earlier machine verification, and a named human who owns the merge.

Inside a software factory: eight stages, don’t overbuild

Fri, 04 Sep 2026 00:00:00 GMT

Paul Iusztin maps factory stages onto the SDLC, puts humans on brainstorm/plan/final check, and warns that first-version autonomy theater often fails.

Harness engineering: the repo enforces what prompts only suggest

Fri, 04 Sep 2026 00:00:00 GMT

Sep 4 Frisinger at 8th Light: verification is the bottleneck; fail-closed gates, evidence ledgers, and provenance turn overnight agent output into accountable inventory — not plausible debt.

1,135 agent PRs: shared blind spots and lying evidence

Fri, 04 Sep 2026 00:00:00 GMT

Lessons from an autonomous GitHub team: same-model author/reviewer blindness, dry-run evidence that lied, roles-as-data, structured envelopes, and refuse-to-guess gates.

Beyond generation: Verification Tax and Production-Qualified Change

Fri, 04 Sep 2026 00:00:00 GMT

Bhati’s Sep 2026 synthesis names the attenuation curve from agent commits to releases, and reframes output as PQC — changes that cleared review, test, security, and ops gates.

SWE-Gate: green tests are not a merge gate

Thu, 03 Sep 2026 00:00:00 GMT

He et al. build a 303-instance benchmark where review constraints from real PR comments are tested separately from functional tests — and 221 of 644 functionally green repairs still fail the review bar.

Codex Watch and fix: a 10-minute heartbeat that owns the PR last mile

Thu, 03 Sep 2026 00:00:00 GMT

Sep 3 RuntimeWire: OpenAI desktop Codex attaches a recurring heartbeat to a PR — re-reads live GitHub state, repairs CI and conflicts in a worktree, optionally merges; continuous ownership instead of one-shot heroics.

The psychological tax of AI in the eng org

Thu, 03 Sep 2026 00:00:00 GMT

Sep 3 case study (N=21): a year into AI adoption, engineers report accountability anxiety, craft-identity disruption, meaning erosion, and workload intensification — including around review.

Guild’s factory: specialized agents, separate reviewer, human merge

Thu, 03 Sep 2026 00:00:00 GMT

Guild’s Sep 3 Software Factory receipts — 34% of merged PRs, separate review agent before humans, maintenance agents that keep the loop alive, ~$12 model spend per merged PR.

The Agentic Pivot: work around code is the product

Thu, 03 Sep 2026 00:00:00 GMT

Atlassian research: 74% see faster generation but 78% still use traditional peer review; leaders want a governed system of work agents can write back into.

A-CX: one person, one week, 152 merged PRs, and the planning was where the hours went

Thu, 03 Sep 2026 00:00:00 GMT

Sep 3, Ilpo Niva at A-CX: unattended parallel agents closed 204 issues and merged 152 PRs in a week, but only against issues with an objective, design, checkable criteria, and dependency order. Worktrees, an atomic claim registry, and a fail-closed merge checkpoint keep it safe; about one in twenty machine findings is rejected on review.

LLM-as-a-judge is not an oracle—gate changes with deterministic checks

Wed, 02 Sep 2026 00:00:00 GMT

Eleven failure modes from production self-improving loops, and PROCTOR’s Teacher-Student design with hermetic sandboxes and frozen holdouts.

Laycock: stop waiting until the PR for the important talk

Wed, 02 Sep 2026 00:00:00 GMT

Thoughtworks CTO Rachel Laycock answers Houck — same Meta/DX crush numbers, different fix: shift judgment left via pairing and design, review by exception, don’t preserve PR ceremony with an AI pretend-reviewer.

HELIX: risk-tiered verification rungs, not more generation

Wed, 02 Sep 2026 00:00:00 GMT

Sep 2 Maya Shenoi methodology note: HELIX rebuilds the SDLC around verification contracts — R0–R3 autonomy, machine gate then adversarial AI then human intent review, and no merge without a verification record.

Secure the ADLC at its control points, not with a bolted-on AI review

Wed, 02 Sep 2026 00:00:00 GMT

Sep 2 Endor Labs field note: agent PRs break the one-reviewer-per-change ratio; put security context at generation, architecture-aware review on every PR, and reachability before remediation.

The AI code review packet: receipts for agent PRs

Wed, 02 Sep 2026 00:00:00 GMT

A practitioner template that forces intent, surface area, risk, evidence, edge cases, rollback, and review focus onto every agent-written pull request.

Six guardrails so an unattended bug-fix loop can leave the desk

Tue, 01 Sep 2026 00:00:00 GMT

Sep 2026 Brett Ridenour: draft PRs only, P2/P3-only autofix, never push main, halt on broken baseline or three consecutive fails, hard wall-clock cap — scar tissue as the safety model.

HarnessEvolve: diagnose against reference trajectories, then dual-gate the harness edit

Tue, 01 Sep 2026 00:00:00 GMT

Sep 1 Jiang/Chu/Tian et al. (Huawei): reference-guided root-cause vs failed runs; quality gate (leak/bloat) + performance gate (batch↑, recent↘ε); CloudCoreNetwork-QA 43.4%→86.9% (+21.6pp over GEPA).

HarnessDev: the unit of eval is the harness

Tue, 01 Sep 2026 00:00:00 GMT

Sep 1 benchmark: can models create and evolve their own agent harness? Generated code/search harnesses lag human ones; evolution gains are unstable and model-tied.

Harness-of-Harness: loops that separate build tests from independent eval

Tue, 01 Sep 2026 00:00:00 GMT

Sep 1 HoH wraps existing coding harnesses in planning–coding–testing iterations that force new capability each loop, schema-constrained artifacts, and an independent tester — +52% avg relative vs standalone harnesses.

Factory throughput without review capacity is a measurement failure

Tue, 01 Sep 2026 00:00:00 GMT

Sep 1 Faros field note: software factories coordinate agents, harnesses, and verification — and the 2026 Engineering Report shows AI adoption lifting throughput while PR review time jumped 441.5%.

Copilot code review can now approve — if you turn it on

Tue, 01 Sep 2026 00:00:00 GMT

GitHub’s Sep 1 changelog adds approval assessments on every Copilot review, plus an off-by-default path for Copilot approvals to count toward required reviews, scoped by path.

Checkly: harness first, then one overnight Claude rewrite

Tue, 01 Sep 2026 00:00:00 GMT

Sep 1 field report: 92M msg/day Node→Go with Claude Code overnight — golden-file black-box harness defined correctness; agents shipped ~13k lines; zero incidents after a queue-topology miss taught boundary fidelity.

Validation has to move left twice — not into a bigger PR

Tue, 01 Sep 2026 00:00:00 GMT

Sep 1 Allstacks / CircleCI field note: 28M workflows, main success 70.8%, merge efficiency 3.9→1.3; put checks in the agent loop and adversarial review in the spec.

CI under agent volume: the pipeline has to understand the change, not just run the suite

Mon, 31 Aug 2026 00:00:00 GMT

Aug 31 Pini Shvartsman: continuous integration → continuous verification — comprehend, choose risk-based strategy, investigate failures (no blind retry), emit an evidence package; escalate up, never auto-approve consequential deploys.

Schwarz: plausible specs are not enough—expose SMT-local repair

Mon, 31 Aug 2026 00:00:00 GMT

Aug 31 Ke/Wu/Li: agent proposes specs/lemmas; checker owns authority via SMT discharge + replay; 1367/1475 passes; local lemmas ablation −221 tasks; SV-COMP ReachSafety 91.5% vs CPAchecker 60.1%.

Staff the line: spec author, verifier, fleet ops, line owner

Mon, 31 Aug 2026 00:00:00 GMT

Augment’s org-design guide: agents absorb implementation, so scarce human work is spec, verification, halt authority, and line ownership — with receipts on review share collapsing under PR flood.

Commits doubled; verification didn’t

Sat, 29 Aug 2026 00:00:00 GMT

Signadot/TNS on GitHub’s 2.9B monthly commits: generation is machine-paced while runtime verification still queues through staging—and how per-change isolation closes the gap.

Verification surface: a tool only helps where its reach matches the failure

Fri, 28 Aug 2026 00:00:00 GMT

Controlled study of 1,116 agent-built web apps: boot probes, shells, and screenshots help only for the failure modes they can observe.

openJiuwen: static harnesses lose when long-horizon evidence keeps arriving

Fri, 28 Aug 2026 00:00:00 GMT

Aug 28 openJiuwen team: Rail-composed Inner/Outer loops + runtime adaptivity — SWE-bench Verified 82.6% (+3.4 pp) and Terminal-Bench 2.1 87.19% (+3.39 pp) over strongest selected leaderboard points.

Mouse: seven overnight requirements + house rules that grade events

Fri, 28 Aug 2026 00:00:00 GMT

Aug 28 field note: unattended coding needs verify-outside-worker, admissible evidence (not narration), budget reservation, resumable controller, and stop-at-merge — plus inject/copy/grade playbooks.

HarnessLens: verify the behavior you changed, not a random task batch

Thu, 27 Aug 2026 00:00:00 GMT

Aug 27 Xu et al.: budget-aware harness evolution with behavior-aware verification + attributable-evidence gate — 7.6–13.6% held-out gains at ≤200 interaction units vs thousands of fixed-set rollouts.

PILOT: live steering mid-run beats post-hoc harness updates alone

Thu, 27 Aug 2026 00:00:00 GMT

Aug 27 Xiao et al.: supervisor–worker harness redirects active workers and distils skills/memory live; +14.6/+12.4 pp Terminal-Bench self-improvement, tokens down ~43–47%.

WikiSkill: compile agent experience into reusable skills

Thu, 27 Aug 2026 00:00:00 GMT

Turns scattered optimization history into persistent skill knowledge so agents can reuse what they learned across tasks.

Uber’s software factory is now mostly agents — and they are optimizing the cost equation

Thu, 27 Aug 2026 00:00:00 GMT

More than 70% of Uber PRs are attributed to agents. Their new post is a field manual for measuring and cutting agent spend without starving quality, including how they bench uReview for code review.

MCR-Bench: multi-round code review is not a one-shot classify task

Thu, 27 Aug 2026 00:00:00 GMT

2,269 real multi-round review tasks with defect lifecycle labels—LLMs degrade as rounds grow and miss low-salience defects.

Harness: agent-ready SCM + AI review gates when PRs outrun humans

Thu, 27 Aug 2026 00:00:00 GMT

Aug 27 Harness: Code Repository scaled for agent PR volume + AI Code Review with risk-grouped diffs and required AI Checks — scoped agent identities so continuous factories don’t overwhelm human review.

After the factory works: trust, seats, memory, intent

Thu, 27 Aug 2026 00:00:00 GMT

Aug 27 Travis Frisinger: six months, ~1,700 merges — throughput is the honeymoon; fail-closed gates, calibrated seats, institutional memory, and scarce Purpose form PAAA.

Worktrees isolate files, not plans: Foremerge’s semantic merge tax

Thu, 27 Aug 2026 00:00:00 GMT

Aug 27 Woodhead/Foremerge: parallel agents in worktrees can merge clean and still break design — declare semantic scopes before edits; acceptance gated on harness-run checks, not agent claims.

37 agents, one afternoon, every test green, and two real bugs only a reviewer found

Thu, 27 Aug 2026 00:00:00 GMT

Aug 27 Ralph Duin, Inspired by Frustration: a lead agent that never edited a file ran 37 subagents across seven worktrees, finished 15 of 16 tasks for about $49, and merged nothing to main. The two bugs worth reporting got past 1,374 passing tests and were found by fresh-context reviewers who opened the rendered PDF and reproduced a stale cache.

Completion is a harness verdict, not a missing tool call

Wed, 26 Aug 2026 00:00:00 GMT

Aug 26 WonderLab / MyCodeAgent: single immutable loop + TransitionReason traces; completion gate PASS/UNVERIFIED/FAIL with feedback injection; terminal reasons exhaustively named.

Warp Factories that score themselves and open fix PRs

Wed, 26 Aug 2026 00:00:00 GMT

Warp’s self-improvement loop: scorers for efficiency/quality/compliance sample runs, then a scheduled agent proposes skill/config diffs from failures.

The coding session is too small a boundary for the work

Wed, 26 Aug 2026 00:00:00 GMT

Aug 26 Sumant Thakur: Spotify Xirp, AWS Kiro Crew, Warp Factories — durable state, isolation, event starts, and verification outside any one agent session.

When review alone no longer scales: layered supervision

Wed, 26 Aug 2026 00:00:00 GMT

Aug 26 Stolze & Strässle (ESEM 2026): five-practitioner study — preventive + executable + human layers; review 3–4× generation; promote recurring findings into lint.

Blocking is not diagnosis: the Verdict–Diagnosis gap

Wed, 26 Aug 2026 00:00:00 GMT

Aug 26 paper: MalPR-Bench scores attributable PR security review; CodeRabbit blocks almost as often as PRGuard but identifies fewer target vulnerabilities — especially when evidence sits outside the touched files.

Same model, different harness: the solver is the pair

Wed, 26 Aug 2026 00:00:00 GMT

Aug 26 Lewis (Yuj): paired control vs closed-loop harness on fixed weights — tight 20k Verified F2PF 28%→49%, solutions 43→72; frozen treatment transfers across four model designs; wide window mostly erases the gap.

Exit code 0 is not done: DoneGate’s fail-closed completion gate

Wed, 26 Aug 2026 00:00:00 GMT

Aug 26 DoneGate: worktree + path leases + independent validation — commit claim must match HEAD, allowlist/denylist, required checks, and execution evidence before completion.

Build a review agent: guidelines, guards, journal, judge

Wed, 26 Aug 2026 00:00:00 GMT

Cline walks through a two-pass PR reviewer on their SDK—record findings, then a stricter judge—plus deterministic tool guards so the reviewer stays read-only.

Foreman: four stations, cross-vendor review, draft PR

Tue, 25 Aug 2026 00:00:00 GMT

Aug 25 Vercel guide: eve software factory moves labeled work through triage → plan → implement → independent review on a different model vendor, with bounded request_changes loops and evidence against acceptance criteria.

Agentic engineering: trust lives in the system

Tue, 25 Aug 2026 00:00:00 GMT

Scott Spence’s practical definition: agents write; engineers own context, scope, validation, evidence, and review—with harness numbers from real projects.

468 merged PRs in 23 days from an always-on agent fleet: the validation stack is the boss, and AI review is capped at two rounds

Tue, 25 Aug 2026 00:00:00 GMT

Aug 25, Maurice Kleine. He used to be the message bus between 4 to 10 agent tabs and felt stressed whenever his laptop lid was closed. He moved the fleet to a €100/month server with supervisor agents that route tickets around the clock. 468 PRs merged and 537 tickets completed in 23 days. His first AI review setup nearly froze the fleet, so review cycles are now capped at two before a ticket blocks and comes to him.

StarHarness: stratified search, proposer-hidden selection, then held-out enterprise transfer

Tue, 25 Aug 2026 00:00:00 GMT

Aug 25 Esakkiraja et al. (ServiceNow/Mila): evolve prompts/tools/skills/MCP/subagents with failure-mode strata + hidden selection; +20–35 pp on ITBench / EnterpriseOps-Gym / AutomationBench, frozen transfer across GPT and Qwen.

Build-verify as control flow: 34% broken agent PRs → gate blocks the PR

Mon, 24 Aug 2026 00:00:00 GMT

Aug 24 Genovese/Regolo: OpenCode in GitHub Actions with a bash verification gate — pytest, forbidden paths, secret scan — PR opens only if success(); prompt alone is not enough.

Night shift: OS schedule → isolated worktree → morning accept/discard

Sun, 23 Aug 2026 00:00:00 GMT

Aug 23 Quiroga TaskKeeper: scheduled Claude/Codex runs in a fresh git worktree; VS Code closed; local review inbox with cost + diff; accept merges locally, discard deletes the worktree — never auto-push.

Factory metrics: generation vs review bandwidth

Sun, 23 Aug 2026 00:00:00 GMT

Augment’s metrics guide treats the agent fleet plus human reviewers as one production unit — Faros/LinearB receipts show PR volume up while review waits explode, and DORA’s verification tax names the spend.

Loop engineering in the wild: 217 confirmed always-on agent loops, almost no committed state

Sat, 22 Aug 2026 00:00:00 GMT

Aug 22 Lulla et al.: gray-lit definition + mine of 36,710 repos; 217 confirmed autonomous loops (mostly event PR-review/cron triage); configs committed, runtime state almost never versioned.

Harness → attention interface: human attention as the scarce resource

Sat, 22 Aug 2026 00:00:00 GMT

Dan McAteer on Latent Space traces harness eras from AutoGPT to Claude Code, then argues what’s left after models absorb scaffolding is an interface to human attention.

Put the gate at the station that produces the defect

Fri, 21 Aug 2026 00:00:00 GMT

Aug 21 Augment field note: agent PR volume turns merge-only review into terminal inspection — station-level quality gates (intake→spec→build→verify→ship) with named owners stop defects where generation is still cheap.

Closed-loop AI-to-AI review is real — and still a minority

Fri, 21 Aug 2026 00:00:00 GMT

Selvanayagam and Ghaleb link 248k agent-authored PRs that got at least one AI-attributed review on CodAGE — cross-product review is ~1.6% but grew two orders of magnitude from 2025-Q1 to Q3.

Human judgment doesn’t leave the factory—it relocates

Fri, 21 Aug 2026 00:00:00 GMT

Osmani on lights-on factories: encode taste in the environment, budget verification like performance, and keep ownership on what ships.

Mandible: 12k agentic review verdicts and the round-4 elbow

Thu, 20 Aug 2026 00:00:00 GMT

Aug 20 Watson Labs / Swamp: Gymwasp’s Mandible factory — 372 issues, 1,801 review rounds, 11,967 verdicts. 65% mergeable after one round; after round 4 the loop oscillates; 29% shipped on warn; volume 3× with median rounds flat.

Own the review standard as a skill, not a vendor slider

Thu, 20 Aug 2026 00:00:00 GMT

Tessl’s Aug 20 Code Review launch: team standards live as versioned skills you can inspect, eval, and point at another repo — whole-PR context, stateful re-review, and a path toward an autonomous merge gate.

Verification debt: when generation outruns trust

Thu, 20 Aug 2026 00:00:00 GMT

Aug 20 field note: verification debt as the queue behind cheap AI implementation; independent layers beat same-model self-checks; measure verified delivery throughput.

Bounded Loops: done is a gate verdict, never the worker’s claim

Thu, 20 Aug 2026 00:00:00 GMT

Aug 20 Bhardwaj et al.: worker + independent gate + declared budget; prove termination under repair and ledger-supported done; instrument finds 47 vacuous gates in shipped loops.

SemaPLC: agent cannot say done until the runtime agrees

Wed, 19 Aug 2026 00:00:00 GMT

Aug 19 arXiv 2608.18565: verification-gated PLC harness — spec check, real toolchain compile, live execution. Project-context tasks: 89.4% compile vs 52.2 dynamic behaviour — the gap is the argument.

Compressing file reads isn't the bill — tool-schema filtering is

Wed, 19 Aug 2026 00:00:00 GMT

Aug 19 Chen/Shi (Paritok): instrumented Claude Code/Codex sessions — tool filtering saves ~21–57K tokens/turn (linear); content compression ~2%/turn but compounds ~3,350 N²; single-shot SWE-bench quality ≠ multi-turn cost.

Start with the harness, budget the verification, and expect each feature to get slower

Wed, 19 Aug 2026 00:00:00 GMT

Aug 19 Addy Osmani, advice file in his reference software factory for Claude Code and Codex: a plain harness plus a clear issue gets surprisingly far, the factory layer earns its place through back-pressure and independent verification, and ten features with an independent verifier cost at least 160 minutes of suite time before any human looks.

Mandiant’s AVDH: skeptical validation for agentic code review

Tue, 18 Aug 2026 00:00:00 GMT

Google Mandiant details an agentic vulnerability discovery harness—threat model, entry points, hypothesis generation, multi-temperature validation, then human PoC.

Danjou on Yegge's Land Rush: the batch was too slow, but attribution was the cheap part

Tue, 18 Aug 2026 00:00:00 GMT

Aug 18, Julien Danjou (who runs a merge-queue company) grants that one-commit-per-green-build can't survive agent commit rates, but argues that landing 150 commits on a red main throws away the link from each failure back to the agent and intent that caused it.

RBB LAB: cap overnight agent runs at what you can review the next morning

Mon, 17 Aug 2026 00:00:00 GMT

A small studio on what changed when agents started running for hours unattended: every mid-task judgement moves into the brief, and the morning review queue, not launch capacity, sets how many runs are worth starting.

22 nights, 56 PRs, 54 merged: the overnight loop stalled on tickets, not code

Mon, 17 Aug 2026 00:00:00 GMT

Yohei Nakamura audited his attended overnight coding-agent loop against its records: 72 tickets, 56 PRs, 54 merged, 0 reverted. Ticket defects outnumbered code failures 4.5 to 1, 18 of 38 constraints were learned only by failing, and verification needed a layer that distrusts the evidence itself.

Agents ship to dev overnight unreviewed, and one ticket field decides which work may finish alone

Mon, 17 Aug 2026 00:00:00 GMT

Goat Squad Studios: plan by day, agent loops by night, review in the morning. Every ticket carries an autonomy field, a missing field means it waits, and only work a test can prove is allowed to reach done without a human. Production still moves through one PR a day that the human reads.

Screenshot as oracle: done requires a fresh capture, not green DOM tests

Sun, 16 Aug 2026 00:00:00 GMT

Aug 16 yureki_lab: Playwright capture harness + report.json (overflow, console errors, failed requests) + stale-hash gate — multimodal screenshots close the UI loop agents already have for pytest.

The overnight agent loop needs a half that stops, and a rule against well-reviewed fences

Sun, 16 Aug 2026 00:00:00 GMT

Cristiano Mazzella on his own overnight harness: a read-only verifier that records the command it ran, three deliberate ways for a run to stop, and why a rigorous, green, reviewable workaround can still be the wrong change.

More review agents can mean less real disagreement

Sun, 16 Aug 2026 00:00:00 GMT

A new paper finds naive multi-agent review collapses into false consensus, and that a smaller setup with an explicit critic does better.

Dependabot Janitor: overnight merges only when the suite can prove boring

Fri, 14 Aug 2026 00:00:00 GMT

ZenBusiness (Witt, Aug 14): nightly Dependabot Janitor — deterministic semver/scope filter, then Claude Managed Agent skills; ~250 auto-merges / 800+ parked with receipts across ~30 repos in seven weeks.

A routine that runs is not a job that got done

Fri, 14 Aug 2026 00:00:00 GMT

Aug 2026 Paul Kuo: Cherny’s 388→180 maintenance PRs vs 150 days of launchd/Cowork jobs — silent failure when green logs lie; verification must sit outside the executor.

Cole's dark factory: holdout validator and a perimeter the factory can't amend

Fri, 14 Aug 2026 00:00:00 GMT

Aug 14 coleam00: Archon cron factory — dumb label dispatcher, independent validate-PR holdout, governance files auto-reject; humans file issues and promote releases only.

Code review is evidence — it does not own the outer SDLC loop

Thu, 13 Aug 2026 00:00:00 GMT

Aug 13 Qodo field note: independent review re-enters judgment under agent volume, but lifecycle authority still decides whether the task may advance after commits, CI, repairs, and stale approvals.

24 CI-green merges, ≥9 live regressions: burst heroics vs gates

Thu, 13 Aug 2026 00:00:00 GMT

Aug 13 Claude Code #86278: ~12h autonomous session merged ~24 PRs with green required checks; user hit ≥9 live regressions the same evening — green CI ≠ production-qualified.

PR-to-merge is the factory bottleneck — specialize the loop

Thu, 13 Aug 2026 00:00:00 GMT

Aug 13 Augment: generation is cheap; review/verify/repair/decision are the drag. Cosmos Experts (Risk, Deep, Pair, Verifier, Fixer) keep working until a human can merge with evidence — loop engineering, not a better comment bot.

Evals judge; guardrails constrain; the harness runs the night

Thu, 13 Aug 2026 00:00:00 GMT

Aug 13 Winston/Voss (Arize): long-running agents fail where chatbots never go — separate eval vs guardrail vs harness, give judges the trajectory, and feed failure explanations back as the next engineering brief.

AI SDK’s factory authors 25–35% of merges — humans still own the stamp

Wed, 12 Aug 2026 00:00:00 GMT

Aug 12 Grammel/Dodds: issue→classify→analyze→implement→review→backport in sandboxes; 70–80% of closed issues via factory; reviewer efficiency is the product, not more agents.

The harness never believes the agent

Wed, 12 Aug 2026 00:00:00 GMT

Aug 12 Denis Morozov: 37 packets, 37 merges, $3.89 — planted tests, named invariants, harness-run gates; the implementer’s ‘verification’ block is recorded and ignored.

Nightwatch: Captain plans by day, fleet ships overnight

Tue, 11 Aug 2026 00:00:00 GMT

Aug 11 Conveo: /nightwatch skill — Helm plans, remote Field Orchestrator spawns worktree workers, cheap per-ticket verify loops, one green master PR after 7.5h on 3 agents.

OpenFactory: always-on orchestrator, separate Box for review

Tue, 11 Aug 2026 00:00:00 GMT

Aug 11 Ascii Box template: continuous factory Box queues GitHub/Linear/Sentry issues; build Box plans/commits/tests; fresh verification Box gets the branch as a Git bundle — humans merge, never the factory.

OpenLoopEvolve: the Loop Policy is a versioned asset—Champion–Challenger before activation

Mon, 10 Aug 2026 00:00:00 GMT

Aug 10 Wang/Li/Li/Li: externalize obs/plan/memory/act/verify/recover/stop/budget as Loop Policy Bundles; online+offline evolution with paired eval, robust gate, task-boundary activation, rollback—YC-Bench lifts returns and survival vs fixed π₀.

Four agents, 90 PRs a day, 15 merges: tier by risk surface and let revert rate move the line

Mon, 10 Aug 2026 00:00:00 GMT

Aug 10 steezr (Johnny Unar): an inherited client repo had 300+ open agent PRs and two exhausted senior reviewers. Path-based risk tiers took about 60% of agent PRs off human review; commit trailers recording provenance showed a 1.8% revert rate on auto-merged low-risk PRs against a 2.3% human baseline, and 11% on high-risk surfaces.

OpenCodeReview: inject determinism into uncertain review agents

Mon, 10 Aug 2026 00:00:00 GMT

Rule-guided dispatch, grounded file tools, and an asymmetric reflector that filters hallucinations—higher SEM-F1 at far lower token cost on AACR-Bench.

Dark factories: turn lights off only as far as proof reaches

Mon, 10 Aug 2026 00:00:00 GMT

Aug 10 Jeff Kazzee: stage-by-stage SDLC switches; verification buys autonomy; deterministic checks + independent judgment + real surface; a pilot of seven image calls with preservation and browser gates.

alibaba/open-code-review: battle-tested OSS review CLI and agent plugins

Mon, 10 Aug 2026 00:00:00 GMT

Open-source Open Code Review tooling from Alibaba’s internal assistant lineage, with Claude/Codex/Cursor plugins.

RETRACE: verify the patch by reconstructing the problem it actually solves

Sun, 09 Aug 2026 00:00:00 GMT

Aug 9 Li et al. (Concordia): training-free bidirectional reconstruct-and-verify after agent patches; +7.0 / +3.6 Pass@1 on SWE-bench Verified vs mini-SWE-agent; Self-Refine with similar budget loses.

Ouroboros: the harness evolves through reviewed commits—and Hope ran that loop for 161 days

Sat, 08 Aug 2026 00:00:00 GMT

Aug 8 Razzhigaev et al.: self-developing coding harness with free + experience-driven core evolution under a blocking diff gate; Opus 5 Terminal-Bench 2.1 86.74% audited; Hope—161 days, 1,085 self-mod commits, 63.5% recent review block rate.

When agents flood the queue, review capacity is the real gate

Fri, 07 Aug 2026 00:00:00 GMT

Aug 7 Assembled: agent PRs left humans behind (TTF review 3.5h→16h); dual-model auto-approver + policy ridge hit 2.4× merges, 3.5× large-PR throughput, 43% auto-approvals — without raising reverts.

Auto mode: 9× between interrupts, three PRs by morning

Fri, 07 Aug 2026 00:00:00 GMT

Aug 7 Anthropic: Nuro 10pm–5am hill-climb → 3 PRs; classifier vs permission fatigue; FOMO answered only when eval signal + deny rules + PR-as-boundary exist.

DVx Overnight Build: the handoff is the product, judged by cost per accepted change

Thu, 06 Aug 2026 00:00:00 GMT

DVx Field Manual 002 splits unattended agent work into three shifts (human decides, agent executes, human accepts), with green/amber/red autonomy and a morning review packet read tests-first.

Houck: the invisible output of review is the org

Wed, 05 Aug 2026 00:00:00 GMT

DX’s Brian Houck on Meta’s review crush under agents — RADAR’s risk-stratified automation works, but automating diffs without protecting knowledge transfer and intent debt just hollows the team.

Merge rate became AI coding’s scoreboard — and the boards don’t agree

Tue, 04 Aug 2026 00:00:00 GMT

Aug 4 Pipeline Mag: four 2026 merge-rate studies disagree; rejections are mostly silence; Faros shows review time +441% and churn +861% under heavy AI — throughput ≠ what survived.

Overnight agents: infrastructure, not a longer Claude session

Tue, 04 Aug 2026 00:00:00 GMT

Aug 4 field guide (updated Sep 14): task queue → worktree-isolated Codex exec/Goal Mode → watchdog → morning triage — atomic verifiable tasks, AGENTS.md constitution, and when not to go lights-off.

EA-Graph: anchor verification claims to artifact content, or admit unprovable

Tue, 04 Aug 2026 00:00:00 GMT

Aug 4 Hsu/Chi/Everett: sub-path anchors + freshness≠evidence; after upstream drift Haiku ANCHOR F1 median 1.0 vs PROSE 0.27; refuse when support withheld—never guess.

Overnight delivery at Tilt means review-ready by morning, not merged

Tue, 04 Aug 2026 00:00:00 GMT

Aug 4 HILLS Lab field note on how one engineer runs AI-assisted delivery overnight at Tilt: a written definition of done before the run, a fresh agent session per ticket, isolated worktrees, explicit stop conditions, and a morning manifest of checks, blockers and the decisions still needed from a person. Merging and deploying are separate outcomes that need their own authorisation.

Astro’s issue factory: triage as a label-driven agent pipeline

Tue, 04 Aug 2026 00:00:00 GMT

Cloudflare/Astro cut open issues from 200+ toward zero with isolated subagents that reproduce, diagnose, and ship preview fixes inside GitHub Actions.

ADLC: software factories need Workflows, not babysat SDLC steps

Tue, 04 Aug 2026 00:00:00 GMT

Aug 4 Cloudflare: replace human-paced SDLC with Agent Development Lifecycle — programmatic, preview-per-agent, event-driven Workflows that spawn heal/review agents; @cloudflare/ci self-heals and stages fixes for merge.

HarnessCompass: evolve the harness under a generalization gate

Mon, 03 Aug 2026 00:00:00 GMT

Aug 3 Zhang et al.: constrained + proactive-feedback + component-wise harness evolution — SWE-bench Verified 54%→66% Pass@1 in 5 turns; held-out 60.4% vs AHE 54.7%; frozen harness lifts Claude Sonnet 4.6.

Yegge's Land Rush: 40 agents, 175 commits a day, and a merge queue that stopped bisecting

Mon, 03 Aug 2026 00:00:00 GMT

Aug 2026, Steve Yegge: running 40-plus agents around the clock against a half-hour build, his bisecting merge queue grew past 100 requests and stalled. His fix is to land 120 to 150 commits on main at once and have agents repair whatever goes red.

Agentic software factory: history, then check the utilities

Mon, 03 Aug 2026 00:00:00 GMT

TrueFoundry’s field guide: factories aren’t new; agent-native ones need model supply, credentials, QC, telemetry, and spend metering—not just more agents.

Codex harness files: plans are not evidence

Mon, 03 Aug 2026 00:00:00 GMT

Aug 3 Tony Alaniz (OpenAI Cookbook): phased harness/build files, human gates, and an append-only build-log that records observed verification — so long-running agent work cannot mark a phase complete on narration alone.

LongHorizon-Harness: only audited facts cross the round boundary

Mon, 03 Aug 2026 00:00:00 GMT

Aug 3 Ma et al.: Manage–Execute–Audit keeps task state outside the growing session; fresh-context executors, read-only auditors, and Claude Code/Codex backends — WeaveBench 51.8%→80.7% with the same model.

Claim Plane: pre-write admission buys reliability—mostly by serializing

Sun, 02 Aug 2026 00:00:00 GMT

Aug 2 Nikolaev confirmatory: static Claim Plane lifts pair pass 23.3%→50% and integration 65.6%→96.7% on 30 CooperBench pairs—but serializes 96.7% of runs; dynamic scope fails closed on undeclared regions.

Five hours of agent work only holds up behind critique gates

Sat, 01 Aug 2026 00:00:00 GMT

Aug 1 Sudarshan Shenvi: testable requirements, a separate critic that refuses partial passes, always-on sandbox with permissions off — review the output, not the babysitting.

LoopsBench: end-state pass rate misses the loop—best config resolves 25%

Fri, 31 Jul 2026 00:00:00 GMT

Jul 31 Li et al. (Microsoft): 112 dependency-DAG tasks, 5,300+ units, flow-aware test release + live regressions; Opus-4.7 + Claude Code + outer continuation tops out at 25% resolve.

ECLoop: hold the edit until the trajectory shows the evidence

Thu, 30 Jul 2026 00:00:00 GMT

Jul 30 Xu et al. (Concordia): evidence-conditioned execution layer gates premature commitment on SWE-bench Verified; +4.8–11.8 Pass@1 across two models and two scaffolds while cutting tokens up to 12.1%.

Tycho: never accept the model’s summary — only artifacts it can’t narrate

Thu, 30 Jul 2026 00:00:00 GMT

Jul 30 Swail Tycho: completion-hook ledger that checks files, commands, exit codes, and claims against git/FS — no LLM in the trust path; VERIFIED/FAILED/STALE for overnight review.

Parallel Claude lanes need a local merge queue — and a human promote

Thu, 30 Jul 2026 00:00:00 GMT

Jul 30 funador: FIFO land + checkCommand for multi-worktree Claude Code; promote is human-only; slow gates cap landings/hour; checkCommand≠human review (on purpose).

The expensive overnight run is the one that never crashes

Wed, 29 Jul 2026 00:00:00 GMT

Jul 29 Toolshed Labs Nightshift: four bash habits — rewrite handoff, fail-closed verify-gate, runaway caps, morning brief — so unattended agents stop producing confident unverified inventory.

Shopify: the harness outlasts the model

Wed, 29 Jul 2026 00:00:00 GMT

AppSec’s Dispatch pipeline — partitioned hunters, cross-model verifiers with real integration-test oracles, then fix PRs — so better models don’t just dump confident noise on developers.

nokto: plan → worktree → independent review → PR — never auto-merge

Wed, 29 Jul 2026 00:00:00 GMT

Jul 2026 noktohq/nokto-agent-orchestrator (still shipping Sep 21): Claude/Codex/Gemini delivery loop with allowlisted verify, secondary review, JSONL audit, resume; human merge is hard — PRs carry the receipt, not a silent land.

LinearB 2026: agent PRs merge less without owners; AI review lifts yield

Tue, 28 Jul 2026 00:00:00 GMT

Jul 28 LinearB 2.7M-PR benchmarks: elite orgs merge agent PRs at 79% vs 92% human-only (37% at top 60%); ownership — not model skill — is the limiter; AI code review lifts PR yield up to +5–7pts while generation moves the bottleneck downstream.

CHILL-Harness: don’t always-full the overnight loop — authorize only advantageous interventions

Tue, 28 Jul 2026 00:00:00 GMT

Jul 28 Fu et al.: causal CIEL+ARCO orchestration — GAIA 71.3% at 28.4% fewer tokens; SWE-Verified 65.6% at 13.1%↓ tokens / 25.9%↓ time; Always-Full ablation collapses GAIA to 27.7%.

ralphctl: green overnight means nothing if the agent edited the tests

Mon, 27 Jul 2026 00:00:00 GMT

Jul 27 Lukas Grigis ralphctl 0.17: generator–evaluator ralph harness; evaluator must audit diffs for verification tampering and cite hunks — FAIL and PASS both need evidence.

Audit the deterministic harness: hooks fail open; CLAUDE.md is a request

Mon, 27 Jul 2026 00:00:00 GMT

Jul 27 John Young: inventory vendor defaults for permissions, hooks, sandbox, telemetry — exit 1 ≠ block; Cursor failClosed defaults false; sell blast-radius control, not quality.

AgentRelay: overnight control is PreToolUse blocks + evidence-tagged morning reports

Mon, 27 Jul 2026 00:00:00 GMT

Jul 27 MalyStern: local Rust supervisor for Claude/Codex/ACP — block forbidden tools before they run, gate done on real tests, undoable checkpoints, budget kill-switch, VERIFIED/RECORDED/ESTIMATED report lines.

agent-sessions: freeze the checks, grade separately, stop at the merge gate

Sat, 25 Jul 2026 00:00:00 GMT

Jul 25 lmorchard: unattended issue→PR harness — acceptance criteria frozen before work; separate read-only grader; auto-ok vs needs-review tiers; eligible-for-auto-merge is a finding, never a machine merge.

The Trust Ladder: review autonomy is earned, measured, and demotable

Thu, 23 Jul 2026 00:00:00 GMT

July field note: Elstner’s four-rung promotion architecture for AI code review — shadow → advisory → soft-gate → hard-gate — on trailing precision, availability, latency, and false-block rate, with silence treated as failure.

Verify-first: 46% of agent fix PRs never merge

Wed, 22 Jul 2026 00:00:00 GMT

Jul 22: AIDev shows 46.41% agent fix-PR rejection; instructions alone don’t lift merge rate — put rules/tests/human gates inside the harness before more tools.

Orbital Witness grades its harness: merged PRs per engineer doubled, a nightly bug hunt, and no measure yet of human attention

Wed, 22 Jul 2026 00:00:00 GMT

Jul 22, Andrew Thompson (CTO, Orbital Witness). Scored against Ryan Lopopolo's twelve harness-engineering theses, Orbital's harness roughly doubled weekly merged PRs per contributor on a legacy repo, to mostly 6 to 7, at the same review and CI bar. A nightly bug hunt has sceptic agents try to refute each finding before a fix ships for morning review. Three theses are only half-built: turning feedback into infrastructure, retiring nightly loops, and measuring attention rather than cost.

Nightgauge: issue-to-PR overnight — six stages, twenty-one deterministic gates

Wed, 22 Jul 2026 00:00:00 GMT

Jul 22 Nightgauge (open-core): local issue→PR pipeline with isolated stage handoffs, build-as-hard-gate validation, batch overnight runs, cascade circuit breaker — nothing ships unmeasured.

Overnight agents that never push: Magrathea’s read-only worktree factory

Wed, 22 Jul 2026 00:00:00 GMT

Jul 22 Magrathea field note: labeled-issue dispatcher, per-issue worktrees, test/build/cdk gates, and a hard no-push boundary—defer colliding work, report empty queues honestly, leave closeable bugs for humans.

Cross-model review is asymmetric: Claude reviewing Codex helps

Wed, 22 Jul 2026 00:00:00 GMT

Controlled pairings of Claude and Codex as writer/reviewer show one direction lifts pass rates; the reverse can hurt.

Verification loops as skills: encode the morning tax once

Wed, 22 Jul 2026 00:00:00 GMT

Jul 22 Anthropic (Delba de Oliveira): turn the manual checks you keep repeating into Claude Code skills — standalone, embedded, chained, or on every PR — so modest continuous verify runs without relying on memory.

Light and dark factories: verification is the neck

Wed, 22 Jul 2026 00:00:00 GMT

Jul 22 Addy Osmani: loop → harness → factory; dark ships unread diffs; lit keeps judgment on the outer loop — autonomy only as wide as cheap, hard-to-fake checks.

Harneloop: portable harness units with artifact-aware tests and evidence-gated promotion

Tue, 21 Jul 2026 00:00:00 GMT

Ker102 Harneloop (v0.0.2): self-evolving harness units — capture artifacts/traces, diagnose, candidate patches, promote only with evidence; ViperMesh case study: 2.53× speedup, +8.19 visual, −90.9% tokens vs Blender MCP baseline.

Anthropic’s security job becomes monitoring loops

Tue, 21 Jul 2026 00:00:00 GMT

Deputy CISO Jason Clinton on securing an SDLC where Claude authors ~80% of merged code — multi-agent PR review, risk-tiered automation, shadow mode, and identity boundaries instead of prompt trust.

Custom Code Review rules for Codex via AGENTS.md

Mon, 20 Jul 2026 00:00:00 GMT

July 2026 developer blog: repository rules in AGENTS.md that Codex Code Review can cite on findings.

Two unattended days: gate completion on artifacts, not reports

Mon, 20 Jul 2026 00:00:00 GMT

Jul 20 Solar/Favur: six-session 2048 SOW run — false completion, oscillation, context exhaustion, review cost; validators cite docs/hashes; play-the-game signal beats green suite; task drift still open.

Cursor’s agent factory: merge is a policy over evidence, not a vibes check

Mon, 20 Jul 2026 00:00:00 GMT

Jul 20 Verdi/Gilhuly (Arize Observe): video demos as trust units; ~30–40% ecosystem PRs merge without human review; risk-scored auto-merge; Bugbot learns from corrections; self-driving PRs.

Ten agents overnight, ten PRs by morning, and review fatigue that looks like continued work

Sun, 19 Jul 2026 00:00:00 GMT

Jul 19, Xinwei Xiong. Ten parallel overnight tasks produced ten convincing PRs; by lunch he had reviewed three, the eighth he just merged, and that night he queued ten more. A week of about three times his usual merged code shipped no extra features. His fix is raising first-pass correctness, not adding agents.

Nightshift: a clock-out gate keeps the punch list a contract, not a vibe

Sun, 19 Jul 2026 00:00:00 GMT

Jul 2026 Orwa Mahmoud: Claude/Codex/Cursor shifts with persistent punch list, park-don’t-ask, watchman recovery, morning receipts — AdaptTable overnight: 9 items, gate-green commits, local until human push.

Warden: overnight fleets are easy — the morning merge train is the product

Sun, 19 Jul 2026 00:00:00 GMT

Jul 19 Warden: worktree-per-unit agents, clean-room verify + fail-closed AUDIT PASS/FAIL, then digest ranks survivors and integrate re-runs every gate after each merge — base branch untouched.

Hard execution boundaries: the model proposes; the harness decides what runs

Sat, 18 Jul 2026 00:00:00 GMT

Jul 18 Stanley Yang: vendor-neutral coding-agent harness — disposable workspace pinned to a commit, typed tools, external budgets, independently captured patch, trusted verification; model never owns shell, credentials, or success.

Latch: converge agent PRs to mergeable, then stop — a human always merges

Sat, 18 Jul 2026 00:00:00 GMT

Jul 18 Latch: independent reviewer≠fixer identities, recursion-guarded fix pushes, 3-cycle cap, anti-tamper on workflows/policy — clears the agent-PR firehose without ever pressing merge.

What a software factory is: four observable properties

Sat, 18 Jul 2026 00:00:00 GMT

Factory.ai’s definitional guide: standardized inputs/tooling, measurable output, replayability—and how agent-native factories change the throughput bound.

Agentic reviewers in the terminal: better localization, and re-checking that predicts failure

Sat, 18 Jul 2026 00:00:00 GMT

Melbourne, Monash and Atlassian researchers ran Claude Code, Gemini CLI, Rovo Dev and CodeRabbit on 362 human-verified review tasks. Repo-grounded agents roughly doubled precise, human-aligned comments over static LLMs, but exploration and validation dominated cost, and failed reviews carried more validation, not less.

Night Agent: spend the window before credits reset

Fri, 17 Jul 2026 00:00:00 GMT

Jul 2026 jooray/night-agent: reset-anchored unattended missions in an isolated worktree; separate verify+commit per fix; never pushes — FOMO operationalized as using the subscription before it evaporates.

Zach Wills: ten PRs merged overnight, review became inspecting the proof, and then the RAM ran out

Thu, 16 Jul 2026 00:00:00 GMT

Jul 16, Zach Wills on a year of running agent fleets: an overnight run merged 10 PRs in dependency order, proof-carrying PRs pushed his review ceiling to about 40 a day, and the next wall was a 32GB machine with 27GB of swap.

The production harness: walk away to a draft PR or a BLOCKED receipt

Thu, 16 Jul 2026 00:00:00 GMT

July 2026 white paper from David Reed, PhD: a Claude Code harness where done is an Evidence Ledger, the maker never grades the diff, and autonomy tops out below merge — zero auto-merges by design.

Proof-or-Stop: lifecycle advances only on fresh evidence

Thu, 16 Jul 2026 00:00:00 GMT

Jul 16 Huang et al.: treat reviewed/tested/DONE as claims; admit transitions only with source-bound, mechanically verifiable receipts — zero false-DONE in 10/10; hidden-fail amplification 31→2 per 1,800.

18 of 20 tickets in one night, two human gates, and every failure fixed in the harness

Thu, 16 Jul 2026 00:00:00 GMT

Jul 16 (updated Sep 10) Łukasz Graliński, Pirxey: a one-person software factory ran 50 tickets in a week for about $2.90 each, including a 01:12–05:47 night run that finished 18 of 20. Humans approved every plan and merged every PR. The four things that broke were fixed in deterministic code, not prompts.

Least privilege for agents: delegated identity, ephemeral tokens, MCP intersection

Thu, 16 Jul 2026 00:00:00 GMT

Harness field note: an agent holds no standing privilege — scoped subset of the triggerer’s access, server-side RBAC/OPA, and connector∩agent tool allow-lists on every call.

Warp on self-improving code review inside the factory

Wed, 15 Jul 2026 00:00:00 GMT

Companion Warp engineering note: how cloud software factories close the loop specifically on review quality and agent compliance.

HarnessBank: gene-bank search plus gated screening for trustworthy harness self-evolution

Wed, 15 Jul 2026 00:00:00 GMT

Jul 15 Luo et al.: semantic Harness Gene Bank + validity/activation/significance/gain gates — +5.1% to +15.4% Pass@1 across seven benchmarks; beats GEPA/DGM under the same paired-2σ ruler.

AgentLoop: stop trusting ‘done’ — grade against a sealed held-out oracle

Wed, 15 Jul 2026 00:00:00 GMT

Jul 2026 AgentLoop: harness-agnostic verify-gate wrapper; sealed held-out grading; claimed +16–22% SWE-bench-Verified lift over raw Claude Code / OpenCode by killing false-greens.

AgentLoop: done is a sealed oracle, not the model's DONE

Wed, 15 Jul 2026 00:00:00 GMT

Jul 15 AgentLoop: harness-agnostic verify gate with held-out sealed cases — +16–22% SWE-bench-Verified lift over raw Claude Code/OpenCode by killing false-greens.

Harness engineering, measured: eleven production coding agents at source

Wed, 15 Jul 2026 00:00:00 GMT

Jul 15 Barbaste et al.: source anatomy of Claude Code, Codex, Gemini CLI, OpenHands, Pi, OpenCode, and peers — seven subsystems, 29 patterns, and the harness→platform turn.

Sloop: overnight running hours, evidence verdicts, opt-in merge train

Wed, 15 Jul 2026 00:00:00 GMT

Jul 15 hamish-mackie/sloop: Rust daemon schedules Claude/Codex/OpenCode tickets into isolated worktrees inside configured running hours (e.g. 22:00–06:00), judges from observed exits/commits/tests — not agent claims — with build→review→merge flows and an opt-in verify-before-FF merge train.

Your coding agent said done. Who ran the tests?

Wed, 15 Jul 2026 00:00:00 GMT

Jul 15 Cortés / VichuFlow: agent self-report is not evidence — durable state outside the chat, kernel-run gates, stage advances only on artifacts; worker grades nothing.

Dark factory test: independent context and independent evidence

Wed, 15 Jul 2026 00:00:00 GMT

Jul 15 Camplight/Kacharov: METR harness ≈ coin-flip vs naive loop; self-review can degrade accuracy; dark only where checking is cheaper than doing — and gates still rot if uninspected.

Harness evolution gains might just be repeated sampling

Tue, 14 Jul 2026 00:00:00 GMT

Jul 14 Wang et al.: under matched budgets on Terminal-Bench 2.1, automatic harness evolution fails to beat parallel sampling / sequential refinement; held-out transfer only +0.6 pass@1.

A million PRs later: agentic review is faster, not clearly better

Tue, 14 Jul 2026 00:00:00 GMT

Study of 1.02M reviewed PRs across human, LLM-assisted, and agentic eras—efficiency gains without matching quality gains.

Turn accepted review comments into persistent behavioral rules

Mon, 13 Jul 2026 00:00:00 GMT

Closed-loop framework that codifies review feedback into versioned rules and a self-review checklist—0% recurrence for ruled-against errors in deployment.

Ready for Agent: label the issue, let the harness babysit to merge

Sun, 12 Jul 2026 00:00:00 GMT

Jul 12 berenddeboer/ready-for-agent: local clanker harness turns ready-for-agent issues into PRs across Claude/Codex/OpenCode/Grok — Merge Policy Off/Classify/Always, forge-agnostic, aimed at 150+ PRs/week without session babysitting.

Unattended loop cannot bless its own done — fences outside the model

Sat, 11 Jul 2026 00:00:00 GMT

Excubitor (Shepherd): mechanical PreToolUse guards + falsifiable intent records so a headless coding loop cannot merge, push, or delete on evidence it authored itself; frozen-oracle verification; candidate branch only.

Agentic code that merges still costs more to fix, and skipping review makes it worse

Fri, 10 Jul 2026 00:00:00 GMT

Jul 10 Xia and Miller: a year of commits across 182 repos shows agent-written lines get corrective fixes at a 49% higher rate than human lines, a rising agentic share predicts more human bug-fixing the next month, and projects that merge more without review carry a heavier agentic maintenance load.

Prusik: prove execution evidence — exit 0 with zero tests is NOT PROVEN

Fri, 10 Jul 2026 00:00:00 GMT

getprusik/prusik: independent verify outside the agent — prove parses runner output (executed ≠ skipped), prove-red requires acceptance tests fail without the change; 1,300+ tests, 43 field findings verified-closed.

Greptile: review as a slice of automated validation

Fri, 10 Jul 2026 00:00:00 GMT

Greptile argues PR review is one surface of a larger goal—codebase knowledge bases that catch second-order bugs beyond the diff.

AfterVibe: recover specs and review those, not just the code

Fri, 10 Jul 2026 00:00:00 GMT

Extract NL specs from vibe-coding sessions and validate them by blind regeneration—specs as the human-review artifact when AI code outpaces review.

verify-gate-mcp: the agent that did the work can never mark it complete

Thu, 09 Jul 2026 00:00:00 GMT

Jul 9 Monkeyattack MCP: criteria-at-birth, claim_done with evidence, verify hard-rejects E_SELF_VERIFY; pending→claimed→verified|rejected is forward-only in SQLite.

The PR Hound: an always-on agent that barely reads code and nearly doubled human merges

Thu, 09 Jul 2026 00:00:00 GMT

Jul 9 Andrew Joia (LangWatch): after Claude Code pushed more code in three months than the prior year, PRs rotted unreviewed. A twice-daily agent that gives every PR exactly one owner and one reviewer took human merges from about 10 to 18.5 a workday.

When coding benches rot: auditing SWE-Bench Pro

Wed, 08 Jul 2026 00:00:00 GMT

OpenAI’s July 2026 analysis of evaluation flaws in SWE-Bench Pro—and why noisy tasks poison agent comparisons.

Code Janitor: always-on cleanup that never floods the merge queue

Wed, 08 Jul 2026 00:00:00 GMT

Adapt’s recurring Code Janitor: hourly/daily scan → one small CI-gated cleanup PR at a time, memory-repo dedupe, human merge only — modest verified throughput as the product.

Review is the control point — AI does not pick the sign

Wed, 08 Jul 2026 00:00:00 GMT

Agarwal et al. code 3,100 practitioner docs into a causal model: agent impact on software is decided through how the team structures review and what expertise humans still bring.

SWE-Review: closing the generate–review–revise loop on agent PRs

Tue, 07 Jul 2026 00:00:00 GMT

July 2026 paper introducing agentic accept/reject review with SWE-Review-Bench and trajectory datasets.

Aria: 4,257 Iris lemmas proved by harness + code agent, zero expert intervention

Tue, 07 Jul 2026 00:00:00 GMT

Jul 7 Kan/Kan/Ertel: Claude Code + declarative HHL harness (kernel verify, no-Admitted, timeout, coverage) — full Iris core, 217 RustBelt, 318 reglang, 72 iris-lean; ~380 model-hours unattended.

Agents Never Sleep: PARK the hard ticket, don’t freeze the night

Mon, 06 Jul 2026 00:00:00 GMT

Jul 2026 TokonoMix ANS: unattended ASK→PARK autonomy contract, deterministic test gates + git revert, watchdog resume, morning report that separates trusted done from daylight review.

Moderne / Morgan Stanley: AI plans, deterministic tools execute

Mon, 06 Jul 2026 00:00:00 GMT

Morgan Stanley’s OpenRewrite year at Code Remix — review is the bottleneck, risk-based PR routing via DDRA-style scoring, and agents that invent recipes while CPU tools run the blast-radius map.

Merge-coordinator: 24/7 arming that never bypasses your checks

Sun, 05 Jul 2026 00:00:00 GMT

Jul 5 trimmdev: workers/agents open PRs; a GitHub Actions coordinator arms, serializes risky lanes, revalidates stale heads, re-arms when GitHub drops auto-merge — and cannot merge past required checks.

feature-factory: verification only where a false green can hide

Sun, 05 Jul 2026 00:00:00 GMT

Carreira’s feature-factory: durable run.json transitions, parallel build / serial merge, reviews bound to the judged commit, observed verify after each slice — autonomy without self-reported done.

VV Harness: sort every rule by enforcement tier — exit codes beat prose

Sat, 04 Jul 2026 00:00:00 GMT

Jul 4 Eftimie: harness = list of things you stopped trusting the model to do; mechanical/prompted/structural/instructional ladder; scope hooks, completion gates, fixture-tested rails.

Merge only when out-of-band reviewer and tester both pass

Fri, 03 Jul 2026 00:00:00 GMT

Jul 2026 Gus IT defensive publication: sandboxed authors cannot push main; an external train merges only on fail-closed ledger verdicts from separate reviewer and tester personas.

Is agentic review helpful? Mining CodeRabbit feedback in the wild

Fri, 03 Jul 2026 00:00:00 GMT

Empirical mining of developer reactions to CodeRabbit reviews—field evidence on when agentic PR comments help or annoy.

Tian Pan: the morning review queue turns overnight agent work into reconstruction, not review

Thu, 02 Jul 2026 00:00:00 GMT

Jul 2 Tian Pan: unattended agents move the bottleneck to 9 a.m.; assign trust tiers at dispatch, demand a provenance receipt with a deferred list, and batch the night's questions into one decision queue.

nightcrew: BACKLOG in, typed failures out — verify before any land

Thu, 02 Jul 2026 00:00:00 GMT

Jul 2 nightcrew: overnight control plane around Codex — plan from markdown backlog, worktree isolation, deterministic verify, independent merge review, streak guards, morning report.

Enterprise 2× mandate: throughput doubled, and review moved onto automation

Thu, 02 Jul 2026 00:00:00 GMT

Jul 2 CMU/Stanford longitudinal case: 802 developers, 196k PRs. Per-developer throughput hit 2.09×; reviewer load ~2×; human review coverage fell 89%→68% while automated review rose to ~84%. Merge and revert stayed flat on coarse proxies.

harnesswright: an agent's done is a claim — the gate returns a receipt

Thu, 02 Jul 2026 00:00:00 GMT

Jul 2026 Falco: zero-runtime CLI that never runs agents; ledger slices + verity claims reconciled to filesystem, git HEAD, and exit codes — exit 0/1/2; dogfooded 18+ green claims that caught its author.

Zach Lloyd: software engineering becomes factory engineering

Wed, 01 Jul 2026 00:00:00 GMT

Latent Space interview/dispatch energy around Warp’s CEO on factories as integrations into existing workflows—not a new IDE religion.

Takayuki Kawazoe: an always-on agent line merged nothing for three days because a safety check could no longer see standing approvals

Wed, 01 Jul 2026 00:00:00 GMT

Jul 1, Takayuki Kawazoe on Codens' round-the-clock agent pipeline: every PR approved and green, zero merges for three days, caused by a review freshness check that recovery re-armed past existing approvals. The fix was to read the effective review state, not new events.

Cheap code, costly judgment: making agentic SE governable

Wed, 01 Jul 2026 00:00:00 GMT

A 12-week case study and middle-range theory of “governance conversion”—turning agentic failure modes into durable controls.

The Audit Tax: generation is cheap; verification is the bill

Tue, 30 Jun 2026 00:00:00 GMT

Jun 30 Temrel: agent PRs arrive in ~90s, then burn ~90 minutes of trust-building — LinearB’s 4.6× review lag, graded verification (deterministic → review subagent → human), and evals that make the tax intentional.

Loop engineering: the checker is never the maker, and the stop is a command

Tue, 30 Jun 2026 00:00:00 GMT

Jun 30 sxivansx skill: scaffold LOOP.md + maker/checker/monitor agents + stop-check.sh — one on-disk cycle every trigger; human gate on merge; unattended means isolated.

Sonar Vortex and SonarQube Remediation Agent for agentic loops

Tue, 30 Jun 2026 00:00:00 GMT

June 2026: Vortex guides/verifies coding agents in-loop; Remediation Agent GA clears backlog with verified fix PRs.

Agent in a real repo: worktree isolation, full-suite done, PR not merge

Tue, 30 Jun 2026 00:00:00 GMT

Jun 30 2026 Ramón Chancay: each task gets a git worktree + branch; reproduce with a red test, require the whole suite and that the new test actually ran, ship a PR — never merge unsupervised.

Loop specifications: the check is the skill, not the prompt

Sun, 28 Jun 2026 00:00:00 GMT

Jun 28 Macedo: defines the external loop spec (trigger, goal, verify, stop, memory); codes 50 Loop Library entries—70% autonomous verify, but triggers and durable memory still mostly manual.

The bottleneck for code agents is review — so put gates before the human

Sun, 28 Jun 2026 00:00:00 GMT

Jun 28 2026 Samuel Fajreldines: a verification harness turns agent attempts into reviewable PRs — scope, required proof, risk inventory, stop contracts, and gates that block before human audit.

24 agents, 105 merged PRs, and a dispatcher that idles when the review queue is full

Fri, 26 Jun 2026 00:00:00 GMT

Jun 26 Yuvrajsinh (Trevion build log): an always-on daemon dispatches headless agents into worktrees around the clock, about four commits a day including nights. Two things made it work: a per-area regression gate that auto-reverts before a PR opens, and a MAX_OPEN_REVIEW ceiling added after 50 open PRs piled up and started to rot.

InterWorks: six Titans keep the factory honest overnight

Fri, 26 Jun 2026 00:00:00 GMT

Jun 26: Mnemosyne/Prometheus/Themis/… cron fleet on Curator — review, docs, triage, merge shepherd; deterministic-first Hyperion; shared lessons file; support tickets −50%.

Silent failures: the patch is green and still insecure

Thu, 25 Jun 2026 00:00:00 GMT

Jun 25 paper: across 1,030 agent repair traces, 170 confirmed silent failures (omit / introduce / inadequate defenses); test-pass gates and LLM reviewer roles did not catch them.

Reviewing agent PRs: declare the scope, list the behaviour changes, read the code before the tests

Tue, 23 Jun 2026 00:00:00 GMT

Jun 23 (cross-posted Oct 6) Ahmet Zeybek: after agent PRs passed half of one service's changes and a two-approval PR dropped a schema default in production, the team moved review into failing CI checks. Median time to first review fell from 5 hours to about 2, median PR size from 310 lines to 160, and reverts within a fortnight from 6 a quarter to 1.

Qodo 2.4: cross-repo review, Rules Miner, and Skill Review Standards

Tue, 23 Jun 2026 00:00:00 GMT

June 2026 release expanding governance: cross-repo blast radius, standards mined from PR history, and managed skill standards.

Aharness: process drift dies when the workflow is a state machine

Tue, 23 Jun 2026 00:00:00 GMT

Jun 23 Alfredvc Show HN: Codex workflows as TypeScript FSMs — typed evidence, approval gates, repair paths, inspectable run logs; Codeflow slices roadmaps into verified commits.

ShopBack: prompts plateaued; the harness made MRs land

Mon, 22 Jun 2026 00:00:00 GMT

Jun 22 Yan: production AI engineering harness — Shadow Explorer + multi-agent review + backtrack loops; first-pass review ~40%→70%; Agent = Model + Harness.

Repeat reviewers approve more agent PRs and comment less, while the PRs wait longer

Sun, 21 Jun 2026 00:00:00 GMT

Haoran Yu and co-authors follow 400 repeat reviewers through 11,429 reviews of agent PRs in the AIDev dataset. Approval rises from 27.9% to 42.4% between a reviewer's first and latest tenth of reviews, inline comments fall 22%, and median wait grows from 3.9 to 13.5 hours. PR size stays flat, and approval of human PRs in the same repos falls over the same months.

Foreman: workers are forbidden from writing verification.json

Sun, 21 Jun 2026 00:00:00 GMT

Jun 2026 VisionForge Foreman: gated plan→ADR/PRD→issues→TDD→e2e; PreToolUse deny + read-only evaluator agent; Foreman alone flips pass; retro patches need bench reports.

Overnight on a Mac: 41 commits, ~30% junk, morning review is the tax

Sun, 21 Jun 2026 00:00:00 GMT

Jun 21 Jirák: eight unattended hours — coverage 41%→86%, four real bugs in BUGS_FOUND.md; ~30% reverted; branch isolation + suite gates; review moved to a civilized hour.

Nightcore: autonomy at full speed — merge only after the gauntlet

Sun, 21 Jun 2026 00:00:00 GMT

Jun 21 Nightcore: local Claude Kanban studio — worktree builds, anti-gaming + Structure-Lock gates, independent reviewer PASS/CHANGES/FAIL, flight-recorder ledger; you approve merge.

Bastion: single-concern reviewers that aggregate into a merge gate

Sat, 20 Jun 2026 00:00:00 GMT

Jun 2026 attunehq/bastion (still shipping Sep): agent volume ×100 breaks human diff review; narrow one-property reviewers run locally then in CI; verdicts aggregate to one required check — humans curate reviewers, not every line.

Cloudflare's vuln harness started as a skill: hunters propose bugs, a separate agent tries to disprove them, and nothing trusts its own homework

Thu, 18 Jun 2026 00:00:00 GMT

Jun 18, Cloudflare open-sources the security-audit skill that seeded its fleet vulnerability harness: multi-phase recon and hunt, adversarial validators, schema-checked findings, and a fresh agent that re-verifies every survivor — then a second model system for triage and human-gated fixes.

Himmel: overnight ticket dispatch that still cannot merge without review

Tue, 16 Jun 2026 00:00:00 GMT

Jun 2026 yotamleo/Himmel: worktree-isolated Claude Code harness — /overnight-shift opens PRs while you sleep; hooks block main edits and unreviewed PRs; /pr-check multi-agent gate; handover state survives sessions.

Amnesiac overnight agents: progress lives on disk, not in chat

Tue, 16 Jun 2026 00:00:00 GMT

Jun 16 Meela Dheeraj: hourly fresh sessions 11pm–7am, BUILD_SPEC + PROGRESS.md + git as memory — commit = checkpoint; irreversible ops deferred to ‘needs human’.

All smoke, no alarm: 80% of agent test patches lack strong oracles

Tue, 16 Jun 2026 00:00:00 GMT

Jun 16 Banik et al.: 86,156 agent test-file patches across 33,596 PRs; 80.2% weak/no oracle; strong multi-signal oracles raise merge odds after controls (OR 1.28).

Agent Gate: deterministic merge evidence, not another LLM reviewer

Mon, 15 Jun 2026 00:00:00 GMT

Jun 15 Sung: CI firewall for AI PRs — declared scope, workflow permission escalation, agent control-plane drift, matching test-file evidence — policy loaded from base, never from the untrusted head.

Galea: overnight Claude reclaims the evaporating usage window

Mon, 15 Jun 2026 00:00:00 GMT

Jun 15: 5h Claude windows reset while you sleep — caffeinate + claude -p queue; FOMO as wasted capacity; productive complication: weekly ceiling still taxes daytime.

Factory 2.0: from coding agents to software factories

Mon, 15 Jun 2026 00:00:00 GMT

Factory.ai’s manifesto: model-independent, sovereign, self-improving SDLC loops where engineers build the factory that builds the software.

Agentic code review: understanding stayed expensive

Mon, 15 Jun 2026 00:00:00 GMT

Addy Osmani’s 2026 map: Faros/GitClear/CodeRabbit data, blast-radius tiering, heterogeneous AI reviewers, and humans on the loop—not reading every line.

LLM-as-Code: put the overnight loop in the program, not the model

Sun, 14 Jun 2026 00:00:00 GMT

Jun 14 Qi et al.: Agentic Programming — code owns looping/branching/stop; LLM only at judgment leaves; OSWorld 86.8% in 15 steps vs 80.4% prior in 100; DAG context O(depth) not O(steps).

NIGHT LOOP: CLAIM_DONE is rejected while the harness is still red

Sun, 14 Jun 2026 00:00:00 GMT

Jun 14 apsolut: Claude Code overnight wrapper — Night Contract, question firewall, PreToolUse outside permissions, Stop-hook ratchet vs model claims, SHA-256 event tape, tamper-evident Morning Receipt.

HarnessX: compose the harness, then evolve it under a seesaw gate

Fri, 12 Jun 2026 00:00:00 GMT

Jun 12 Darwin Agent Team: typed harness foundry + AEGIS Digester→Planner→Evolver→Critic — avg +14.5% (up to +44%) across five benches; variant isolation stops GAIA collapse; co-evolution +4.7% over harness-only.

A year running an agent harness, part 2: cry-wolf gates and reviews that only lint

Thu, 11 Jun 2026 00:00:00 GMT

Jun 11, Jason Vertrees (Heavy Chain). Five more failure modes from a year of a client-facing agent harness, this time where the signals mislead: an agent saves itself thirty seconds and spends the operator's afternoon, a config flag that reviews clean and does nothing, a gate with a predictable false positive that everyone learns to ignore, automated review that stops at style nits, and a prototype implemented as if it were the spec.

Same vulnerable diff, different story: Sevra-Bench

Thu, 11 Jun 2026 00:00:00 GMT

Jun 11 paper: reverse real vulnerability fixes, wrap them in 15 social-engineering PR framings, and watch review agents approve under prior-approval and dependency-compatibility narratives they cannot verify.

MakerChecker: deny-by-default roles so the agent can’t approve its own work

Thu, 11 Jun 2026 00:00:00 GMT

Jun 11 MakerChecker: gateway + scan + embedded governor — grant-pinned skills, SoD so maker ≠ checker, Ed25519 hash-chained audit anyone verifies offline.

Personal agent use swung from 44% to 4%; the harness held it at 92%

Wed, 10 Jun 2026 00:00:00 GMT

Jun 10 Štefan Moravík, FlowHunt: across 1,409 merged PRs in ten months at QualityUnit, agent work tracked individual enthusiasm until a risk-tier policy file and a staged, bounded pipeline arrived. Then it jumped and stayed, with a human still making every merge.

Camus: Claude makes; Codex reviews; Camus never lands it for you

Wed, 10 Jun 2026 00:00:00 GMT

Jun 10 Camus: deterministic local control plane — independent maker≠reviewer seats, repo tests as final word, bounded review loops, crash-safe resume; verifies overnight work without auto-merge.

loop-harness: you don't prompt Claude — scheduled loops do

Wed, 10 Jun 2026 00:00:00 GMT

Jun 10 loop-harness: cadence-driven Claude loops in worktrees — primary stages output, second agent must print VERDICT: PASS before PR/comments/Slack ship.

Goal-Autopilot: unattended ‘done’ must pass an executed gate

Wed, 10 Jun 2026 00:00:00 GMT

Jun 10 Deng: durable gated FSM + No-False-Success floor — 3,150-cell paired eval cuts fabrication to 0.95% vs StateFlow 25%; SWE-bench Lite −33 pp; honest stall over confident wrong patches.

NightWatch: logs are claims — replays are proofs

Wed, 10 Jun 2026 00:00:00 GMT

Jun 10 BeamusWayne NightWatch: hash-chained black-box ledger for overnight agents; morning debrief re-runs recorded commands and diffs git vs claims; attest CI gate.

Self-Harness: the same fixed model improves its own harness under a held-out gate

Mon, 08 Jun 2026 00:00:00 GMT

Jun 8 Zhang et al.: Weakness Mining → bounded proposal → promote only if held-in and held-out both non-regress; up to +132% relative Pass across Terminal-Bench-2 / SWE-bench Verified / AppWorld.

A year running an agent harness for clients: the worst failures all reported success

Mon, 08 Jun 2026 00:00:00 GMT

Jun 8, Jason Vertrees (Heavy Chain). From a year of a harness that writes, reviews and ships client software, five failure modes where the system said done and was wrong: a 738-test green board that wrote no rows, lessons that never became gates, one suppressed warning copied by every later agent, a fabricated API client built on by seven agents, and five gates that each assumed another had checked.

failclosed: MERGE_READY only when the reviewer’s output survives a distrust gate

Mon, 08 Jun 2026 00:00:00 GMT

Jun 8 Dan Mercede / OrionArchitekton: fail-closed merge admission — parse LLM review, block on unparseable/schema-invalid/contradictory output; receipts over logs; reviewer is a command seam, not the product.

FrontierCode: grade like a tech lead — would the maintainer merge?

Mon, 08 Jun 2026 00:00:00 GMT

Jun 8 Cognition: first mergeability benchmark — 20+ OSS maintainers, 40h/task, blockers + quality rubrics; Opus 4.8 ~13% Diamond / 34% Main; 81% fewer misclassifications vs SWE-Bench Pro.

Nocturne: label an issue, wake to a verified PR — never a merge

Mon, 08 Jun 2026 00:00:00 GMT

Jun 2026 ba1lly/nocturne: daemon picks agent-labelled issues across repos, isolated worktree + verify_cmd + review/fix budget, opens a PR; optional CI/review shepherding — never merges, force-pushes, or touches main.

Loop engineering: design the system that prompts the agent

Mon, 08 Jun 2026 00:00:00 GMT

Jun 8 Addy Osmani: automations, worktrees, skills, connectors, maker/checker sub-agents, plus disk memory — /goal stops on a separate model’s done-check, not the implementer’s optimism.

Dark Factory: reviewers who literally cannot edit the branch

Sat, 06 Jun 2026 00:00:00 GMT

Stratton / godark: humans author architecture + scenario specs; implementer in Docker; quality + functional reviewers isolated (no file write); retry then squash-merge or needs-human-review.

Vanguard: always-on factory that stops at draft PRs

Fri, 05 Jun 2026 00:00:00 GMT

Jun 5 Vanguard (still pushing Sep 23): watch loop claims tickets from GitHub/GitLab/Linear, plan→implement→review→adversary→repair in Docker, Proof-of-Work verify, then draft PR only — humans own merge.

Augment CIV: separate the judge from the builder — or keep the single-agent burst

Wed, 03 Jun 2026 00:00:00 GMT

Jun 3 Augment guide: Coordinator–Implementor–Verifier splits planning, execution, and spec-grounded validation so generation errors stop grading themselves — and names when single-file bursts still win.

agent-completion-gate: /goal can loop; only artifacts grant complete

Mon, 01 Jun 2026 00:00:00 GMT

Jun 1 zhjai agent-completion-gate: worker proposes candidate_complete; external check on real files + human-confirmed acceptance manifest grants complete — fills Claude/Codex /goal’s transcript-only stop gap.

Coding isn’t the constraint: 76% more PRs to review

Mon, 01 Jun 2026 00:00:00 GMT

Spotify’s Code with Claude talk write-up: 99%+ weekly AI usage, Honk in K8s with CI verification, Backstage golden paths as agent guardrails.

Recall is the metric now: AI review inverted precision culture

Mon, 01 Jun 2026 00:00:00 GMT

Ben Nevile on why agent-side filtering flips review etiquette from “only raise sure things” to “raise everything potentially relevant.”

Make checking the work a job with a number, not a rushed PR step

Mon, 01 Jun 2026 00:00:00 GMT

June 2026 org field note: AI-coauthored PRs carry ~1.7× more review findings — rebuild around four-person pods (lead, two directors, quality engineer) so judgment and verification scale with agent output.

Autonomous agents lift commits 240% but releases only 30%, because human steps are the weak link

Mon, 01 Jun 2026 00:00:00 GMT

Demirer, Musolff and Yang match AI usage telemetry to more than 500,000 GitHub developers. Each generation of tool adds more coding activity, but the gains shrink at every step from commits to projects to releases, consistent with human review and judgment being hard to substitute.

Label-cascade software factory: plan → CI → review → squash without a happy-path human

Sat, 30 May 2026 00:00:00 GMT

Yefremov’s fork-and-own software-factory: four Claude roles on a GitHub label state machine, bounded retries, CI gate, auto-merger with guardrails — continuous verified throughput as a template.

Overnight coding agents: attention-decoupled runs with a ready-or-not verdict

Sat, 30 May 2026 00:00:00 GMT

May 30 2026 Ralph Workflow: overnight pattern = time-decoupled, structured plan→implement→verify→fix, verifiable receipts, reviewable git diffs — not a chat with a sleep timer.

In an unattended review panel, a reviewer that couldn't see the code counts as a pass

Fri, 29 May 2026 00:00:00 GMT

Huiyan Wan's overnight-workflows plugins for Claude Code run multi-agent review panels on stacks of PRs while nobody is watching. One failure gets its own plugin: review subagents without a shell can't fetch the PR, return a BLOCKED report or review main instead, and by morning that empty report reads as clean. Re-dispatched with the diff on disk, one such reviewer found a data-loss bug that 179 green tests had missed.

Otis: eight checks between the agent and the PR — including mutation-light

Fri, 29 May 2026 00:00:00 GMT

May 29 42nights Otis: issue→PR overnight agent whose centerpiece is an 8-check harness (typecheck, suite, plan-tests, mutation-light, lint, diff cap, banned patterns, critic); never auto-merges.

Meta’s RADAR: automate low-risk review with calibrated risk

Thu, 28 May 2026 00:00:00 GMT

Industrial study of RADAR at Meta—risk calibration and selective automation to reclaim reviewer time on low-risk changes.

compass: loops under a gate — agents prepare; you always merge

Mon, 25 May 2026 00:00:00 GMT

May 25 dshakes compass: budget-capped Claude/Codex/Gemini harness, eval-gated guardrails, self-fixing PR loop with cross-model audit; fleet overnight; human merge permanent.

Unattended agents need containment, not more approval dialogs

Mon, 25 May 2026 00:00:00 GMT

May 25 Anthropic: 93% approve rate → approval fatigue; environment boundaries (sandbox/VM/egress) cap blast radius so always-on loops can run without a human watching every tool call.

Bello: independent supervisor, completion review, and adversary around the coder

Sun, 24 May 2026 00:00:00 GMT

Makson179 Bello (May 2026; still shipping Sep 24): Codex/Claude/API coder in a disposable workspace; fresh-context supervisor for risky actions; completion review + adversary refuse 'done' on claims; ProgramBench C+A +26% mean vs raw; Efficient Budget −66% Codex weekly usage at same quality.

Ona's background-agents guide: laptop agents are a false summit

Fri, 22 May 2026 00:00:00 GMT

Ona's engineering-leader guide, built from its May 2026 Background Agents Summit with Stripe, Ramp, Spotify, Uber and Harvey, argues that faster individual agent sessions don't move org throughput; triggered, sandboxed agents that can verify their own work do.

Harness engineering: five layers, and review velocity as a Stage-1 metric

Fri, 22 May 2026 00:00:00 GMT

May 22 Neely Dunlap (Faros): Agent = Model + Harness — tool orchestration, verification loops, memory, guardrails, observability; baseline cost-per-merged-PR and review velocity before buying more peak-session model.

Agentic PR merge/reject labels overstate what agents did

Thu, 21 May 2026 00:00:00 GMT

May 21 Peralta et al. (MSR ’26): of 717 inspected AIDev Agentic-PRs, only 35.7% of rejections were clear agent failures; 31.2% workflow, 33.1% no rationale; 15.4% of merges needed reviewer feedback or commits—outcomes ≠ capability.

Overnight agents that open draft PRs: the harness is the product

Wed, 20 May 2026 00:00:00 GMT

May 20 Pickuma: ingest→plan→execute→verify→package; verify-loop with retry bound; draft PR + agent-generated label; morning review non-negotiable; thin tests = no safety.

AIFactory: verify against the live endpoint, not the diff

Wed, 20 May 2026 00:00:00 GMT

May 20 AIFactory PARR loop: plan→act→deploy→reflect→review — worktree isolation, live HTTPS verify, bounded auto-feedback, human-stop defaults; auto-merge only after reviewer gate.

The agentic metaharness: a room above your coding harnesses

Mon, 18 May 2026 00:00:00 GMT

May 18 SoloTerm/Aaron Francis: models → harnesses → metaharness — shared process graph, scratchpads, todos, timers, and cross-harness spawn so agents coordinate without you babysitting every tab.

A vision paper for agentic code review

Sun, 17 May 2026 00:00:00 GMT

Position/vision piece on how code review should change when AI agents author and critique changes across the SDLC.

code-coordinator: Work → Test → Review → Merge as one overnight drive

Sun, 17 May 2026 00:00:00 GMT

May 17 JDonaghy code-coordinator: Claude CLI fleet across worktrees/machines; Test gates Review; independent review; merge queue + overnight drive-queue — human approves dispatch, harness drains to merge.

OpenSearch: harness-first SDLC — trust the loop, not the review queue

Fri, 15 May 2026 00:00:00 GMT

May 15 field note: Atlas/Ralph/Nitro/Sentinel each run inside a verification harness — live local stack, A/B benchmarks, plan-then-approve — because agents generate faster than humans can review.

HarnessAudit: task-green overnight runs can still violate the harness

Thu, 14 May 2026 00:00:00 GMT

May 14 Liu et al.: full-trajectory harness safety (boundary/fidelity/stability) on 210 tasks — best overall ~0.32; completion misaligned with SAR; multi-agent amplifies resource + info-flow violations.

Capability is model × harness × environment — not the model alone

Wed, 13 May 2026 00:00:00 GMT

May 13 Zhong & Zhu: eleven harness responsibilities, H0–H3 visibility ladder, and episode packages that adjudicate verification autonomy — reframing autonomous SE as a runtime-systems problem.

DeadReckon: two independent keys before an unattended job is 'done'

Wed, 13 May 2026 00:00:00 GMT

May 13 DeadReckon: harness around Claude/Codex/Gemini/Copilot/OpenCode — detachable Job, sandbox checks + fresh semantic judge, HMAC receipt; NEEDS_REVIEW when either key fails.

AgentLens: green tests can still be a Lucky Pass

Wed, 13 May 2026 00:00:00 GMT

Process-level SWE-agent eval showing ~10.7% of passing trajectories are Lucky Passes—missing verification, blind retries, disordered stages.

Reviewing what the overnight agent shipped while you slept

Tue, 12 May 2026 00:00:00 GMT

May 12 Toni Antunovic: overnight diffs lack mid-flight correction — aggregate review, security triage, reasoning traces, semantic not line-by-line; constrain batch size or automate triage.

From fear of missing agent time to fear of unused capacity: the bottleneck moves to planning

Mon, 11 May 2026 00:00:00 GMT

May 11 (edited Jul 28) Teemu Piirainen: once agents ran ten-hour sessions without asking questions, the anxiety didn't go away. It moved from answering agents fast enough to preparing enough work for the night, and mistakes started surfacing in morning review as the right execution of the wrong spec.

The mixed PR queue: reviewer throughput is the binding constraint

Sun, 10 May 2026 00:00:00 GMT

May 10 field note: Tian Pan on agent fleets as backlog generators unless you tier the queue — author class, supervision, risk — with traces to audit and CI never enough for business logic.

wgmesh: an autonomous merge bot counted any Copilot review as approval, so it merged exactly the PRs Copilot had the most to say about

Sat, 09 May 2026 00:00:00 GMT

May 9, the wgmesh project found its review-and-merge bot passed any review submission, including Copilot's COMMENTED findings. Three PRs merged with 4 to 13 open findings; the fix requires an explicit APPROVED from each reviewer's latest review, re-checked just before merge.

OpenHands verification stack: critic before push, review+QA on the PR, iterate until green

Wed, 06 May 2026 00:00:00 GMT

May 6 OpenHands: agent-level critic gates broken trajectories; repo-level review skill + behavioral QA; /iterate closes the loop — 1,000+ reviews, 58% faster MTTM, ~25% coverage lift on their SDK repo.

Hellström: an overnight fleet merged 11 PRs, sat idle 94.5% of the night, and had no quality gates

Wed, 06 May 2026 00:00:00 GMT

May 6 from Felix Hellström: the first unsupervised night of his agent orchestrator ran 686 cycles in 7.5 hours, dispatched 103 tasks against 54 issues, auto-merged 11 PRs, flagged 9 for review, and killed and re-dispatched 8 stalled agents. Every failure lived in the harness, not the agents — and auto-merge on a green suite was luck, not a guardrail.

pr-autopilot: opt-in stages from open PR to green merge — never silent by default

Wed, 06 May 2026 00:00:00 GMT

May 6 FelipeOFF Claude Code skill: Reviewer + Author subagents triage every human/bot comment, attribute CI before patching, stop on business rules; --auto is explicit, bare run only opens the PR.

Greptile's overnight-agents study: agent PRs hold up, but the bugs move

Tue, 05 May 2026 00:00:00 GMT

Greptile mined millions of reviewed PRs and found end-to-end agent PRs rose from 0.86% to 27.6% of merges in fourteen months, with revert, churn and review-cycle numbers roughly matching humans. What changed was which bugs show up.

Long-running agents need harnesses, not hope — eight parts before you sleep

Sat, 02 May 2026 00:00:00 GMT

May 2 Developers Digest: task contract, scoped workspace, tool policy, logs, checkpoints, verification gates, cost/time budgets, final receipt — without them an overnight run is a chat session with more failure modes.

NachtSchicht: the morning brief is the product — merge without re-reading

Thu, 30 Apr 2026 00:00:00 GMT

Apr 30 Florian Riquelme: Claude Code night-shift queue with a trust protocol — worktree isolation, blast-radius verify, halt-on-ambiguity, immutable evidence; the brief is a contract, not a transcript dump.

12 tools in one night: overnight orchestration’s honest cost

Wed, 29 Apr 2026 00:00:00 GMT

Apr 29 Developers Digest: /loop every 10m produced 12 repos and ~60 PRs — and a billing wall, a fake PR number, and a full morning of curation. Parallel overnight works; unverified claims don’t.

AHE: observability turns harness edits into falsifiable overnight contracts

Tue, 28 Apr 2026 00:00:00 GMT

Apr 28 Lin et al.: ten AHE iterations lift Terminal-Bench 2 pass@1 69.7%→77.0% past Codex 71.9%; frozen harness transfers +5.1–10.1 pp cross-family; tools/middleware/memory carry the gain, not the prompt.

Long-running agents: the gap is state, sessions, and handoffs—not the chat loop

Tue, 28 Apr 2026 00:00:00 GMT

Apr 28 Addy Osmani: long-horizon vs long-running vs persistent agency; Ralph, Anthropic brain/hands/session, Cursor planner–worker–judge; write done-conditions and separate evaluator from generator before overnight FOMO.

Pre-commit review gate: the agent cannot commit until CLEAN

Wed, 22 Apr 2026 00:00:00 GMT

Apr 22 Johnston (IMTI): PreToolUse git-commit hook + adversarial sub-agent — Verdict CLEAN pinned to diff hash; blocks review theater and post-commit fix loops.

superhuman: closed-loop merge-probability until a maintainer would merge

Wed, 22 Apr 2026 00:00:00 GMT

Apr 22 gaurav0107/superhuman: multi-agent Claude/Codex harness scores its own PR on a 10-dimension rubric, iterates the weakest dimension to ≥95% over two runs, learns from merge_outcomes.jsonl — with merged PRs into transformers, airflow, oxc.

Broccoli: Linear tickets become PRs overnight — humans still merge

Wed, 22 Apr 2026 00:00:00 GMT

Apr 22 Besimple Broccoli OSS: assign Linear to a bot; Codex/Claude in your Cloud Run or Blaxel sandbox plan, implement, open a PR; review-feedback loops; human owns merge.

OvernightAgent: four gates between every step and 'done'

Mon, 20 Apr 2026 00:00:00 GMT

Apr 20 oa: worktree-per-task overnight queue for Claude/Codex/OpenCode — tail protocol, commit-since-start, verify cmd, AI review; fix-loop; events.jsonl + SUMMARY.md; crash-resumable daemon.

Cloudflare’s AI review isn’t a chatbot — it’s a CI orchestrator

Mon, 20 Apr 2026 00:00:00 GMT

Apr 20 field report: up to seven specialist OpenCode reviewers under a coordinator; 48k MRs / 30 days, median 3m39s, $1.19 avg, break-glass 0.6%.

Night Shift: overnight Claude with Codex that must agree you're done

Sun, 19 Apr 2026 00:00:00 GMT

ppuliu skill: up to 8h on a feature branch; file-gated Codex review per commit; dual consensus to end; never push/PR — self-review only when Codex absent, explicitly flagged.

Kaizen: continuous prompt optimization that opens reviewable PRs

Fri, 10 Apr 2026 00:00:00 GMT

April 2026 GitHub project: DSPy-based prompt optimization that can land changes as PRs for human review.

Factory Missions: multi-day multi-agent execution with scrutiny validators

Fri, 10 Apr 2026 00:00:00 GMT

April 2026 architecture note on Missions—workers, validators, and generate/validate loops for long-horizon work.

Overnight FOMO is a trust problem — nap runs before full nights

Thu, 09 Apr 2026 00:00:00 GMT

Apr 9 Ship with AI: 56h/week idle agent capacity; night-shift skill wraps Ralph with pre-flight, kill conditions, and morning categorization — calibrate small before you sleep on it.

241 PRs in a weekend, and the agent kept trying to stand by

Thu, 09 Apr 2026 00:00:00 GMT

Apr 9 dkdc.net: one orchestrator, a handful of workspace and review agents, CI-gated auto-merge across 12 repos, and an iMessage control channel produced 241 merged PRs by Monday. The owner's verdict: real bugs fixed, lots of suboptimal PRs, and every failure turned into a written convention.

Tenet: 12+ hour runs with independent critics, not self-certified done

Wed, 08 Apr 2026 00:00:00 GMT

Apr 8 Tenet: interview→spec→DAG→execute→evaluate loop — model-tier-shaped plans, grounded+ungrounded critics (code/test/e2e), doctrine that self-corrects between overnight cycles.

thin-supervisor: 'done' is the acceptance contract, not the agent's claim

Wed, 08 Apr 2026 00:00:00 GMT

Apr 8 thin-supervisor: sidecar over Claude/Codex CLI — continue/verify/retry/recover/escalate; verifier + AcceptanceContract gate finish; Stop-hook closes the loop when you leave.

Managed Agents: durable session, cattle harness, disposable hands

Wed, 08 Apr 2026 00:00:00 GMT

Apr 8 Anthropic (Martin/Cemaj/Cohen): session + harness + sandbox as swappable interfaces; pets→cattle; credentials never in the sandbox — meta-harness so overnight loops outlast any one harness recipe.

Nightshift: overnight work only counts behind seven verification stages

Mon, 06 Apr 2026 00:00:00 GMT

Apr 6 Orbit/Sharma: open-source overnight orchestrator — hardening + feature loops, worktree isolation, shift logs; runner enforces policy (blocked paths, verify command, anti-tunnel-vision), agent supplies intelligence.

Last Light: a self-hosted factory that works after dark on your GitHub

Sat, 04 Apr 2026 00:00:00 GMT

NearForm Last Light: webhook/cron triage + PR review, Architect→Executor→Reviewer builds, permission-scoped App tokens, and evals scored on your own repos — not someone else’s bench.

claude-overnight: the swarm improves its own skill library mid-run

Sat, 04 Apr 2026 00:00:00 GMT

Apr 4 Fornace: multi-wave overnight Claude SDK swarm in worktrees — self-review + simplify, cross-agent wave review, final architecture gate, and a librarian that compounds skills wave to wave.

AgenticFlict: 27.67% of 107K unmerged agent PRs no longer merge cleanly, from 15% for Copilot to 32% for Codex

Sat, 04 Apr 2026 00:00:00 GMT

Ogenrwot and Businge replayed git merges for 107,026 open or closed-unmerged agent PRs from the AIDev dataset and found 29,609 with textual conflicts and 336,380 conflict regions, with conflict rates varying sharply by agent.

Leaving agents idle overnight is shutting down CI every evening

Fri, 03 Apr 2026 00:00:00 GMT

Apr 3 Escape/Carossio SF field report: harness + AI-factory layers; FOMO as asset utilization — overnight produces candidates not commits; verification beats advice.

AutoAgent: overnight hill-climb that rewrites the harness, not just the prompt

Thu, 02 Apr 2026 00:00:00 GMT

Apr 2 Thirdlayer: meta-agent edits agent.py (tools, verification, context, orchestration) against Harbor evals; keep score gains, revert regressions; >24h runs hit 96.5% SpreadsheetBench and 55.1% TerminalBench (GPT-5) from a minimal baseline.

Guides before, sensors after: Böckeler’s map for trusting agent code

Thu, 02 Apr 2026 00:00:00 GMT

Apr 2 Thoughtworks/Fowler: coding-agent harness as feedforward guides + feedback sensors, computational vs inferential, keep quality left — and why behaviour harnesses still demand human supervision.

Auto-harness: overnight failure mining with a living regression gate

Tue, 31 Mar 2026 00:00:00 GMT

Mar 31 NeoSigma: coding agent iterates the harness overnight — mine failures, cluster, propose changes, keep only what clears a self-maintained eval suite + held-out val; 0.56→0.78 on Tau3 with fixed GPT-5.4.

ABCA: fire-and-forget background agents that open PRs on AWS

Tue, 31 Mar 2026 00:00:00 GMT

Mar 31 aws-samples ABCA: submit via Slack/Linear/Jira/CLI/webhook → MicroVM agent clones, tests, opens PR; blueprint mixes deterministic + agentic steps; memory from PR feedback; budgets and blast-radius limits for 24/7 runs.

Meta-Harness: overnight harness search needs the full trace filesystem, not a score summary

Mon, 30 Mar 2026 00:00:00 GMT

Mar 30 Lee et al.: agentic outer loop over harness code with filesystem access to prior code/scores/traces — +7.7 pp vs ACE at 4× less context; TerminalBench-2 #1 Haiku / #2 Opus among reported agents.

Nightshift: the builder proposes; the harness owns the commit

Sat, 28 Mar 2026 00:00:00 GMT

Mar 28 Nightshift/AgentForge: Ralph loop with separate evaluator, mutation-guarded feature queue, and metrics dashboard as receipt — 29/30 features, 41 attempts, ~$0.95 in 76 minutes.

SWE-PRBench: grade AI review against real PR feedback

Fri, 27 Mar 2026 00:00:00 GMT

Benchmark that scores AI code-review quality against pull-request feedback rather than only against generation/pass metrics.

Idle-agent anxiety has a formula: work time over instruction time

Fri, 27 Mar 2026 00:00:00 GMT

Mar 27 essay by Eduardo Aguilar Pelaez names the low-grade dread of an agent waiting on you, models it as a queue with the human as the server, and argues the lever is longer autonomous work intervals plus an orchestrator, not more agents.

TheBotCompany: 142 hours of unattended multi-agent development, where a separate verifier rejected up to 18.6% of milestones

Thu, 26 Mar 2026 00:00:00 GMT

An ASE 2026 paper ran a self-organizing agent team on four real projects for 142 wall-clock hours and 164 milestones with no human intervention after setup, for about $1,500. A dedicated verification phase rejected 18.6% of milestones on one project, unit-test gates still missed regressions the planner later caught, and reverts restored the baseline every time.

Kitchen Loop: 1,094 merged PRs under unbeatable tests and pause gates

Thu, 26 Mar 2026 00:00:00 GMT

Mar 26 Roy: six-phase coverage-exhaustion loop over two production systems; 285+ iterations, ~$0.38/PR, zero oracle-detected regressions; drift control + drain/starvation gates keep the night honest.

BCG Platinion: harness engineering and intent thinking

Thu, 26 Mar 2026 00:00:00 GMT

Enterprise framing of the Agentic Software Factory—humans relocate to intent and outcomes; five transformation pillars including governance and scenario tests.

Generator vs evaluator: separate the judge from the builder

Tue, 24 Mar 2026 00:00:00 GMT

Anthropic Labs’ GAN-inspired planner/generator/evaluator harness—sprint contracts, Playwright QA, and shedding scaffolding as models improve.

Ten months of CCA in dotnet/runtime: harness beats heroics

Mon, 23 Mar 2026 00:00:00 GMT

Toub’s receipts: 878 CCA PRs, merge 67.9% vs 87.1% human; instructions 38%→69%; phone-flight FOMO creates review debt; fully autonomous 55.1% vs 86.2% with human commits.

Anthropic: every night without agents is progress left on the table

Mon, 23 Mar 2026 00:00:00 GMT

Mar 23: Discovery-team field guide — CLAUDE.md plan, CHANGELOG memory, CLASS test oracle, tmux detach, Ralph loop; FOMO framed as opportunity cost of idle agents.

Keen: token anxiety is the pull to keep an agent running, and it isn't about the credits

Fri, 20 Mar 2026 00:00:00 GMT

Sam Keen names the low-grade pressure to always have an agent working: unused usage feels like waste, an idle GPU feels like lost output. He argues it's an identity problem, not a budget one, and that the answer is deliberate goals rather than round-the-clock runs.

SFLO: lights-out only advances when the gate artifact exists

Thu, 19 Mar 2026 00:00:00 GMT

Mar 19 Simon Factory Lights Out: six sequential gates (Discover→Build→QA→Security→Verify→Ship), each requiring a named markdown artifact — no artifact, no progress; inner Build/QA loop capped at 10 rounds.

Optio: self-hosted ticket-to-merged-PR loop with CI-resume feedback

Thu, 19 Mar 2026 00:00:00 GMT

Mar 19 jonwiggins/optio: K8s (or laptop) control plane assigns GitHub/Linear/Jira tickets to Claude/Codex/Copilot worktrees, watches CI every 30s, auto-resumes on failures and review comments, then squash-merges — always-on swarm orchestration you own.

Fabro: a dark factory as version-controlled workflow graphs

Tue, 17 Mar 2026 00:00:00 GMT

Mar 2026 Helmkamp/Fabro: Graphviz workflows with human gates, multi-model routing, cloud sandboxes, git checkpoints, and verification fix-loops — walk away while the line keeps running.

Isolated milestones look solved; continuous evolution does not

Fri, 13 Mar 2026 00:00:00 GMT

Mar 13 Deng/Chen et al.: on SWE-Milestone’s 98 dependency-linked milestones, frontier scores fall from >80% independently to ≤38.03% continuously—recall still grows, precision saturates, regressions snowball.

Overnight agents need a verification layer, not a longer session

Thu, 12 Mar 2026 00:00:00 GMT

Mar 12 Vincent Tran: builder/judge separation, one browser agent per acceptance criterion, artifact-backed promotion — overnight coding without verification is morning cleanup.

CR-Bench: do review agents help in the real workflow?

Tue, 10 Mar 2026 00:00:00 GMT

Benchmark aimed at real-world utility of AI code review agents—not just comment similarity scores.

Datadog: harness-first, then telemetry closes the loop

Mon, 09 Mar 2026 00:00:00 GMT

Mar 9: redis-rust and Helix — DST/TLA+/Kani pyramid replaces line-by-line review as primary correctness; production telemetry refines the harness when invariants were wrong.

Anthropic brings multi-agent Code Review to Claude Code

Mon, 09 Mar 2026 00:00:00 GMT

March 2026 research preview: parallel find/verify/rank agents on GitHub PRs for Team/Enterprise, ~20 minutes and $15–25 per review.

Copilot code review moves to an agentic, Actions-backed architecture

Thu, 05 Mar 2026 00:00:00 GMT

March 2026 changelog: tool-calling reviews with broader repo context, running on GitHub Actions (including self-hosted caveats).

60 million Copilot code reviews: GitHub’s postmortem on signal vs thoroughness

Thu, 05 Mar 2026 00:00:00 GMT

March 2026 product essay on CCR usage scale, agentic redesign, and redefining “good review” as high-signal progress rather than exhaustive nitpicking.

Pairing with an LLM is delegation — shared judgment still needs humans

Wed, 04 Mar 2026 00:00:00 GMT

Mar 4 Wynne: agentic tools make soloing easier, but ‘pairing with Claude’ is a siren; as delivery speeds up, humans still co-create architecture, debt, and the constraints that check the robots.

AgentsMesh: the repo is the harness — and three parallel streams is a hard ceiling

Wed, 04 Mar 2026 00:00:00 GMT

Mar 4 AgentsMesh field note: 52 days / 600 commits building the fleet console with itself; Loops (cron) vs Tickets; tech-debt amplification; Autopilot after cognitive bandwidth broke at ~3 workstreams.

Martian Code Review Bench: F1 from real developer behavior

Tue, 03 Mar 2026 00:00:00 GMT

CodeRabbit’s write-up of Martian’s independent bench (~300k PRs): recall and F1 leadership claimed from whether developers act on comments.

Six idle agent sessions feel like lost weeks, and the reviewer approves at 4 PM what they would reject at 9 AM

Mon, 02 Mar 2026 00:00:00 GMT

Charles Sieg on the psychology of supervising coding agents by hand: parallel sessions finish in twenty minutes and then wait on him, an idle terminal feels like a stopped factory floor, and decision fatigue and craving arrive together. The cost lands in review quality.

Kelos: the always-on factory is a Kubernetes CRD, not a laptop left open

Mon, 02 Mar 2026 00:00:00 GMT

Mar 2 Kelos: coding agents as K8s Tasks/TaskSpawners — GitHub/Jira/cron triggers, isolated pods, GitOps-able workflow YAML; self-development pipeline ships its own PRs.

Overnight agents need episodes, not one immortal session

Mon, 23 Feb 2026 00:00:00 GMT

Feb 23 Nightcrawler: bounded Claude Code episodes, HANDOFF.md cross-checked against git, immutable tasks.json, eight stop conditions including diminishing returns.

Agent PRs: reviewer engagement and stable loops beat more commits

Mon, 23 Feb 2026 00:00:00 GMT

Feb 23 MSR ’26 study of 33,596 agent-authored PRs (AIDev): 71.5% merged overall, but merge odds track reviewer engagement and coordination stability more than iteration volume. Force pushes and large changes hurt; actionable review loops explain most qualitative successes.

Nightwire: independent verify, then baseline-relative gates

Mon, 23 Feb 2026 00:00:00 GMT

Feb 2026 HackingDave/Nightwire: Signal-orchestrated workers; fresh-context verifier on the diff; test snapshots that fail only on NEW regressions; ≤2 auto-fix attempts — same brain never grades its own patch.

Stripe Minions: unattended blueprints beat shoulder-surfing Claude

Thu, 19 Feb 2026 00:00:00 GMT

Feb 19 Stripe Part 2: 1,300+ minion-only PRs/week — blueprints interleave agent nodes with deterministic lint/push/CI; hot-ready isolated devboxes; at most two CI rounds before human review.

kodo: overnight Claude Max with an architect that rejects the worker nine times

Wed, 18 Feb 2026 00:00:00 GMT

Feb 18 kodo: Gemini orchestrator drives Claude/Cursor/Codex workers overnight; independent architect+tester gates accept only after multi-round rejection — blackopt run: 3h, 9 verifier catches, 73 tests.

Shipwright: Code Factory merges only with current-head evidence and a harness-gap loop

Tue, 17 Feb 2026 00:00:00 GMT

Shipwright Code Factory pattern: agents write 100% of code; risk-policy gate, SHA-tied evidence (API/CLI/DB/browser), automated remediation, and incident→harness-gap SLAs before merge.

Harness-only: Terminal Bench 52.8 → 66.5 without touching the model

Tue, 17 Feb 2026 00:00:00 GMT

Feb 17 Trivedy (LangChain): deepagents-cli hill-climbs via traces, PreCompletionChecklist self-verify, loop detection, and reasoning sandwich — same gpt-5.2-codex throughout.

OpenAI harness engineering: humans steer, agents execute

Wed, 11 Feb 2026 00:00:00 GMT

Ryan Lopopolo’s report: ~1M lines, ~1,500 PRs, zero manually written code—environment design, legibility, and agent-to-agent review as the job.

Ralph Loop: overnight agents need stop hooks and machine-checkable done

Sun, 08 Feb 2026 00:00:00 GMT

Feb 8 Blake Crosley: fresh-context iterations + filesystem memory + stop-hook criteria shipped 3,455 LOC / 141 tests overnight — spawn budgets, non-trivial criteria, and cleanup beat one long Claude session.

StrongDM via Willison: holdout scenarios so agents cannot review their own homework

Sat, 07 Feb 2026 00:00:00 GMT

Feb 7 Simon Willison on StrongDM’s dark factory: no human-written or human-reviewed code; scenarios as holdout sets; Digital Twin Universe clones of Okta/Jira/Slack for high-volume satisfaction scoring.

Cursor: one week, ~1k commits/hour — harness over heroics

Thu, 05 Feb 2026 00:00:00 GMT

Feb 5 research: Wilson Lin’s multi-agent harness ran continuously a week (~1k commits/hr, 10M tool calls); planner/subplanner/worker handoffs; 100% per-commit correctness killed throughput.

Qodo 2.0: multi-agent review and a self-published precision–recall benchmark

Wed, 04 Feb 2026 00:00:00 GMT

February 2026 launch of Qodo 2.0 with specialized review agents and Code Review Benchmark 1.0 claims on real PRs.

AACR-Bench: repo-level automatic code review with expert-verified defects

Tue, 27 Jan 2026 00:00:00 GMT

A multi-language ACR benchmark that expands defect coverage ~285% via AI-assisted, expert-verified labels and stresses context retrieval choices.

Continuous Agent: a 24/7 queue that spawns workers and validates before the next goal

Sat, 24 Jan 2026 00:00:00 GMT

Jan 24 jackzhaojin/continuous-agent: PM2 executive loop picks prioritized goals, spawns multi-vendor workers into an isolated sandbox, runs deterministic verifiers, and only sleeps when the queue is empty.

Where agentic PRs fail: abandonment, duplicates, and CI — not just bad code

Wed, 21 Jan 2026 00:00:00 GMT

Jan 21 study of 33k agent-authored GitHub PRs: docs/CI/build merge most; performance and fix least. Among 562 labeled rejections, 38% closed with no meaningful human review, 23% were duplicates, 17% failed CI — agent misalignment was only ~2%.

How Cursor built Bugbot: resolution rate, BugBench, and autofix

Thu, 15 Jan 2026 00:00:00 GMT

January 2026 engineering essay on Bugbot versions, resolution-rate metrics, internal BugBench, and Bugbot Autofix beta.

Evaluating AI reviewers: pilots beat leaderboard cosplay

Fri, 09 Jan 2026 00:00:00 GMT

CodeRabbit’s David Loker on why multi-bot-on-one-PR comparisons mislead, and which behavioral metrics (acceptance, noise, coverage) actually matter.

Anthropic: demystifying evals for coding agents

Fri, 09 Jan 2026 00:00:00 GMT

Engineering post on graders, pass@k vs pass^k, transcript review, and why SWE-bench-style outcome tests are necessary but incomplete.

Foundry: a persistent control plane that validates before it persists

Sun, 28 Dec 2025 00:00:00 GMT

Dec 28 ai-supervisor-foundry/foundry: BYOK orchestration for Cursor/Copilot/Gemini — provision sandboxes, dispatch worker nodes, hybrid local-then-LLM validation, DragonflyDB resume across multi-day runs.

Zeroshot: the writer never grades the work — the graph does

Thu, 25 Dec 2025 00:00:00 GMT

Dec 25 Open Engine Zeroshot: authored executor–verifier graphs, independent reviewers with no shared session, SQLite ledger, bounded repair; delivery only after checks pass.

Spotify Honk part 3: verification loops before the PR exists

Tue, 09 Dec 2025 00:00:00 GMT

December 2025 post on deterministic verifiers + LLM-as-judge gates that block bad agent PRs from opening.

Codacy launches AI Risk Hub and hybrid AI Reviewer

Thu, 04 Dec 2025 00:00:00 GMT

December 2025 launch: governance for AI-generated code plus a hybrid deterministic+LLM PR reviewer.

OpenAI: verifying code at scale needs repo tools and high precision

Mon, 01 Dec 2025 00:00:00 GMT

Alignment blog on training/deploying an agentic code reviewer—precision over recall, repo-wide tools, and verification cheaper than generation.

Greptile v3: agentic review with published upvote and action-rate deltas

Wed, 26 Nov 2025 00:00:00 GMT

November 2025 engineering write-up comparing v2 vs v3 on upvote ratio, action rate, and cache economics after reviewing over a billion lines.

Anthropic: initializer + incremental coding across context windows

Wed, 26 Nov 2025 00:00:00 GMT

How to keep long-running coding agents coherent: feature lists as JSON, git+progress artifacts, clean handoffs, and end-to-end testing as a human would.

Amazon CodeGuru Reviewer stops new repository associations

Fri, 07 Nov 2025 00:00:00 GMT

November 2025 docs notice: no new CodeGuru Reviewer repo associations; AWS points users to successor capabilities.

Carnegie Mellon's Cursor study: a 3–5x burst in lines added, gone after two months, while warnings rise 30% and complexity 41%

Thu, 06 Nov 2025 00:00:00 GMT

A difference-in-differences study of open-source projects that adopted Cursor, against 1,380 matched projects that did not, finds a large but transient velocity jump and a persistent rise in static analysis warnings and code complexity, which in turn drags later velocity down. Sonar cited it this week to argue verification belongs inside the agent loop and CI, not only at the pull request.

Spotify Honk: 1,500+ merged agent PRs on Fleet Management

Wed, 05 Nov 2025 00:00:00 GMT

Part 1 of Spotify’s background coding agent: prompt-defined fleet shifts, pluggable agent CLI, 60–90% migration time savings, and new validation needs.

Greptile’s $25M Series A and the Greptile v3 agent rewrite

Tue, 23 Sep 2025 00:00:00 GMT

September 2025 funding + product post: Benchmark-led Series A alongside v3 agentic review, MCP server, and rule-file autodetection.

CodeRabbit’s $60M Series B and the CLI quality gate for AI coding

Tue, 16 Sep 2025 00:00:00 GMT

CodeRabbit frames AI code review as the quality gate for vibe coding, announces Series B, and ships a CLI that plugs into Claude Code, Codex, Cursor, and Gemini agents.

OpenAI Codex upgrades: GPT-5-Codex and first-class PR code review

Mon, 15 Sep 2025 00:00:00 GMT

September 2025: GPT-5-Codex launch with trained code-review behavior and GitHub auto-review / @codex review flows.

Sonar AI CodeFix generally available in SonarQube Cloud and Server

Thu, 31 Jul 2025 00:00:00 GMT

July 2025: one-click LLM fix suggestions for Sonar-detected issues in UI/IDE, included on eligible plans.

Gemini Code Assist in GitHub: PR summaries, inline findings, /gemini commands

Thu, 31 Jul 2025 00:00:00 GMT

July 2025 Google Cloud blog with Delivery Hero notes on Gemini-as-reviewer in the PR workflow.

GEPA: reflective prompt evolution that beats RL on few rollouts

Fri, 25 Jul 2025 00:00:00 GMT

ICLR 2026 oral: Genetic-Pareto prompt optimization that diagnoses trajectories in natural language—strong results including code-optimization search.

Cursor Bugbot leaves beta as a PR bug-finding agent

Thu, 24 Jul 2025 00:00:00 GMT

July 2025 launch: Bugbot GA with BUGBOT.md rules, one-click handoff to Cursor/background agents, and beta stats on PRs reviewed.

Gemini Code Assist for GitHub hits GA on Gemini 2.5

Tue, 20 May 2025 00:00:00 GMT

May 2025 Google I/O update: free individual + GitHub code-review agent generally available, powered by Gemini 2.5.

CodeRabbit brings free AI reviews into VS Code, Cursor, and Windsurf

Wed, 14 May 2025 00:00:00 GMT

May 2025 product note: CodeRabbit ships IDE reviews as a lightweight pre-PR pass, with fuller PR reviews still recommended for production.

Amazon Q Developer in GitHub preview: generate, review, and transform

Mon, 05 May 2025 00:00:00 GMT

May 2025 AWS News Blog: Q Developer GitHub app for feature agents, automatic PR code review, and Java transforms.

GitHub Copilot code review reaches general availability

Fri, 04 Apr 2025 00:00:00 GMT

April 2025 changelog: Copilot code review GA for paid subscribers, with automatic or on-demand PR reviews.

Graphite: AI review for the outer loop

Wed, 26 Mar 2025 00:00:00 GMT

Graphite positions Diamond/Agent as accessible AI PR review paired with stacking, merge queue, and inbox—keeping the post-write toolchain as fast as generation.

Graphite’s $52M Series B and the Diamond AI reviewer launch

Mon, 17 Mar 2025 00:00:00 GMT

March 2025: Accel-led Series B and Diamond as a standalone AI PR reviewer (later rebranded into Graphite Agent).

CodeRabbit adds Bitbucket Cloud to its AI review surface

Tue, 04 Feb 2025 00:00:00 GMT

February 2025 integration note extending CodeRabbit beyond GitHub/GitLab/Azure DevOps into Bitbucket Cloud (beta).

Snyk Agent Fix / auto-remediation for Snyk Code goes GA

Thu, 24 Oct 2024 00:00:00 GMT

October 2024 product blog celebrating Snyk Code auto-fix powered by DeepCode AI.

SnykLaunch Oct 2024: DeepCode AI Fix in the IDE

Tue, 08 Oct 2024 00:00:00 GMT

October 2024 SnykLaunch: DeepCode AI Fix merged into IDE workflows with verified remediation emphasis.

Greptile 2.0: full-codebase graph review, not just diff chat

Tue, 08 Oct 2024 00:00:00 GMT

October 2024 rewrite pitch: Greptile indexes a codegraph and reviews PRs with whole-repo context on GitHub and GitLab.

CodiumAI becomes Qodo: Gen, Merge, and the code-integrity framing

Mon, 30 Sep 2024 00:00:00 GMT

September 2024 rename from Codium/CodiumAI to Qodo, with Qodo Merge (ex PR-Agent) as the review/automation product.

Qodo’s $50M total funding and the integrity-platform roadmap

Mon, 30 Sep 2024 00:00:00 GMT

Same-day-as-rebrand funding note: $40M Series A to $50M total, with roadmap items for Cover, Merge learning from PRs, and Gen agents.

Survey: LLM-based agents for software engineering

Wed, 04 Sep 2024 00:00:00 GMT

Broad 2024 survey of LLM agents across SE tasks—useful map of roles, tools, and feedback loops including review/testing.

Bito’s AI Code Review Agent lands in VS Code and JetBrains

Fri, 30 Aug 2024 00:00:00 GMT

August 2024 product post: @codereview agent in Bito’s IDE extensions for pre-PR reviews, with optional Git context.

SWE-bench Verified: human-filtered tasks for cleaner agent eval

Tue, 13 Aug 2024 00:00:00 GMT

OpenAI × Princeton subset of 500 solvable SWE-bench tasks—still the common public yardstick, with known contamination caveats later.

CriticGPT: train critics to catch LLM bugs

Fri, 28 Jun 2024 00:00:00 GMT

OpenAI’s LLM Critics Help Catch LLM Bugs—specialized critique models that find errors in LLM-generated code more reliably.

TextGrad: backprop for prompts, code, and compound systems

Tue, 11 Jun 2024 00:00:00 GMT

Stanford’s TextGrad treats LLM feedback as textual gradients over a computation graph—including measurable gains on hard coding problems.

Sourcery brings instant AI review into the IDE sidebar

Fri, 07 Jun 2024 00:00:00 GMT

June 2024 release: Review tab in the IDE, PR summaries/guides on GitHub, and multilingual review config.

SWE-agent: the interface is part of the agent

Mon, 06 May 2024 00:00:00 GMT

Custom agent-computer interface for editing, navigating, and testing repos—large SWE-bench gains over shell-only baselines.

Sourcery AI code review GA on GitHub with inline comments

Mon, 25 Mar 2024 00:00:00 GMT

March 2024 changelog: Sourcery code review generally available on GitHub, replacing legacy bot PRs with inline review comments.

Semgrep Assistant enters GA for AppSec triage and remediation

Wed, 20 Mar 2024 00:00:00 GMT

March 2024 PR: Semgrep Assistant GA for false-positive triage, autofix context, and custom rule workflows.

Why AI writes code better than it reads it

Mon, 01 Jan 2024 00:00:00 GMT

Early Greptile thesis: codebases are graphs, not file bags—reading at scale needs structure-preserving index, not naive embeddings.

AgentCoder: programmer + test designer + test executor

Wed, 20 Dec 2023 00:00:00 GMT

Multi-agent code generation with iterative testing and optimization—explicit separation of writing code and writing/running tests.

SWE-bench: real GitHub issues as the coding-agent exam

Tue, 10 Oct 2023 00:00:00 GMT

ICLR 2024 oral introducing 2,294 issue→patch tasks graded by fail-to-pass tests—the default yardstick for agentic software engineering.

Reflexion: agents that learn from linguistic critique

Mon, 20 Mar 2023 00:00:00 GMT

A 2023 framework that stores verbal reflections from failed trials and reuses them—strong gains on coding benchmarks without weight updates.

Memory-assisted prompt editing after deployment

Thu, 01 Dec 2022 00:00:00 GMT

EMNLP 2022: grow a memory of misunderstandings plus user clarifications, then edit prompts for similar future queries—no retraining.

CodeReviewer: large-scale pre-training for review tasks

Thu, 17 Mar 2022 00:00:00 GMT

Early foundation model for automating code review activities—quality estimation and comment generation from diffs and review data.