Articles, research, and commentary on code review technology.
Nick Perkins let one coordinating agent work 13 issues for about seven hours, with 12 implementers and 10 adversarial reviewers. Nine pull requests merged, and the reviewers caught a monitor that would fail on every run and an unlimited mail-sending hole.
Meta describes Autopilot, a system that keeps on-call and proactive maintenance moving with a lead agent, persistent task state, and evaluated specialist agents. Landing code stays behind two distinct human acceptances on the current revision, passing CI, and no blocking comment, enforced outside the model.
Oct 29, the Testkube CTO argues that speeding up review just moves the queue downstream: approved AI-written changes pile up behind CI test suites built for a handful of changes a day, so testing has to become its own system that agents can trigger directly.
Oct 29, Sohit Kumar of Prinevo argues that adding coding agents won't shrink a review queue: define what a change needs before handoff, give review and QA agents separate jobs, and stop starting work when qualified reviewers are saturated.
Compare the Market ran its internal AI merge-request reviewer four ways across 79 annotated merge requests. A code graph lifted inline-comment coverage to 0.696, vector retrieval dropped it to 0.577, below the 0.658 the reviewer got from the diff alone. The graph also cost about four times as much per review.
Mayur Agnani works through why more agent pull requests back up a merge queue: each failure reruns everything behind it, so throughput tops out near 1/(p×T). At 30-minute CI and 5% failures that is about 40 an hour, and adding pipelines buys little.
With pull requests up 3.5x since January, Daylight stopped asking humans to read everything. An AI judge scores each change's customer impact, deterministic rules turn that into a low or high lane, and about half of changes now merge without waiting for a person, with no rise in reverts.
More than 40% of September's pull requests on GitHub began with an agent's draft, up from 17% in March, and by September 90% of them merged within two weeks against about 86% for everything else. The report counts merges, not what happened after.
Oct 28, GitHub made Agentic Workflows generally available: Markdown-defined agents running inside Actions, read-only by default, sandboxed, with safe-output validation and a threat-detection job before any change applies.
At Universe, GitHub put Agent Merge into pull requests and made Copilot approvals generally available. Paired with auto-merge, a PR can go from red to merged with no human in the room, unless you leave one in on purpose.
Oct 28: replaying merged PRs through two independent model reviewers found 3 to 14 critical issues each behind one-word LGTMs. The same harness also showed how an unattended review loop fails to converge without a severity cutoff.
Oct 28, the DevOps co-founder argues that when agents open PRs faster than people can read them, the fix isn't a better reviewer but four controls that let a change earn the right to proceed: prevent, verify the outcome, locate the risk, and contain the damage.
Oct 27, Karell Ste-Marie on Foundry, his Architect, Worker and Auditor agent setup: audited candidates that never reached main, a recovery watchdog that read prose instead of verdicts, and a fix left uninstalled rather than forced past a timed-out approval.
Fiodar Ryzhykau argues the engineer's new job is building a round-the-clock software factory, then warns that continuous availability is not continuous useful work, and proposes measuring accepted outcomes per unit of human attention.
Nomadware describes how its agents keep working overnight: sessions that survive reboots, one task per worktree and PR, a merge gate on its own boxes, and a single coordinating session that is the only one allowed to merge.
Jellyfish's October report, covered by R&D World, finds agents opened or committed to 27.7% of PRs at the median company in September, while merges approved on AI review alone sat at 0.5% (8.5% at the 90th percentile) and end-to-end factories opened 9.4% of PRs only at the 99th-percentile company. Engineers who kept editing agent context files merged more.
Oct 27, Chi Cong: a test passing 6 runs in 7 let a coding agent merge three confident fixes for a race that never existed. The cure was a nightly test-reliability file, isolated reruns against the merge base, and a no-reproduction, no-concurrency-PR review rule.
Tanveer Gill's build log: chores first, a blind reviewer that never sees the author agent's reasoning, the agent that opens a PR owns it to merge, a 4:24am user report merged in 23 minutes, and 43 of 332 factory PRs improving the factory itself.
Jeff An, CTO of the testing startup Momentic, in Forbes Technology Council: when agents write in parallel, verification moves from reading diffs to proof artifacts, several independent checks, and spending expensive test runs at pre-merge instead of on every push.
Milanese et al. compare agent-supported and human-authored merged PRs within 391 Java repositories. Inside the same repo, agent PRs are 1.65× likelier to touch tests and rarely delete them; coverage of production changes looks the same, but agents' test-only PRs exercise less of the code they target.
Oct 26, Samuel Medeiros on a run where the builder said done and the gate scored 55 with a token in the diff. His rule is builder, critic, and reviewer in separate contexts, with a fail-closed security gate, and he lists four ways a passing check can hide a failure.
Oct 26, a how-to on waiting for CI and review on agent-opened PRs finds that Codex, Claude Code Review, Copilot and Gemini mostly comment or post neutral checks. Auto-merge waits only for required checks, and with none required it merges at once.
The Kiro Crew team's follow-up says human line-by-line review lasted less than a day; confidence now comes from a single PR Readiness check that means every check ran and passed, tiered findings, reviewers split by role and model family, and a loop that looks back after several rounds and recommends stopping, reverting or simplifying.
Wenxin Jiang and Mikola Lysenko describe factories built from only standing-objective cron agents and a GitHub board. In one CLI factory daily merges rose 15x, agents filed 95% of new issues and wrote 40 of 58 CI-changing PRs, and at least 5.6% of issues marked fixed had wrong fixes.
Filip Papranec describes a bash-driven loop where code picks the work and decides when it's done, a red machine check can never be overruled to pass, a repeated reviewer finding stops the loop, and new work pauses whenever agent PRs are waiting on human review.
A Stockholm fintech's ticket-to-PR factory: 40% one-shot, 50% fixed after a review comment, 10% closed; reviewer comments become AGENTS.md guidance on merge; and lead time is set by who owns review, not by agent speed.
Jacob Archer tears down one eight-hour night in which agent lanes merged 53 PRs with no human review. A batch bisector caught a real bug for free; a timeout on a busy machine stopped every merge for half an hour.
Oct 25, Denis Zontak writes up one sleepless night building a small software factory on a Mac and a $200 subscription: plan, plan-check, sandboxed build, 1,100 tests, diff review, then his sign-off. The limits he lists are usage caps, review attention, and what the checks can test.
A Cornell and Alibaba study compares 17 ways to add verification to a coding agent's harness across seven models. A fresh-context critic that reviews the changes adds 4 to 10 points of accuracy at little extra cost, while gates that reopen the agent's own work cost up to 1.76x for smaller gains.
Oct 25, Charbel Wakim on running headless Claude Code jobs overnight with a reviewer agent and a merge step. The scheduler kept failing quietly: usage limits, a sleeping laptop, an exit message that said a finished job had died. What made the night shift trustworthy was a row each job updates after every step.
David Trapp distils 24 harness practices from tinycmdr, a single-file agent harness he built and measured in public. The loop guards are concrete: refuse an identical call after two repeats, stop after six, warn on a stall at eight minutes and abandon at twenty.
Since July, a self-built agent has reviewed, merged, resolved conflicts, and deployed across one company's repositories on a 10-minute loop. Compared against Claude Code Review and GitHub's new PR security features, its owner kept the loop and borrowed the designs instead.
Oct 25, the third weekly log for karasu, built entirely through Claude Code. 50 PRs merged in a week. Re-reviews on every fix burned the CodeRabbit quota, so he moved a local review ahead of the web one. When agents skipped that written step almost every time, he turned it into hooks that refuse to mark a PR ready.
Oct 25, Ahmad Dogar on making coding agents prove their work: checks ordered cheapest first, a fixed report format with pasted output, a Stop hook that keeps the agent working while typecheck or lint is red, a bundle budget printed on every run, and screenshots opened by someone who did not write the code.
A solo engineer runs a live legal and claims product with headless agents. Tests are written and frozen before the builder exists, a separate session reviews every change, and the limits he hit at that volume were laptop load and CI budget, not the method.
Oct 25, John Crowley at Port on closing the loop in an AI software factory: capture each run, grade a sample on a fixed rubric, find the pattern across a batch, and write a reviewed fix back into the skill the next run reads.
Drawing on Graphite's Greg Foster and AMD's Anush Elangovan, a Chain of Thought explainer argues that review at agent volume moves from reading every line to rerunnable tests, small stacked diffs, and human sign-off on what enters and leaves the agents' working enclosure.
A study of 349,156 infrastructure-as-code pull requests finds agent PRs much bigger and more likely to touch core modules. Agents that disclose themselves get merged 8.4 points less often for the same kind of change, while their Terraform is no smellier than a human's.
Oct 24, The New Stack: read the factory as a control loop where agents are the controller and verification is the sensor. Track time to signal, escape rate, and concurrent verification capacity instead of PRs or tokens per engineer.
Zobaer Hossain had two Claude Code agents build a client's mobile app while a separate cloud Claude reviewed every PR; 173 of 174 merged between Oct 2 and Oct 22. Review, not code, set the pace, and screenshot diffs arrived only after a broken UI passed every test.
An OpenAI engineer splits review into objective checks (does it work) and subjective ones (is it the right code), argues agents should own the first, and says consequences, not the split, decide how much to trust them alone.
Sohail, Kurmantayev, Mahmoud and Nadi study real Copilot coding-agent sessions instead of benchmark runs. Most PRs run untouched; when developers step in, it's brief, and only the first nudge measurably shortens what the agent does next.
A scheduled content pipeline merged only the exact commit it had verified. The run record landed on the same branch after the merge, so it never reached main, and the human-facing report counted a zero day. 17 of 41 after-the-fact records had stranded the same way.
A 20-agent overnight pipeline reported green every morning while its last verification job re-tested a tree that had already passed. Binding each verdict to file hashes and re-checking them at merge then refused all 23 open agent PRs.
Oct 23: a design for an overnight, human-reviewed software factory where reviews bind to a commit SHA, failed results survive, and the first pilot deliberately feeds it broken checks, changed candidates, and duplicate events.
CircleCI's CTO argues the pull request carries too many jobs at once. When agents write both the code and the tests, a green suite proves little, so CircleCI is trying mutation testing, earlier reviewer feedback, and risk-routed human attention.
Vladislav Mandryk traces September green runs from TechFabric's coding agent that never covered the changed code: an exit code from tail, skipped database tests counted as passes, a release script watching someone else's run, and an output filter that always printed 'no issues'.
A Snowflake engineer scheduled agents to keep pull requests mergeable and answer review comments unattended. The lessons: a separate watchdog, permission checks in plain code, assertions on output rather than run status, and a noisy alert that hid a real break for twelve days.
Responding to the Claude Code team's review post, Smith argues revert-rate guardrails depend on huge volume, internal users and fast rollback, and that small teams should first add contract tests, post-deploy outcome checks and branch protection. His own merge gate once merged a failing PR because of a shell pipe.
Oct 22 Shai Kappel on six weeks of agents across solo projects: a hub loop has run unattended since mid-September, every job ending in a PR a human approves, while the fast per-project loop is parked. Autonomy is a per-project level set by oracle coverage and escalation rate, and no project is above level zero yet.
Oct 22, Harness engineers share a year of internal data from running agents on their own code: pull requests and CI executions up about 7x, contributors up about 30%, and peak-time pipelines queuing on concurrency limits even with spare compute.
GetYourGuide's platform team describes a pipeline that approves, merges, and repairs Dependabot pull requests across 250+ repositories. It handles about 97.2% of them automatically, fixes roughly 1,200 CI and security failures a month with AI, and its funniest failures (deleting the failing test, mocking missing secrets) came from asking the agent to make CI pass rather than to find out why it failed.
Oct 22 Unni on how Cortex's open-source engrams runs: every automation, including its two-agent PR reviewer, is one checkpointed workflow that can park for days on a reviewer, survive deploys mid-run, and perform each side effect once, down to a git clone inside a snapshotted VM.
A small call center company publishes six months of logs from an agent fleet that never stops: 158 merges a day, a night shift that never goes quiet, two fresh AI reviewers per change at about a dollar each, and a review queue that blocks almost as often as it merges overnight.
Rayed Bajwa traced every agent turn in his spec-to-merge pipeline for a month. One feature looped implement, review and verify twelve times while the report never moved, 73 of 303 handoffs failed basic checks, and 82% of verify passes changed nothing.
Oct 22, Jim Bennett at Arize built a seven-station factory and turned every human gate verdict into a label for its three reviewer agents. A "look harder" prompt fix changed nothing; an optimizer-written prompt took correctness misses from 2–3 PRs per run to zero across 19 PRs, and a retuned router cut unnecessary human reviews from 9 of 17 to 6 with no unsafe auto-merges.
After four months of rebuilding its development cycle around coding agents, Alpic reports shipping about twice as fast with quality and uptime flat. Peer review moved from the PR to the spec, every agent PR has a named owner, and weekly codebase sweeps can't run again until their last batch of PRs is closed.
Oct 21 Dautović and Wu: V-Model build-verify pairs across 1,454 matched agent and human PRs; agents looked less build-only, but the instrument failed its own calibration bar and kappa was 0.154.
Trail of Bits reviewed every Patch the Planet fix its engineers got merged upstream and confirmed 21 regressions in over 130 PRs, from 58 candidates (about 36% precision). Its new regression-hunt plugin replays every suspected regression on a running build in a fresh-context agent and produced 25% fewer false positives than a plain Codex review prompt, at the cost of leaving some real regressions unconfirmed.
Oct 21, Saqib Jan on DevOps.com. Agent adoption moved the constraint to the review queue. Qodo's Itamar Friedman puts the arithmetic at two 100-line PRs a week becoming two to five 1,000-line PRs a day; the recommended fixes are capping generated PR size, review agents rewarded differently from coding agents, and putting review wait time and depth next to velocity.
Oct 21, Ralf Anton Beier measures 20 agent-written PulseEngine repositories: changes got smaller and features gave way to fixes, while issue trackers held 26 confirmed checks that passed without checking, surfaces that disagreed about the same fact, and an agent-filed backlog outgrowing human decisions. His fix is manufacturing's: gates proven to fail, one source per fact, and a cap only on the queue waiting for people.
Oct 21: a small issue-to-PR agent workflow measures its own verifier. 5 of 5 seeded bad PRs caught, 9 of 10 real issues merged with no human code edits for $8.43, and one rebuild where the builder wrote a test just to satisfy the gate.
An agent pipeline merged at a 5/5 automated review score and looped the agent on anything less. Drafts were never scored, so "no score" fell into the retry branch. The fix is a third state: not there yet means wait.
Oct 21, jamie.codes on a factory that reshaped itself while building Kiln: the reviewer bot retired once CI could do its job, green PRs waiting to merge were 78% of open time, and open-to-merge rose from 14 to 77 minutes as risky PRs queued for a human.
Oct 21, Checkly's Stefan Judis on the half of the factory loop most teams skip. Tests and review lower the cost of checking work; little lowers the cost of catching a failure after it ships. His setup: a failed synthetic check fires a webhook, an agent follows a runbook skill, and opens a draft PR that proves the fix against the preview.
Violet clustered five weeks of bot review into 1,072 distinct issues and replayed its rules prompt on nine models. The cheapest valid finding beat the priciest model, two reviewers got cut, and Low-risk PRs still needed follow-up fixes.
A worked guide to the coding-agent and AI-reviewer loop argues that the long tail of overnight review rounds was doing hidden work. In its simulation, stopping at one clean pass let 24.4% of bugs through, against 13.1% for the noisy loop; requiring two clean passes kept escapes flat with half the fix rounds.
Oct 17–20, pingdotgg/t3code. Maintainer Julius Marminge's agent merged almost 400 web-performance PRs in 30 hours, then took over contributor PRs from the backlog and landed hundreds more. On Oct 20 he reverted 14 desktop fixes in one PR: all had passed CI and code audits, none had run on a real desktop build, and one broke every existing install. He says the open PR and issue counts have barely moved because intake is so large.
Oct 20 Peter Steinberger: a planner-plus-coder loop moved OpenClaw's web frontend from Lit to Solid 2.0 while he checked in every other day. The piece that kept it on track was a separate agent challenging the work, and the misses were the visual ones.
A medical-software engineer seeded 14 production bugs and four clean traps across 18 pull requests and ran two review skills on the same models. Both caught 15 of 15 bugs. The skill that had to prove serious findings with a test it ran backed 76-78% of real findings, against 14-43% for a checklist prompt, with far fewer nits.
Oct 20, William Kassebaum describes a merge gate where a different agent identity reviews each PR, the tier comes from a path-based risk map, verdicts are pinned to the commit, CI stays a separate required check, and the top tier needs two reviewers and a human merge.
Oct 20, Gojko Adzic. He open-sourced the Claude Code setup behind Votito as a lights-on factory boilerplate: a container, a Bash command regulator, lint rules that turn repeated review comments into hooks, and a plan-to-workflow orchestrator with gates and a final agent that tries to refute each step. His GOTO Copenhagen slides call human review the new bottleneck and claim 80% fewer tokens and 6 to 10 times faster runs once orchestration moves out of the model.
Oct 20, Chandler Gibbons (GitLab). Launching an AI-Native Engineering Playbook built with IT Revolution, GitLab cites its April survey of 1,528 practitioners: 85% say AI moved the bottleneck from writing code to reviewing and validating it, and 78% say review was not designed for AI-scale code. Its answers are a four-stage autonomy ladder with no skipped rungs and a layered verification system where humans leave the loop only when risk and evidence justify it.
Rinaldo Festa's overnight factory on two DGX Sparks shipped six PRs at zero marginal cost, but rejections outnumbered deliveries six to one, and the worst offender was a three-day-old gate he wrote himself.
Treating the agent and AI reviewer loop like retry-until-green on a flaky suite, Agnani simulates a 50%-recall reviewer: stopping at one clean pass let 24.4% of bugs escape versus 13.1% for the noisy loop, while a two-clean-pass gate kept 13.1% and halved fix rounds.
Denys Zaichenko logged every subagent his Claude Code setup launched from August to October. What broke was shared files, cleanup scripts, lying success signals, and an agent that edited its own checks.
Oct 19, Jag Reehal. Shepherd fixes a PR inside the coding agent, Stamp approves or refuses it in CI, and once a week a gardener reads the GitHub record and proposes one small change to skills or lint rules, as a PR with evidence a person merges. It counts only unreverted fixes, needs three accepted cases before proposing a rule, and its first run over fourteen PRs produced one edit.
Oct 19 arXiv paper from Zhejiang University and Alibaba. On 43 enterprise feature requests with 750 audited acceptance points, the best agent fully delivered six; unfinished work and missing wiring caused 72% of lost credit. Three in-house harness releases moved the mean from 59.4 to 63.0 on the same model, inside the error bars.
Oct 19, Marco Mornati: a lead agent, an Opus spec writer, Sonnet developers and an Opus reviewer built Graft's first phase from 22:17 to 08:35. 31,000 lines, 881 tests, 2 blockers and 22 majors caught before commit, about $220–250 API-equivalent. The bottleneck moved to a 51-item human to-do list.
Oct 19, IronBee AI. It ran 21 raw models, up to three effort levels each, three times over Martian's 50-PR Code Review Bench: 7,980 reviews, all open. Ranked with 21 commercial tools, the top group holds twelve model configurations and three tools. Claude Sonnet 5.5 at low effort scored highest for $0.14 and 9 seconds a PR. At their highest effort level most models found more issues but were less precise. IronBee says it will soon launch its own review tool.
Oct 19, Everfinity counted two weeks of public GitHub: 14.5% of opened PRs came from agent branches, Claude Code PRs were closed unmerged 1.3% of the time against 6.3% for everything else, and drew about eleven times fewer change requests per merge. AI bots wrote 37% of reviews, but 92.6% of those were comments, not approvals.
Oct 19, Adam Drag. His Triglav setup runs a long-lived Staff agent, Senior coders and researcher agents that post stances on a shared board. Over Oct 7 to 19: 1,648 messages on 76 jobs, 42 challenges, 192 items routed to him and 176 decisions. 112 of 311 blockers only said Staff was offline, and he estimates just 15 to 25% of messages were real discussion. Two agents agreed on a premise neither checked.
Oct 18, Ben Vinegar's agent factory experiment: personas on their own Linux VMs, a private IRC server, standups and "review my PR". By Saturday afternoon the reviewer agent had scheduled its next check for Monday at 8:00.
Oct 18, engineering leader Fiodar Ryzhykau sketches a team whose agents keep investigating, testing and preparing work overnight and over weekends. His test is blunt: if people come back to 40 new pull requests, the bottleneck only moved. He asks for a review-capacity budget, independent verification, and no expectation that humans stay on call for the agents.
Oct 18, Marco Mornati releases Graft v0.0.1: a GitHub App that records which agent wrote a PR, has one model write an intent card and a second model check it against the diff alone, and applies a base-branch policy file. Shadow mode only; the 90% card-accuracy gate has no pilot data yet.
Oct 18, Hiroki Kondo's weekly log for karasu, a project built entirely through Claude Code. 88 PRs merged from Oct 10 to 17. A cap of 8 open Dependabot PRs split 24 updates into three waves that each needed its own triage, so he raised it to 25. He classified 261 CodeRabbit threads and found about half were real defects; the repeat false positives came from badly scoped rule text.
Oct 18, Patrick Altman. Sentinel turns Sentry alerts into PRs as they arrive and picks two or three backlog issues each evening, so there are fresh fixes to review each morning. In one week 29 errors were fixed, reviewed, merged and released, with a median of 8.7 hours from alert to merge. Other agents review every PR first, then a human merges.
Oct 18, Enes Oğuzhan Alataş. Several persistent Claude Code loops working on his tool Spool left approved PRs stuck in rebase-and-rerun round trips. His Actions-based queue rebases each labelled PR onto the current main inside the runner, reruns the existing CI, and turns green only if the head, base, label and auto-merge are unchanged. Any push after approval removes the PR from the queue.
Across 197 merged PRs and 444 Codex reviews in two weeks, the median PR needed one review but 30 PRs took 261. Sharma capped the review-fix loop at four rounds, moved the bar after round two, and the reviewer then found a bug in the cap itself.
Oct 17, Will Larson. Imprint's engineers merge about 4x the PRs they did two years ago, and median review time rose from about 33 to 77 minutes. Its answer is a build-managed small-change check: under 1,000 lines, no serious findings from the AI review bot, an author with 10+ merged PRs in that service, and a narrow blast radius. A review skill can only approve or ignore, never comment. About 10% of changes qualify so far.
A KaaS newsletter piece pulls together benchmark data, studies and maintainer policies on why agent pull requests stall: at the 75th percentile they sit 17.6 hours before first review, against 3.4 hours for unassisted ones. Its fix is to make the author verify, explain and shrink the change first.
Oct 17, a loop-engineering guide with a noisy-evaluator simulation: "beat the best so far" showed +2.37 against a true +1.16, while a gated keep rule told the truth at about six times the eval cost.
Oct 17 from Balázs Csorba, a practical guide to cloud coding agents that open PRs while you are away (Codex cloud, Copilot cloud agent, Claude Code cloud, Jules). The queue entry is the specification, every run is time-boxed and resumable, CI is the first reviewer, and work in progress is capped per human reviewer rather than per agent. The cost that counts is cost per merged PR, including review minutes.
Oct 16 Wei, Yao, Peng, Wang, Wu, Cheng, Meng and Gao (Microsoft Research / UVA / CMU): Self-HPO alternates harness edits by the current policy with RL under that harness. Average resolve rate 57.8%→64.8%; most of what the agent wrote into the harness fires when it says it is done.
Tessl's factory team swapped a generic AI reviewer for one that checks its own written standards. Real-defect rate rose from 68% to 91%, but the sharper number is that PRs stuck past five review rounds used 47.4% of review-loop spend.
A pre-registered study seeds 46 faults into agent-built services. An independent verifier in a fresh context beat the builder's own tests; swapping in a second provider's model did not add detection.
Oct 16: merges went from about 17 a day to over 600 in six weeks. CI was 79 seconds of a 3.4-minute merge; rebuilding the queue inside git cut the slowest 5% from 114 minutes to about 7 while keeping main green.
Oct 16, Marcin Skalski on Kong's internal agentic review platform. Agents run in GitHub Actions behind a sandbox with credentials stripped. A static triage step replaced an LLM orchestrator, and benchmarks scored models on blocking recall, precision, and cost. 22 teams and 135 repos enrolled; the bot wrote 901 of 1,573 commits in its own repo.
Oct 16, Donn Felker (Polygon): a remote Claude agent invited into Slack channels now opens PRs for engineers, product owners, marketing and the CEO. Every PR gets AI review, then a human review. The safeguard they lean on is the repo harness and merge rules, not who typed the request.
Markus Eisele, in O'Reilly Radar, argues that if every increase in generated code needs a matching increase in line-by-line reading, review caps how much agent work a team can accept. His alternative: agree behaviour and blast radius before generation, require a short review record with each PR, check drift in both directions, and spend inspection where mistakes are hard to undo.
Oct 16, Anthropic's internal study of its own workforce finds code commits per engineer up 5x, no rise in reverts for agent-written code, and human attention as the binding constraint. Meta-work, such as specs, agent-on-agent review and pipeline monitors, rose from 10% to 22% of sessions.
Amp’s documented way of working: core product engineers push to main without mandatory human PR stamps, run agents across the whole ship loop, and still pass SOC 2 with restricted access, signed commits, automated CI, and thread-linked audit trails.
Oct 15 preprint: 60% of sampled Linux mm review threads involve a design trade-off, yet the strongest baseline review agent reaches only F1 0.213. DesignLore reuses structured historical decisions and lifts that to 0.259.
Zach Lloyd says don't start an always-on factory by auto-fixing the backlog, because unreviewed agent PRs sour the team; start with low-risk chores with clear owners, track the share of merged PRs through the factory, then cost per PR, which fell from $80 to $20 at Warp in six weeks.
Oct 15 survey of 239 practitioners (matched HPR vs AIPR questions): 64.9% start with lower trust in AI PRs; 61.1% demand more evidence; would-block under residual uncertainty rises from 19.4% to 43.9%. Review becomes an evidence-ownership problem.
Addy Osmani argues agent-era code quality lives in the constraints around the agent, not in reading every diff. When change volume exceeds what the checks can consume, a queue forms that moves at human speed, and the only moves are more verification capacity, slower generation, or a lower bar.
Oct 15 Mujahid (ForkTrip) on leaving Claude Code agents running unattended: read-only by default, a deny list for every GitHub write, a parsed verdict line or nothing happens, a $25 daily cap, and a missing third answer, BLOCKED, for runs that could not judge anything.
Oct 15 Kacharov (Juma Labs): a fixed plan → work → simplify → review → PR loop took a ten-person team from ~70 to 469 merged PRs a week. Review is the busiest station; the learning-capture step ran 8 times against 187 reviews.
Oct 15 follow-up to Warp's self-improving factory notes: always-on agents that open fix PRs from Sentry alerts, score past sessions and rewrite skills from failures, and turn customer interviews into roadmap candidates — still with a human merge gate.
Lior Grossman on running twelve agent chats at once: Tibo's 'never been this tired' post (500K views), BCG/UC Riverside's AI brain fry survey, attention residue, and Bainbridge's 1983 ironies of automation. Self-reported productivity rose with a second and third tool, then dipped.
A pre-registered study ran 465 agent-authored PRs through a frozen five-agent review-and-fix pipeline on three LLMs. Healthier baseline code meant fewer tokens, independent of PR size, because review agents iterate more on messy code.
Anthropic's managed pull-request reviewer left research preview for Team and Enterprise. Reviews run about 3x faster than in preview, admins choose Quick, Standard, or Deep per repository, findings pass a verification step before posting, and the merge decision stays with the team.
Fabijan Bajo reads Spotify's QCon London talk on Honk — 1,000 merged PRs in three months became 1,000 in ten days, and review became the bottleneck — as a capacity-planning problem: move verification into existing CI, then plan reviewer throughput like any other queue.
Anthropic's Claude Code team replaced 'did a human read it?' with risk tiers set by the change, caps agent-to-agent review at three rounds after one PR ran 50+ overnight, and turns every incident into a daily eval for the reviewer.
Oct 14 field report from an agent team's 16-hour sprint: four gates on every change, five releases, and the best catch was a reviewer reading source who found a security feature tested only against a fake client. Plus the operating lessons, from waiting after the first PASS to resetting the manager agent at 12 hours.
Oct 14 preprint: a two-stage benchmark where models build a coding-agent harness from an interface contract and a consolidated development doc. GPT-6 Astra tops out at 53% pass rate; the weakest frontier model hits 27%.
Florian Bruniaux argues that faster generation only shows up as a verification queue unless review and CI can absorb it. His stop rule: pause new authoring when the shared verification queue exceeds its agreed capacity, because a budget on each agent run does not bound the total queue.
Oct 14, Florian Bruniaux, part 3 of his Engineering the Review Loop series. Freeze the diff, criteria and budget before a review run starts. Record whether the reviewer finished separately from what it found. Stop after two iterations with no defined progress. A missing result never counts as a clean review. The guide is a design to test, and he says the failure paths have not yet been exercised.
Oct 14, Inside Atlassian. In a large frontend monorepo where each team owns its own review rules, a reviewer holding every guideline left confident comments about code nobody touched. The Jira team put a matcher agent and a reviewer agent in separate Bitbucket Agentic Pipelines steps, connected by one file, and spent most of the effort on the guideline documents.
Oct 13 Zou, Zhou, Luo and Zhang give Claude Code a running record of focus, findings, affected callers and unfinished work. Bug fixes rose from 35.1% to 39.5% on a chain of issues, but pass-to-pass regressions rose from 16 to 23, and the heavier multi-role harness did worse than no harness at all.
Oct 13 from Shijie Xie (MiniMax / MCode): after ~50 days of Agent Lord (271 tasks, parallel cross-review), he tore the custom scheduler down. The bottleneck wasn’t parallelism — it was proving the work in a real app environment, with one owner and another model family as judge.
Jonathan Wage maps TradersPost's agent pipeline stage by stage: a gate at every station, risk that agents can raise but never lower, and a person still deciding every merge.
Practitioners from Docker, Keycard, Socket, Barclays and others published an open reference architecture for AI software factories. It puts four boundaries (sandbox, network, supply chain, identity and access) outside the agent, keeps them fixed across three levels of autonomy, and argues that the more an agent decides, the less it can be trusted to police itself.
Oct 13, Thomas Salnot on a night when six specialised agents merged 30 pull requests with CI green and shipped zero features. Three quarters of pull-request time went to stacks, review loops, and hourly handoffs. He switched to owners per product slice and counting screens that work.
Oct 13 from Root Logic: a six-month-stable Maintainer burned half a weekly quota in two days after another agent started handing it work. No permission bit flipped — contact became authority — then a second agent invented a switching cost to defend a pointless route.
Oct 13 Organ: as its agents went from 54 to 126 PRs a week, approved PRs piled up. Five causes in the merge step (a stricter up-to-date rule than the repo's, one-shot merge attempts, shallow-clone rebases, CI inside the rebase budget, a cleanup job killing rebases). Merges rose from 68 to 112 a week; 62 PRs are still open, 38 in conflict.
Oct 13, Jonathan Muszkat at Selenios: with agents writing much of the code, merges per month grew 3.3x and test cases 74x since January. Keeping branches up to date ate about half of each PR's CI minutes. A merge queue helped modestly in week one, and the bigger fix was telling the agents to stop updating branches.
Oct 13 Lamplit Labs: a four-stage review-agent workflow (triage, one review with blocking vs advisory, agent re-checks its own threads, then the human) cut median human review from 31 to 14 minutes across 1,106 PRs.
Harness and Augment's Cosmos describe a continuously running software factory where agents stay with a service after merge, take back reverts and failed verifications, and earn wider auto-approval from their own revert and verification record. The agent cannot merge to main even if prompted, and a human still makes every merge.
Oct 13, a builder running a two-model agent loop on one 24 GB GPU found the judge ran with write access through a dead code branch. The trustworthy signal was an audit log of real before-and-after directory snapshots, not any agent's claim.
Google ran Gemini 4 Argon overnight on the 205,000-line ADK Python repo with maintainer rules in .agents/skills and a pinned goal. It opened one review-ready PR with six optimizations; review kept two (about 40% less session-prep overhead in long sessions, 21% faster cold import) and cut four.
Oct 13, Chi Cong models an agent pipeline as queues: twenty agents, 41 open PRs, one reviewer approving in 90 seconds, merges flat. The fix was a hard cap of three open agent PRs per reviewer, enforced in code, with blocked agents splitting diffs, adding tests and writing verification notes. Fewer PRs opened, more merged, regressions down; the cost was idle agents.
Oct 13, leocavalcante.dev. Two autonomous Pi agents ran unattended from Friday to Tuesday on two small extensions, surviving restarts and a Pi upgrade. They merged 693 self-reviewed PRs at a median of 2 to 6 minutes. Of 664 new issues, only nine came from real use. One extension, whose job is naming a tmux window, grew from 738 to 5,659 lines. Without an outside source of work or an idle state, steady momentum turned into busywork.
As agent volume pushed Anthropic past human-scale source control, a seven-person team replaced git with Scion in five months. Merge-queue locks failed; the durable answer was post-merge, pre-deploy testing plus humans who tell Claude no.
Oct 13, Ziad Antar on the Claude Code process behind Twinpage: a builder and a fresh-context QA agent that do not trust each other, a retro after every feature that adds a rule, and a prune every 10 features that removes rules which never fired.
Oct 12 Zoroval / Chandan KS: an Intelligence Change Gate fingerprints protected diffs, requires a matching eval report, and lets evidence go stale the moment the protected code changes again.
Oct 12 Vapi: an in-house review bot that runs code before it objects cut average open-to-merge from 130 to 65 hours over 1,694 PRs a month. Three cheaper runs beat one maximum-effort run, and no single run found more than 10 of 32 real issues.
Peter Vanhee's adapted Ralph loop runs headless Claude Code 18:00–06:00 on an always-on laptop: GitHub issues labelled ready become one PR each, merging is denied at the permission level, a usage gate fails closed, and the morning is a review shift.
Oct 12, Sonar's write-up of Fin principal engineer Kesha Mykhailov on The Next Commit. Technical plans nearly vanished once agents made code cheap, so Fin made small PRs auto-approvable, harvested senior reviewers' comments into a React skill that both writes and reviews code, and learned that a cheaper review model with identical recall was far noisier.
Oct 12 SoluteLabs field note: INFORM → SPEC → BUILD → VERIFY after a three-line Tailwind change broke sticky nav — named failure modes include verification debt when AI produces faster than the team can prove safe.
Oct 12: an agent running delivery across five repos kept asking “want me to merge this?” Hourly batching made the queue worse; merge-on-green was worse in another way. The policy that stuck: CI green and zero unresolved AI-review threads, then a per-repo post-merge check — plus a race where a resolved thread beat its fix commit.
A testing vendor lines up the software-factory pictures from Gartner, BCG, Stripe and OpenAI and finds the same gap: every check reads code, none opens the running product, and the teams closest to lights-out built that station anyway.
Erik Yu of Parall counted fifteen thousand of his own messages to the repo. Once a merge queue and self-review landed, "rebase" and "I merged it" fell to zero. What's left for people is a two-layer gate that skips reading code, and it is now the bottleneck.
Oct 12 Paperclip guide to unattended overnight runs: sort work into proceed-to-review, prepare-then-decide, and wait lanes, and put each limit in the layer that can actually enforce it.
Oct 12 from Manish Tiwari: provenance trailer, scope diff against the task contract, a clean no-network sandbox, mutation score on changed files, then a CODEOWNERS human. 221 of 412 agent PRs merged, review rounds 1.4 vs 2.1 for human PRs, zero merged without human approval.
Oct 12, an agent named Ishikawa writes up ten days running the merge loop on akhighlight, a C port of the Pygments lexers: why a five-minute-cron fleet could not finish, why the shared registry file was the real bottleneck, and why its own verification script and briefs failed it.
Christian Koch describes a harness where support and sales staff start fixes, agents cleared a dependency backlog overnight with one human 'yes', an evening review session reads the day's agent sessions, and a bug under a human-merge requirement waited seven weeks.
With pull requests per engineer doubled in a year, Clipboard built a narrow LLM reviewer that can only approve low-risk changes or hand them to a human. It approves 28% of merged PRs, cut median time to merge for low-risk PRs from 14 to 19 hours down to 1.2, and costs $0.35 a review.
Oct 11 from Chris Vella: more parallel agents can just grow the review queue while the branch moves under it. He proposes coherent slices with explicit acceptance criteria and stop conditions, an evidence handoff on every PR, a two-in-progress / two-awaiting-review limit per reviewer, and a scorecard that ends at acceptance.
Oct 11 from Favio Vázquez’s showtime harness manual: one person and a crew of agents, 16 PRs merged by ~12:35 on 9 Oct, each through his gate. Relayed approvals do not count; both reviewers must say MERGE on the current head before he is even asked; rules come with the incidents that wrote them.
Jacob Stephens replayed 183 review sessions from thirdshift, an unattended factory whose PRs merge themselves. A fresh same-model session matched or beat a second family at every checkpoint; the clear win was a second look after a whole feature — and making authors run a test for every finding.
Mike Nakhimovich gave one agent a goal and a leftover token budget. It fanned fixes out to isolated workers overnight, then switched to batching merges when asked not to flood the queue. He declined one PR in the morning.
Sabry E. Farrag measured 92,042 inline review comments on agent PRs across 2,034 GitHub repos: 6.5% name a repository item outside the diff, rising with odds ratio 1.35 per tenfold increase in code files. Human PRs show a similar size gradient; a robust agent-only excess is not established.
Oct 11 from Mike Curry: agent diffs are three times larger and arrive without a first human judgment, so reading harder fails. Push lint, types, coverage, and a second-agent first pass into the harness before anyone opens the PR; keep humans on intent, shape, edges, blast radius, and operability — and never let a machine approve a merge.
Oct 11, Frank Csehan (AJAGARA). After a role-based agent team produced tests and reports but few usable features (78 of 87 tasks with no clear evidence), he moved three projects to a loop: a fixed task list, a scenario the agent cannot edit, a time budget, a serial merge gate, and a nightly regression run that reopens anything that slips. 93 of 100 game tasks green by Oct 11; a speed gain is not yet shown.
Oct 10 from ROBOCO (Dohyun Jung): GitHub API counts across seven open-source factories. OpenClaw lets a bot merge ordinary code behind risk tiers and an opt-in/stop command (576 of 803 automerge-labeled PRs merged). Everywhere else people press the button, but 368 of Bun's 446 September robobun merges (82%) went in with the approval requirement still outstanding.
Oct 10 from ROBOCO, the last of four posts on open-source software factories. Across Bun, Astro, OpenClaw, mastra, Beads and others, failures get fed back into agent instructions, CI checks, workflow mechanisms, and decisions to scale automation back. OpenClaw added a success-rate circuit breaker after its implementation worker went 52 for 308 in a week. The missing step is verifying that new rules actually work: no project in the survey regularly measured it.
Oct 10 from ROBOCO: with Bun's robobun opening 2,313 PRs in September, open-source factories make reviewers read evidence before code. Bun's PR evidence block (1,827 PRs), mastra's CI that runs new tests against the unfixed base, Vercel AI SDK's reproduction → failing test → fix commits. And the gaps: mastra bot PRs merged with the gate's failing label, an Astro fix reverted despite a confirmed repro.
Oct 10, Lawrence Liu ran a plain Ralph loop on a small library: four rounds, 72 cents, every test green including 13 held out. A fresh-context review found a spec violation none of them covered, and he lays out what an unattended loop needs before anyone depends on it.
After 24,000 prompts of harness work on an Android app, Lal found that rules in CLAUDE.md kept coming back until they became hooks, that review results moved more between identical runs than between most prompt changes, and that 3,460 green unit tests let three never-built tasks through.
Oct 10, GuildBuild on running a 26-role agent factory on Codex with outside investigators on Claude Code. Most failures were not in the agents' code but in the evidence around it: a check ran on a deployed revision that never had the fix, a skip counted as success, and a copied factory still loaded old instructions.
Thomas Fairhurst on a two-founder startup whose agents merge 50 to 100 changes a day on one server: a merge line cut median merge time from 16 minutes to 86 seconds, and an overnight agent run cut held-out gate time 29% under an anti-cheat rule.
Jamie Duncan's Leaf factory runs four loops, and only the one that adds features needs a person. Over a month the simplification loop merged more PRs than the feature loop (617 to 529), cut 18% of a monorepo's product code while tests grew 12%, and was implicated in later fixes about 1% of the time. His design test: a loop runs unattended only when a machine can tell it is done.
Oct 10 field guide from Cloudsail: the agent is the smallest part of the factory. Merge policy lives in deterministic code with a named owner, a fresh-context reviewer checks the exact candidate commit, and checks-only merging is unlocked per task type only after the numbers hold.
Oct 10 from Patrick Altman: agents write almost all the code, merges went from 8 to 24 PRs a week, and a cross-model review loop catches regressions tests and human review missed. The cost was the slow thinking that writing used to force.
Oct 10 from Aitherium: every PR gets Atlas on architecture and Demiurge on code, both on a self-hosted runner calling a local DeepSeek. Demiurge REQUEST_CHANGES’d a small voice fix with seven real findings; a machine-readable VERDICT: line gates auto-merge.
Oct 9 Google Cloud AI Research preprint (code open): Propose–Probe–Compose refines an agent’s prompts, tools, and skills from behavioral contrasts alone, then keeps an evidence-backed lesson library for the next task. Gains up to 11.8 points with the model frozen.
Oct 9 field note from Robin Wieruch: every developer owns a personal AI review skill, authors answer with agents, and a weekly scorecard turns review quality into something you can tune.
A matched study of 26,999 pull requests finds agent changes that edit application code plus CI or provisioning files take 3.24× longer to merge when reviewed, while 43–56% skip human review entirely, sometimes passing a pipeline they just edited.
Oct 9, Prime Intellect. A swarm of more than 2,000 agents ported Prime Agent from TypeScript to Rust over two weeks. The human work was the verification: four kinds of parity check, a separate adversarial reviewer on a different model, and a fresh-sandbox verifier. Then a three-day hillclimb merged 69 performance changes with no numeric target.
Oct 9 Mock & Russo, Empirical Software Engineering: in a TDD experiment the autonomous agentic workflow was fastest and passed ~95% of baseline tests, but added branches the spec never asked for that its own tests left uncovered; humans wrote the better test suites.
A review vendor ran 30 mostly AI-written apps, sampled 20 of 160 findings and had two checker models verify them. 7 of 16 agreed verdicts were false, 0 of 9 critical labels survived, and its fixes changed the labels rather than the false positives.
Oct 9 Neuron livestream with Microsoft Research's Alex Lavaee: separate author from judge, put deterministic checks first, save execution evidence, and scale the verification budget with risk while agents run for hours.
Oct 9 Maurice Heumann: agents decompiling a shooter for three months showed steady visible progress on semantically wrong code. The reviewer agent accepted the workers' own justifications. A byte-matching check, hashed in CI so agents couldn't edit it, removed the reviewer and let cheap models carry the run to 99% of functions present, 83% byte exact.
Oct 9, Peter Eisentraut. About 8 to 10 significant features were reverted from the PostgreSQL 19 branch after beta began. One of his reasons: LLM-assisted review now surfaces a long tail of edge-case defects at once, each with a fixed handling cost, and then you run out of time. The finding rate outran the fixing rate, and the code predates vibe coding.
Oct 9 preprint from CMU and Microsoft: 1,800 review tasks built from human-verified bugs. The best review agent catches 71.9% in its top five findings, up from 47.7% in five months, but drops up to 25.5% on proprietary code. Lights-off only for narrow, low-consequence PRs.
Oct 9 write-up from Cockroach Labs: a GitHub Actions agent pipeline modeled on a teaching hospital, where plan review, adversarial code review, and a discharge audit sit between always-on agents and main.
Harvard study of Jellyfish data from 700+ software firms: coding agents raise lines, commits, and PRs, but issue and epic resolution stays flat while time from PR to merge grows 49%.
Oct 9 essay from AdrienF: the feeds sell 2,500-PR months and durable orchestrators, but a working harness that ships for the business beats chasing the perfect always-on setup.
Oct 9 from 8090 Labs (Arjun Goray): seven models × five harnesses on DeepSWE’s 113 real tasks. GLM moved 24%→59% by harness alone; Claude Fable 5.1 scored 41% in Claude Code and 61% in Codex; cost per success, not cost per run, flips which wrapper looks cheap.
Oct 8 from Swizec Teller (Plasmidsaurus, 21 engineers): a few hundred PRs a week and 16% auto-approved, but six draft PRs by 3pm and one small bugfix shipped by evening. 40–50 human reviews a week, half of BugBot’s comments rejected, and every repeated review comment turned into a deterministic linter before auto-merge is allowed.
At LPC 2026, Google's Roman Gushchin updated Sashiko, the open-source agentic harness that watches Linux kernel mailing lists. Phoronix reports 191k patch reviews across 99 lists, 463 CVEs citing it this year, and 1,277 upstream commits. On a 1,000-regression benchmark, Gemini 3.1 Pro found 53.6% of bugs that had already passed human review and merged — and the slides' own caveat is 'Clean pass != bug-free.'
Oct 8 from Reveneau: a claude -p job that publishes every hour with no human review started a dev server to confirm its page rendered, couldn't kill it, and blocked five runs. Fixes: the runner owns the verifier, time caps set from 40 measured runs, bounded retries.
A survey of 239 practitioners finds reviewers don't reject AI-authored PRs outright. They gate them on scope, rationale, validation beyond CI, responsiveness and a named owner, and call the missing work reviewability debt.
Fortan Pireva's team of five engineers, each driving a Claude Code session, built a procurement backend over two months and wrote every incident down next to the rule it produced. The repeat offender was a check that covered less than the report built on it, plus an hourly reminder loop nobody read.
Oct 8 field report from OCDevel: a scheduled Claude Code session works a shared queue every night, merges routine changes after review, caps itself at three items and one new issue, and emails what needs a human.
Oct 8 Stephen Margheim (ZAR): seven engineers already generate plenty of code with Claude Code. The pain is attention — rare-but-widespread production failures sit until a human trips over them. He wants always-on agents watching unified telemetry, investigating anomalies, and opening PRs — a foreman walking the floor, not another generation pipeline.
Miikka Koskinen (quanttype, Oct 8): coding agents produce more code than reviewers can keep up with and review agents are now half-decent. His answer is to experiment bottom-up, including post-merge review and no review on low-stakes work, and to name the fear behind holding on to review as the last human checkpoint. Two cures: stronger guardrails and exposure.
Oct 8 Hosaka: in a fleet where agents write, review and merge every change, four 2026 papers on agent review and LLM judges become gate rules. An approval covers only the tasks the reviewer can do, a rubric score is a noisy sample, and a second reviewer should be told to refute, not to agree.
Oct 8 systems note from Lucas Firmo: markdown rules are requests, hooks are decisions; a second-model signed review and proof artifacts gate every PR before teammates see it.
Oct 8 from Bobby Donchev: human-authored merges went from 1,111 to 3,683 per fortnight with the same 97 authors and 80 approvers. The response is 222 evidence-backed rules, checks the PR under review cannot loosen, a fail-closed verdict owned by the platform rather than the agent, and an admission that the review machinery itself became the bottleneck.
Oct 8 DHH on the Campfire rewrite: agents run for a day across competing implementations, he goes through every PR they opened, verifies the work, merges what's ready, then sends the agents back with architectural improvements for each one.
Oct 7, the YATFA team traces a single ticket through a night of unattended work: two writing agents, three reviewing agents, two platform bounces on a branch that went stale, and a merge at 03:00 UTC. Their argument is that the queue, not any one agent session, is what carries work through failure.
Oct 7 Yao, Wang, Sen, Chen, Ray and Song audit 4,487 benchmark-passing trials from frontier coding agents across five benchmarks. 1,545 (34.4%) fail a new test that the reference solution passes, cutting the overall resolution rate from 50.6% to 33.2%. A cheaper sampling version finds 72.4% of those failures for 15.4% of the cost.
Oct 7 Bowen Xu and Boyu Chen run a second coding agent from a different provider as an advisory reviewer. It found something material on 8 of 20 paired turns. Testing the harness turned up three ways a failed review was counted as a pass: cut-off input, cancellation while the process was shutting down, and a nonzero exit that still printed a tidy verdict. The last one was found after formal data collection had begun, so that data had to be set aside.
Oct 7 Stack Overflow (series finale): agents are brilliant for an afternoon and mediocre for a quarter unless the harness supplies a constitution, autonomy boundary, tiered context, durable memory, multi-lens review panel, and post-edit hooks — then shared written state for fleets. The model is maybe a tenth of the outcome.
Oct 7 from Rupesh Poojary: OpenHarnX freezes the agreed test suite, verifies the locked copy in a sandbox the agent can’t reach, and issues check-only verdicts — execution rights vs settlement authority.
Oct 7 Pankaj Nakhat reads the CMU/Stanford study of an enterprise 2x mandate: PRs per developer hit 2.09x while reviewers grew 1.5x, human review coverage fell from 89% to 68%, substantive review from about 39% to 21%, and silent approvals roughly doubled per reviewer. His fix is lanes at PR creation, narrow bots as gates, agent-supplied evidence and a WIP cap.
In an Oct 7 JetBrains Air post, Gleb Malkov reports discovery calls across agencies, telcos, game studios and consultancies: individuals get faster with agents, but team delivery stalls. Shared setups stay informal; a mapping company saw PR time go up; and overnight automations create a new ops burden of ownership, distribution and governance.
Oct 7 Li, Li, Jiang and Lou: 2,124 matched runs of Codex, Claude Code and Kimi Code with dynamic concurrency on or off. Runtime rose in 14 of 15 agent-benchmark cells, Claude Code fell from 83% to 59% on SWE-bench Verified, and gains showed up only on the longest tasks. Shared-file and merge problems were the largest failure category at 33.2%.
Jang Young Jeong audits HoYeon Lee's report of a Software Factory that produced 300 commits a day over the Chuseok holiday. The workflow is real and thoughtful, with tasks carrying intent and a verification method, human review at key points, and an Observer agent, but the post reports no merged, accepted, reworked or rolled-back counts. Jeong's fix is an acceptance ledger.
An Oct 7 DEV post by Info Inlet: a pipeline of author agent, review agent, tests, staging and auto-merge ran fully closed for about nine days until a write path that acknowledged before persisting locked a paying customer out at 2am. The fix kept agent review and moved the one human to owning each production merge, plus one added question on every write path: what happens if the client retries?
Oct 7, F200.ai. A pre-release Apache-2.0 daemon runs unmodified Claude Code and Codex CLIs under a deterministic policy gate, fenced leases, stop-hook checks on a clean checkout with tests taken from base, a cap of 15 open factory PRs per repo, and a merge train bound to head commits. The team's own work has run on it since Oct 5, and the post says it has measured no productivity, cost or quality gain yet.
Oct 7, Aaron Breckenridge: one command turns a Jira ticket into a worktree and a detached Claude Code session that runs until a watcher confirms on GitHub that checks pass, the PR is approved, and every thread is resolved. A 'needs answers' state replaces guessing.
Oct 7, Nikolas Barwicki. More than 100 tickets across three projects with no human in the implementation loop, the longest run about 10 hours. His setup is deliberately boring: one source of truth, questions answered in advance, a cold plan review by another provider's model, and checks the agent can run itself. One screen still went blank while every test stayed green.
Zhu et al. run 2,500 trials on Claude Code, Codex CLI, Gemini CLI, gptme, Qwen Code and Copilot. Auto-approve raises utility and nearly triples attack success; allowlists and denylists cut attacks with far less utility damage than network isolation.
Oct 6 Paul Stack: when one agent writes the code and its tests, moving tests up or down the pyramid doesn't make them independent. What matters is where the expectations come from and who is allowed to change them, so product-level checks have to be defined and protected outside the implementation loop.
Oct 6 SignalFire, from a closed CTO show-and-tell: the teams running the biggest agent fleets built almost nothing to generate code faster. Their machinery feeds the agents and checks the output, auto-merge is allowed only when a failing regression test can prove the fix, and reviewing 40 parallel outputs is now an interface problem.
Replay's pilot chained Fullstory (detect), Replay QA (diagnose and verify) and Obvious (fix) on its own repo, with an engineer's merge as the only human step: 124 distinct bugs, 76 fixed on main at about 5.4 a day, and the argument that proof, not generation, made the PRs mergeable.
Michael Quoc of Product.ai and the security writer ToxSec on running agent loops for hours unattended: a fixed outcome file with result, limits, tie-breaker and check, a read-only verifier that grades every finished chunk, restarts from the last unit that passed, and human judgment moved to the two ends of the run.
Oct 6 from Máté Németh: LLMs stay non-deterministic, so his autonomous loop aims for the same landing conditions, not the same diff. 653 rounds since August, 108 PRs merged by plain code since Sep 29, refusals pinned to commits, and a one-way guardrail ratchet that still needs consolidation.
Oct 6 field report from Daniel Krydynski: an unattended scout/implement/review loop let a 4B reviewer approve gutting INSTALL.md from ~256 lines to 36. The fix was a deterministic deletion cap, not a better prompt.
In an Oct 6 AI Engineer talk, Tessl's Dru Knox splits factory progress into autonomy (how often humans correct the agent), automation (how much ships without review) and quality, in that order. He says the loop work that drives them is unplanned and keeps losing to feature deadlines, and proposes manual takeovers, human PR comments and agent-started PRs as the scoreboard.
Oct 6 Joxo: when several people's agents open PRs the same day, sort by dependency rather than arrival, review shared-file PRs together and rebase after each merge, let a fresh second agent check first, and keep the merge decision with a person. If the queue grows faster than people read, a reviewer agent won't fix it.
Oct 6 JetBrains Research with Lund University: from workshops with 17 practitioners and a 43-person survey, a framework for reviewing AI-generated changes that starts with an overview, ranks files by risk before anyone reads a line, and only then drills into code. Agents present every line with the same confidence, so reviewers need tools that tell them where to look hard.
Oct 6 Cate Huston, Twill: PR volume rose 24× from April to September while features fell from 66% to 16% of PRs and guardrail work rose to 36%. Same-day hotfixes fell from about ten to one; median PR open time went from 1.2 to 3.5 hours.
Oct 6 paper from NVIDIA, UCLA, MIT and Tsinghua on a Claude-builds, Codex-reviews loop with 72 mechanical gates. 118 public postmortems: builders over-claimed in 45, and two thirds of rounds came after the work was already accepted.
Oct 6 GitHub made stacked pull requests generally available on all plans. Stacks now enter the merge queue as a single group, auto-merge for whole stacks is rolling out, and the gh stack CLI supports worktrees. GitHub reports 9% more merged code in repos using stacks during the preview.
Oct 6 GitHub engineering blog (Brian Celenza): September saw 7.38 billion commits, more than five times a year earlier, with pushes up 4.9x and pull request merges nearly 4x. Agents that checkpoint after every action make push latency their speed limit, and merge queues funnel the work onto a single reference. GitHub is separating durable storage from serving compute and says branch protections and required reviews stay intact.
Oct 6 Ferris and Bonilla: a six-month case study of a production healthcare platform built by coding agents and run by an operator with no formal software training. Reviewer agents, tests and monitors helped, but some measured proxies, one audit stopped without failing, missing checks vanished from pass rates, and an automated repair caused an outage.
Oct 6 dsalathe, Greens That Prove Nothing: in an agent factory's findings ledger, the largest defect family is guards and tests that stayed green while asserting nothing. Seven of one shape got past writing, rereading and review, several by agents that had just read the postmortem, and fell within minutes to a deliberate mutation. The house rule is to show every guard failing before believing its pass.
Oct 6 CodeHerder: merges landed every two to five minutes while a full main pipeline took ten, so runs were cancelled before they finished. A per-job time cap, checks picked from the import graph, and hourly full suites whose failures become agent tasks took them from about 74 merges a day to over 200.
Oct 6 Codecot factory sheet: a spec-driven agent loop with a rival-vendor reviewer opened 440 PRs over five months and ran a product for 19 days unattended. Its own defect list shows why a passing suite is not the same as a checked claim.
In an Oct 6 LeadDev essay, Spotify engineering manager Emma Bostian argues that AI moved the bottleneck from writing code to reviewing it, that reviewers end up reconstructing the author's reasoning, and that engineers who launch several agents and switch tasks pay for it in rebuilt context. Her fix is author responsibility, a defined bar before review, and measuring review time and rework next to output.
Oct 5 yureki_lab: after one bad 'cleanup' task poisoned four later tasks overnight, every task now gets a full working-tree snapshot on a hidden git ref, automatic rollback when verification fails, one commit per passing task with a Task-Id trailer, and a permission rule sorted by whether an action can be taken back.
Oct 5 Ewan Mak reads the pstack repo behind Lauren Tan's 2,500-PRs-a-month claim: the agent that wrote a PR never merges on its own verdict, a swarm of fresh verifiers must clear the exact head, and CI must pass on a freshly rebased branch. Most of the volume is maintenance, and the approach stops where changes are not reversible.
Oct 5 Routines Runbook: Anthropic's own docs say a green status on a Claude Code routine only means the session started and exited without an infrastructure error. Three ways unattended runs fail while reporting success, and an eight-point pass that makes an unreadable source look different from an empty one.
Oct 5 Barnaby Robson maps pstack's six mental models. The review-relevant one is the gardener: an agent queues smells without fixing them, she groups the queue every few days, samples merged PRs daily, and changes lint rules, types and skills when several agents take the same shortcut.
Oct 5 Lawrence Liu (Orbi): an issue-to-merged-PR loop shipped two regressions its own review passed. A night of graded reruns found the fixes were generated inputs and an oracle independent of the patch, then found the stronger harness cost about twice as much and caught nothing extra on nine held-out bugs.
Oct 5 Bob Odell: audit of 14 unsupervised local-lane tasks — ~36% first-pass success, five rescues, one 11-hour freeze, one prod revert — and zero failures at a model-capability ceiling. Deadlocked test gates, scope leaks, blind green checks, and broken recovery paths were the seams. Prefer gates over prompts.
Oct 5 Marc Nützel: a dark factory is supposed to replace the human reader with specs, harnesses and gates that fail loudly. The data he lines up says many teams got there by accident. Agent PRs mostly go unreviewed, AI PRs merge a third as often, and the gap is set by what surrounds the agent.
Christoffer Niska (Oct 5) turned his plan, build and review skills into dim-factory, which runs each step as a separate locked-down Claude Code worker. A station is done when the factory says so, a single append-only log overrules git, and a commit is kept only if it is one new commit, leaves the test script alone, and passes the project's check on a clean copy. He has not yet measured whether it saves him time.
Oct 5 Sumanyu Muku (Amira Learning): partitioning declared scopes and ordering merges along the dependency graph lifts clean integration from 1/9 to 9/9 simulated scenarios. On a live contract rename it goes from 0/15 to 15/15 on Opus 4.8, where reactive conflict warnings stay at zero.
Oct 5 Adam Monago on Lauren Tan's 2,462 changes in a month: verification loops make checking cheap enough to stop being the ceiling, but the number of changes a named person can honestly stand behind grows on a human schedule, not a model one.
An ESEM 2026 paper from Meta traced AI-suggested code character by character through review and landing across 10K+ developers and 400K diffs. AI code was in 52% of diffs and about 8% of landed bytes. It was associated with roughly 8% shorter review time per unit of log AI fraction, but it lands disproportionately in less central code, test files and larger diffs, so naive with/without comparisons mislead.
Oct 5 Law Zava: measure offered review demand in review-minutes against effective reviewer capacity, tier review by risk before capping anything, and when the cap binds, put agents to work shrinking reviewer minutes rather than adding submissions.
Oct 5 Andrew Kesterson devlog: a self-hosted factory of six agent identities polls a Forgejo forge, grooms tickets until a checker script calls them ready, and works them while he sleeps. About 980 PRs merged across 38 repos since mid-August, with 73 times his earlier lines per week. His own read: the bottleneck moved from typing to reviewing, and the always-on loop was not entirely healthy for the human.
A difference-in-differences study of 669 developers finds agent adoption tied to longer PR descriptions and more testing. The slide in review thoroughness shows up in control repositories as well.
Oct 5 GitHub launched ReviewBench, an open benchmark of 219 pull requests sampled to match 103.9 million real ones. It scores review agents on precision and recall by severity, gives credit for valid findings outside the answer key, and its offline results pointed the same way as a production Copilot review experiment.
Oct 5 Adam Faik walks Uber's factory through 70+ public sources. Engineers' first review grew from 3 hours in 2024 to 9 in 2026. The useful details are the shape of the response: a flag-cleanup loop that runs when CI is idle but limits how many diffs reach engineers, checks moved before CI, and a PR that arrives with a table of everything it passed.
Oct 5 Luke Chadwick (Far Horizons): always-on Mac mini with ~eight tmux sessions, multi-model plan review before build, a verification gauntlet the agent can't talk past, and scheduled Pulse/Hygiene skills that re-check 'done' and open tickets for drift. Spec and verification absorb the time writing used to.
Oct 4 Zest (Ilya Volodarsky): every agent PR in two repos carries a self-written session block. Median 37 minutes from open to merge. Matched on repo, engineer and week, the authoring model barely moved the AI reviewer's should-fix rate, while 1,000-line PRs were flagged 36 points more often than sub-300-line ones.
Teamscale's Sven Amann cites Faros data showing review time up 441.5% as AI adoption rose. A plausible-but-wrong agent investigation on a 2.2M-line codebase was settled by system tests run against the real implementation, not by closer reading or a second model.
Oct 4 Arjun Iyer (Signadot): Anthropic's CI jobs grew 25x in six months and Linear's suite nearly quadrupled, yet faster CI still verifies a repo, not the distributed system. He argues verification moved from a human gate to an automated one without anyone asking what the gate checks.
Oct 4 Alexey Indeev (Spare): coordinator-plus-sub-agent goals that run overnight and for days, auto-merge in small flagged pieces behind tests, Bugbot, Strix, and a Mergify queue — and tokens, not review, become the bottleneck.
Oct 4 Alexey Indeev, on his company's internal operations platform: one coordinator agent per goal, sub-agents doing the work, auto-merge behind tests, bots and a merge queue, and goals left running overnight for days. He reads progress reports instead of diffs, and the new limit is tokens.
Oct 4 Emmanuel Goossaert: nine weeks and 421 commits building a seven-stage factory, with tests that must fail first, OS-level locks on test files, and a hermetic verify. Then he stopped reading code, let it build itself, and found zero evidence it beat a plain Claude Code session.
Oct 4, Gong, Zhang, Jahangirova and Wang (KCL / Bristol) show feedback-guided code revision beats text-only PR attacks on AI reviewers: 20.8% success vs Sonnet 5 and 15.7% vs GPT-5.5 across 159 CVE PRs, rising to 44% when the attacker can also edit PR text and replies.
A team that has built its product with coding agents since May keeps people at three stations: writing the task, answering the agent, and reading the diff before merge. Its commit history shows why, starting with a check that shared the worker's blind spot.
Oct 3 Saagar Patel: one prompt at 9:56 PM, a ten-hour fleet run across 269 projects, 654 pull requests merged in the window. Every merge needed an independent review of the exact commit, green required checks, and proof the merged tree matched the reviewed one. Writing was cheap; proof was the cost.
Liu and Han (Shandong University, arXiv 2610.04433) hold five models fixed and swap the harness across Claude Code, mini-SWE-agent and OpenCode on SWE-bench Verified. Swapping harnesses flips 13% of hard tasks, the same as rerunning one harness. Cost per task moves up to 3x, set mostly by the preamble each harness resends on every step.
Oct 3 Vinod Kurien Alex, an engineering lead at a B2B SaaS company, went from 11.6 to 113.6 merged PRs a month with 3 to 4 reverts in a year. He credits a layer of handoffs, a router, scheduled routines and review agents, and is candid that the fair range is 4x to 10x.
Oct 3 FlowVerify: agent PRs wait 17.6 hours at p75 for a first review against 3.4 for unassisted work, yet review itself is only modestly slower once started. A simple queue model shows why, and why bursty agent batches make it worse.
Oct 3 Matt Cockayne: an architecture review of go-tool-base produced 21 critical and high findings, so a fleet of agents took one each. All 21 merge requests came back green, and he couldn't tell a working fix from a test that could never fail. The fix was to change the shape of review, not its intensity: every agent had to show its test failing on untouched main first, with sibling tests passing as a control.
Oct 3 Associates AI: every AI-written PR must bring a numbered spec, a test seen failing then passing, a fresh-context review, green checks on the head commit, and a stated blast radius and rollback. One fix took twenty hours from issue to merge but twelve minutes from opening the PR. The gate still leaks, and the post lists where, including a health metric that stayed green while an agent answered nothing for days.
Oct 2 Zhang et al.: RSI authorization testbed; historical scores keep unsafe programs in 22/48 framework histories despite correct alternatives; score refresh + validated rollback restore correctness while retaining >43% deployment savings.
Oct 2 yureki_lab: an always-on Claude Code system reviewed each morning. Failed runs write a believed-vs-actual incident file, a separate agent distills rules, and a rule enters the playbook only if it fixes a replay that failed without it and breaks nothing else. Capped at 60 rules.
Oct 2 Yang et al.: MCRI (Metadata/Constraints/Resources/Instructions) on 63,812 OpenClaw skills; 58,275 skill-conditioned runs; top-1 selection lifts +17.7 / +22.8 / +19.6 percentile points vs strongest baselines on BigCodeBench / BFCL / Mind2Web.
Oct 2 Wiedmann et al.: 8,640 main runs on 4 scientific coding tasks; ~54% score variance is run-to-run noise; Information outranks Model and Budget; prompted self-verify barely moves behavior—oracle verification tool triples it.
Oct 2 Wang et al.: letting a harness optimizer rewrite its own prompts and tools made things worse until it could run draft edits against real tasks; with execution checks it learned to add regression tests, and its edits still broke nearly as many tasks as they fixed.
Oct 2 Vo, Doshi, Sheinin (IBM): 240 trials; 162/175 runs that pass final numerical checks still fail a process check (92.6%); template regression suites + live-API ground truth across specs, harnesses, and models.
Oct 2 Song: 900 review sessions; CCR and top-tier cross-model review share Jaccard 41.2%; one CCR + one XMR matches 56.7% planted errors vs 42.7% for two CCR reviews; lightweight cross-model reviewers add nothing.
Oct 2 Rehman et al.: 944 filtered GitHub issues across 21 LLM multi-agent repos—Orchestration & Execution is 22%, Memory & State 14.7%; Workflow/Tool/Memory problems dominate causes; Optimize Workflow leads fixes.
Oct 2 Qualiteg field note: five ways model “Done” slipped past (syntax-only, assert-free tests, listen-but-crash, 2/22 APIs); harness starts the app, hits every specified route, and inspects test bodies—pushbacks carry failing lines and how to investigate, never the answer.
Peng, Calvo, Kalu and Davis (Purdue) study 1,130 agent and 1,130 human performance PRs through June 2026. Agentic PRs merge 54.4% vs 73.3% for humans, use similar optimization strategies, and report validation at comparable rates — but lean more on static reasoning than benchmarks, and about half of validated PRs in both groups report no quantitative performance metric.
Oct 2 Nalivator3000: solo Archon/Paperclip/LiteLLM dark factory — ~$0.015/task, ≤4-file cuts at 40–50% one-shot; silent credit fallback, alert-to-agent trap, 14-day improver cooldown, script-owned merge.
Oct 2 Mehta/Ritchie/Chen (Surge): one epoch of GSPO on 1,700 expert coding tasks with partial-credit reward zeroed on any pass-to-pass failure lifts Kimi K2.7 Code on all six external benches (+4.7–20.0 pp), shorter trajectories, four last-mile habits.
Oct 2 LaunchDarkly (Alexis Roberson): a seven-station factory built from GitHub Actions and a headless agent, where the builder is never told about its reviewer, the default token silently skips the factory's own CI, and a post-merge smoke check opens the revert.
Oct 2 Kato: coding agents fit the tests they see, so GTDD has a separate tester generate fresh inputs after each committed candidate, a trusted evaluator return reduced counterexamples as regression tests, and fresh random audits decide acceptance.
DispatchSEO counts its own unattended agent pipeline: 40 merged PRs, nobody in the loop for the typical one, 4 held for a human by a path allowlist, and a review check that only reported on 15 of 40.
Cabinlytech (Oct 2) runs a standing team of Claude Code builders plus one advisor agent that merges and deploys, all day and night. Each ticket carries its own failing and passing commits, a red ticket is dropped from the release train rather than debugged inside it, and the advisor reads the running commit back from the server before closing anything. Their verdict: more rules do not make it faster; model usage limits and disk do.
Oct 2 Bindschaedler et al. (MPI-SWS / MIT / Purdue): approval as ledger state + credential-confined commit adapter; four-condition predicate; 271k traces replay; docs-only bypass rule doubles W-Tran disagreement—ledger attributes it.
Oct 2 Ask HN from ruffrey: a senior engineer says Claude merge requests take five times longer to review and they barely understand what they approve. The replies split three ways: stop reading code and invest in the harness, keep the agent on short leashes you can review in minutes, or tier by how much the code matters.
Oct 2 Dan Apushkinsky: an AI-written PR passed an AI reviewer and a human who pasted it into AI, missing a bug flagged by a comment fifty lines up — correlated passes are one opinion repeated; make review a credited, estimated ticket.
Oct 1 Zhang et al. (Google Cloud AI Research): same-model agentic verifier with disagreement resolver + consensus challenger; +6.2/+6.4 over single rollout; verification skills self-evolve from failure feedback.
Oct 1 Zhang et al.: SFT+RL teaches when to compact, what working state to keep, and how to continue; +9.2 / +5.0 abs pass on SWE-bench Verified / SWE-PolyBench; 256K never overflows, 16K shares length-triggered fallback.
Oct 1 Yang, Chai & Wen: HAT anneals teacher trajectories under weaker harnesses so the model internalizes what to investigate, revise, and stop; tools-alone annealed ckpts ≈ full-harness starters on SWE-QA (+5.01 / +3.09 H₁ for 9B H₂-targeted).
Oct 1 Wang & Huang: local-first harness with ten mechanisms (byte-level net-zero prefill, finish gate re-reads task, signature loop detection); LRAB 0.886 vs goose 0.631 / opencode 0.479 / agent-mini 0.405 on 288 published cells.
Oct 1 QA.tech: at Nathen Harvey's GOTO Copenhagen keynote, half of 440 engineers picked reviewing changes as their biggest SDLC friction and 3% picked building. One team went from about 15 PRs per release to 150-200 with no new reviewers. The proposed split: invariants, a rules-loaded review agent, behaviour checks with an explicit unverified state, then a shorter human question.
Oct 1 Park et al. (Microsoft/POSTECH/KAIST): ActiveSaddler treats scenario selection for harness optimization as a non-stationary bandit over failure-pattern arms; +4.4 / +7.5 pp Pass@1 over the same optimizer with a fixed scenario order.
Oct 1 (updated Oct 9) from Lawrence Liu: an issue-to-release loop that delivers its own repo merged a PR 71 seconds after a human labelled it ai-blocked, because the gate re-read CI but not labels. Also: headless claude -p exits 0 when every write was denied.
Oct 1 from Máté Németh: moving his autonomous loop to a claude -p runtime took the merge away from agents. The runtime re-tests the exact commit in a fresh checkout. Background commands cut off 4 of the first 6 agents, and a busy machine and a stale check run caused two false refusals.
Oct 1 Li et al.: 66 configurations × three terminal benchmarks; Claude leads GPT by 7.9 pts in OpenHands on TB4 but trails by 30.2 in PI; best harness flips for 4/5 models—evaluate the pairing, not the model alone.
Oct 1 Heng: observation-aliasing bound + shared-budget TeamBench—showing the live count still overspends 4/5; commit ownership or escrow hits 0/5. Models propose; the harness owns distinctions and commit.
Oct 1 Guo et al.: 411 execution-labeled traces; official evidence lifts weak reviewers (5/6 improve catch and cut over-rejection); deployable cascade recovers catch 0.76–0.80 but over-rejects ~0.66—decisive checks beat bigger judges.
Oct 1 Fang et al. (Peking): training-free selective clarification discovers candidates, filters relevance/resolution, then keeps questions only when programs under alternate answers disagree on shared inputs—beats harnesses and baselines on ClarifyCodeBench.
Sep 30 Zhao/Li/Zhang/Du: hindsight-distilled 0.6B–1.7B sentinels choose Allow / Redirect / Hard-Pause pre-execution—up to +14pp on SWE-bench Verified Mini and +10pp on Ask or Assume.
Sep 30 Zhang/Wang/Xu/Metaxas: a small RL editor adapts a globally optimized harness per task from the outer-loop playbook—SWE-smith-MR 50.7→64.0 (Haiku) and 70.7→88.0 (Gemini), often with fewer steps.
Sep 30 Zabolotnii: RAC binds gates→authority transitions; score-only admits 80/100 block-required injections at vendor weights; conjunction admits 0; score matches only when τ≤min weight.
Sep 30 Yang/Lai/Wang/Harari/Gabrilovich/Chang: research-guided module mutation + crossover around a fixed backbone—Qwen3.5-27B AppWorld Challenge TGC 49.6→63.6%; GPT-5.4-mini τ² Telecom pass@1 72.7→81.9%; Meta Harness falls short under the same search budget.
Sep 30 Yang/Jiang/Liu et al.: Git-style requirement↔work snapshots; Git Agent picks compatible historical state; +wins in 28/30 backbone–harness settings; Code tokens −73.6% vs native while resolved rate rises.
Sep 30 Xu et al.: on MATH-500, identical baseline copies already give 2.16pp oracle headroom; generated harnesses show persistent losses on 100 tasks vs 1 win; frozen selector +0.00pp; at 27 executions portfolio ties clones at 98.70%.
Sep 30 Wei, Tan, Wang: optimizer-agent harness search; B=5→100 lifts held-out 47%→67%; unconditional accept drifts down in ≤10 rounds; Champion–Challenger raises held-out 51%→67% over 30 rounds.
Sep 30 Wang et al.: hold model+task fixed, vary workflow reliability; frontiers use good guidance yet drop under bad/mixed; SFT on bad workflows hardens override, RL restores selective use of held-out good ones.
Sep 30 Vangala & Malik (Missouri): 1000 agent runs across Claude/Codex/Gemini and four languages show functional green while declared, installed, and runtime-traced dependencies systematically disagree—SBOM-grade manifests are not trustworthy without independent tracing.
Sep 30 Tan et al.: 265 repo runtime errors; best Codex+GPT-5.6-Terra PR 38.11% / CR 28.68%; HealGuard flags 17.4% of passing heals; 100% recall on 342 unsafe cases at 68.42% FPR.
Sep 30 Tan/Gu/Li et al.: capability-overlap-without-solution-overlap bench for skill-evolution harnesses—static skill banks slash multi-hour coding time >55% as an execution compass, without magically bypassing exact implementation.
Sep 30 Shi et al.: leave-one-out sandbox ablations under the same rule+task; witnesses = both runs pass tests under strictly less authority; 314/314 AIShellJack attacks detected, 3.75% FP on benign GHAgentFiles.
Sep 30 Rovai case study: dual Python/SPARQL redundancy gate printed ALL CROSS-CHECKS AGREE while 3/7 published counts were wrong (disposition 932→220); 72/75 LLM “independent” paths still returned the defective 932.
Sep 30 Qian et al.: VeriCodeBench 400 self-spec problems (C/Java/Rust/Python); Direct joint success ~44% vs ~71% code validity; CodeNova lifts aggregate joint to ~49% (Claude Sonnet 5 65.5%); richer specs can cut validity while raising coverage.
Sep 30 Qian/Zhu/Li/Wang/Ji: Builder learns when/provide/use meta-skills from Target feedback; full-bank +8.95pp over no-skill construction, +12.02pp over handing the same bank to the Target; same-model self-improve +18.71pp.
Sep 30 Qi/Xu/Huang/Shang/Wang: inject into test-reached paths → 1,300 Java cases; best agent (Codex+GPT-5.4) averages 68.9% witness success; long contexts 80%→52%, depth-10 transforms 80%→59%.
Sep 30 Qi et al.: of 1,262 AIDev performance fixes, 57% merge; acceptance tracks repo history with the agent (33%→84%) more than coded content; only 11% carry a performance test; of 30 re-executed merges, 18 deliver, 9 show no gain/regress, 14 change untested inputs.
Sep 30 Poličar/Špendl/Hočevar: humans set scope + correctness + verify; Claude Code overnight on openTSNE (4.4–7.7×), ssGSEA (136×), and a new graphlet algorithm (~160× vs Orca).
Sep 30 Peng et al.: 100 enhancement-proposal tasks across 29 large systems (~2.4M LoC, ~5.5k-LoC PRs); best of 28 agents hits 14% resolved / 52.7% F2P; localization dominates; file-tree+API specs lift resolved +16–22pp.
Sep 30 Pavlenko et al. (Microsoft): Hotspan targeting + Darwin evolutionary swarms on ECTs; Autoperf 0.35 PRs/engineer-hour (3× agentic coding); Darwin up to 22× token-efficient vs OpenEvolve; OSS top-20 avg +13.8%.
Sep 30 Pan et al.: 1,199 PR review comments (764 trustworthy / 435 untrustworthy); frontier LLMs miss most bad comments (GLM-5.3 negative recall 20.77%); Sentinel evidence-seeking judge reaches 76.60% accuracy (+19.78 pp over base, +6.13 over GLM-5.3).
Sep 30 Oota et al.: Plan+ReAct preserves declared structure on only 22–45% of trajectories (worse as plans lengthen); pattern-specific executors lift ALFWorld 0.48→0.92 and SWE-bench Verified 0.36→0.44; mode selection still trails best fixed pattern.
Sep 30 Mohsenzadegan/Tavakkoli/Kyamakya: deterministic deny>review>allow governance; GovernBench 60k cases → 0.9888 accuracy, zero aggregate false allows / GASR; remove evidence gate → FAR 0.2873, GASR 0.5715.
Sep 30 Li/Jiang/Huang/Yang/Peng: HERO gates efficiency until the rollout group can resolve, then cuts unproductive low-entropy turns—on Qwen3.5, better resolve than GRPO with ~40% less token use.
Sep 30 Kashmira/Dantanarayana/Tang/Mars: persistent execution-dependency graph + observed reuse; at 6K history budget, matches/beats full history on AppWorld & 8-obj QA while cutting inference cost 10–32%.
Sep 30 Kang et al.: sample N candidate terminal actions and verify at the model–harness boundary—GPT-5.6 Sol verifier lifts TMAX-9B Pass@1 50→68% on TerminalBench-Lite; pairwise + distillation recover part of the gap without retraining the generator.
Casey Holt approved a tidy background-agent PR adding webhook retries. Three weeks later a customer found zero retries: the config key sat where the loader never read it, and every test built the policy by hand. Holt now reads the task first, the tests second, and runs every agent change once with real configuration.
Sep 30 Gibert/Despujols/Rochet: CIB deck harness beats same-model direct generation by 20.4–33.6 pts (text-only), but repeated LLM judges shift identical decks ~8 pts MDC—progress across rounds can be indistinguishable from grader noise.
Sep 30 nick-delgado: review-agent-pr → address-pr-review → improve-agent-process — parallel standards/smell/spec/test reviewers, falsifying verifier, root-cause log on a tracking issue that batches process PRs.
Sep 30 Cheng et al.: automate reproducing real harness bugs from open-source agents into Live-Harness-Bench (200 executable bugs); +10.67–27.56 pp reproduction vs SWE-Factory/SWE-bench-Live; distilled repair skills lift harness-bug fix rate +6.32%.
Sep 30 Chen et al.: K semantics + verification kit; HumanEval 164/164 Pass after two repairs; audits catch 12/12 seeded defects that still prove; KleverBench kit vs rules/advice mixed (± intervals include 0).
Sep 30 Chen/Zhang/Wang et al.: reflective search jointly learns request division + specialist programs on a mixed workload—family-mean 52.6→70.6 vs 62.5 for single-program GEPA at ~18k scored calls.
Sep 30 Brilliantov/Hernández-Cano/Abbé: under matched backbone, hardware, and 24h budget, a minimal single-session coding agent (Malena) matches or beats elaborate open-source MLE harnesses—coding-agent access dominates further scaffolding.
Sep 30 Bellibatlu/Wang/Zhang: executable contracts over 34 mutating tools in four benchmarks find seven tool defects + one ungrounded grader; MedAgentBench writes report success with no write; on 1,120 constructed paths tau2’s evaluator rewards the telecom defect and fails its repair.
Sep 29 Ziadé/HF: nightly Transformers CI agent — reproduce on GPU, relore memory, 5× red-to-green verify, then PR; 29 merges / ~80 days; funnel throws away weak candidates early.
Sep 29 Zhang et al.: 43 pairwise + 18 irreducible 3-way failures across AgentDojo/ASB; incremental interaction hypergraph + runtime monitoring cuts residual CFR to ≤1.27% while checking ~20–34% of actions.
Sep 29 Yang/Shi et al.: 11 native-ecosystem from-scratch tasks; GPT-6 Astra 10/11, Claude Opus 5.5 9/11, Grok 4.7 High 7/11; failing submissions still pass 89.9–99.4% of hidden tests—long-tail rules and single omissions, not missing subsystems.
Sep 29 Xu & Wu: of 3,001 MCP error messages in 150 servers, half of next-steps depend on caller capabilities agents lack; a terminal-auth hint drops recovery to 45% (6% on GPT-6 Astra); naming a server tool raises it to 84%.
Sep 29 Britton Russell (Rangle): when generation outruns checking, senior engineers turn into full-time review queues. Track senior attention per unit of net value, raise the floor with automated checks, and point people at what the checks stop.
Sep 29 Qi/He/Liu et al.: alternate agentic RL with trajectory-driven harness revision; candidate kept only if validation improves with model fixed—17/44 proposals rejected; +6.43/+9.09pp over weight-only RL on OfficeQA/AutomationBench.
Sep 29 Pham/Nguyen/Chen/Ye: four long-context tasks stress coding harnesses (OpenCode, mini-swe-agent, RLM, ReAct); best 68% macro; GPT-5.6-sol RLM vs mini-swe-agent on memos 92%/$7.23 vs 100%/$0.583.
Sep 29 Orlanski/Zhang/Trost et al.: 15 design tasks × 242 problems; score = correctness² × simplicity vs production-library references; Opus 5.5 48.9 vs production 46.6; 64% excess code from rigid/hard APIs, not missing features.
Sep 29 Mao/He/Wang et al.: execution-driven self-evolution + prefix-preserving sampling + MegaMCP; WEFT-14B beats Agent-World-14B by +6.41/+2.23/+12.27pp on BFCL V4 / τ²-Bench / Claw-Eval.
Sep 29 Li: fixed judges make version-dependent errors on coding-agent pairs (250 SWE-bench Verified); 8 judge-only intervals declare upgrades execution cannot; false accepts rise with capability; prefer paired audits over judge-only release.
Sep 29 Kuang et al.: Harness Primitives mined from failed trajectories + deterministic compilation—SWE-bench Verified Pass@1 73.0→80.5%; Terminal-Bench 2 60.0→72.2%; composition cost ~2.7% of an actor attempt (638× cheaper than codegen).
Sep 29 Hsu et al.: shared runtime + Adherence audit; unconstrained teams realize collaboration ~47%; tool/budget/topology knobs push above 90%; Duo SOTA on Terminal-Bench 2.1 (83.15%) and DeepSWE (69.91%).
Sep 29 Guo/Zhou/Zhang et al.: contrast success vs fail on the same issue; executable predicates verify gain; retain rules only if ≥2 repos agree; DeepSWE +10.3pp over no-skill, +5.0pp over Trace2Skill.
Sep 29 Gan/Gong/Liu: AppWorld-P preference benchmark + approximation/generalization/optimization decomposition—context vs control reachability; memory length U-curve (~0.77→0.20 then rebound); ACE/TEPA/TRACE plateau ~0.4–0.5 vs oracle 0.071.
Sep 29 Dong et al.: branch harness search with evolving development subsets + proposal policies, then route among branch heads — relative gains over Meta-Harness of 34.8% Olympiad math, 11.6% Terminal-Bench 2.0, 3.8% SWE-bench Lite (66.0% resolution) with development-only selection.
Sep 29 Ding et al. (Meta): 186 empty-workspace tasks × 11 languages; pass@1 11.7–67.7% across 13 models; iterative static+rollout audits flip only 2.2% of outcomes; front-loaded planning peaks in first ~4–12 turns.
Sep 29 Dai et al.: across 16 frameworks no evidentiary record; examiners attribute faults 74–91% but <10% of citations are independently witnessed; only an outside second writer catches omissions/fabrications a self-hash-chain misses.
Codesteward agentic review: policy loads only from base branch; specialists + verifier + non-model judge; degraded graph fails closed; silence returns a machine-readable why-empty rationale.
Sep 29 Chu/Lee/Park et al.: retrieve GitSkills sections, Cross-Harness Adaptation into target tools; RASI init without rollouts + RASU feedback updates; beats SkillOpt/WikiSkill/GEPA on OfficeQA/Spreadsheet/ALFWorld/WebShop.
Sep 29 (revised Oct 1) Tom Baldwin, Polymorphism: a sizing model for agent fleets built on contention, coordination and rework. Rule of thumb: up to five agents per product and no more than reviewers can check. In a first trial, even a 21-second model reviewer became the bottleneck at ten agents.
Sep 28 Zhu et al.: FTA fixes failed observations before generation; false-success 22.8%→0.8% and fabricated detail 28.3%→0.8% under a structured evidence contract across 3,600 human-annotated responses.
Sep 28 Zhang et al.: meta-learn harness edits with a frozen solver; 4B proposer lifts mean single-step OOD Reasoning Gym scores 0.32→0.62 and beats its 35B teacher on average; HotpotQA training transfers to MuSiQue/2Wiki.
Sep 28 Zhang et al.: Effect Commit Contract — bind approval to one occurrence, compare actual persistent changes to approved multiset, EXACT/DIVERGED/INDETERMINATE; 0 incorrect commits across 39 task–fault cases ×3.
Sep 28 Tiwari/Vass/Singh: bind every agent claim (tests-pass, no-secrets, behavior-preserved) to the Merkle hash of its dependency cone — cone binding re-verifies 7.9–81.9% of claims vs 100% for repo-hash; gate blocks 9/9 scripted adversarial behaviours including deleted failing tests.
Sep 28 Teng, Cai & An: executable judge transfer lifts cold-start WebShop mean reward ~13% over fixed-rubric guidance (exact success +36% relative); better final selection alone closes only ~10% of the gap.
Sep 28 Roy: purpose-based access for proprietary solver modules fails under ordinary Claude Code controls — containers, permission rules, and sandboxes cannot name which program is reading; 5 of 15 routes evade both rule surfaces, including gfortran -E and allowed make targets.
Sep 28 Rajput et al.: Security Debt Line Integral couples SAST risk to historical-best test progress — 3.9%/26.2% agree2-positive among working SWE-bench/ProgramBench runs; gowsdl debt arrives in one write and persists.
Sep 28 from Máté Németh: a plan, implement, adversarial-review, merge-gate loop has merged 1,263 of 1,388 PRs on his app since June. Checking takes 42% of the tokens against 36% for writing, and the loop's own machinery grew from 35% to 62% of merges.
Sep 28 Ma et al.: over 3,000 multi-turn repo runs, agents read 83.6% of relevant code at turn 1 but only 35.4% by turn 5; self-reuse falls even under saturated recall, and 50.8% of task chains hold a cross-turn re-implementation while pass rates barely move.
Sep 28 Luo/Luu/Elfeki et al. (Scale): process-verify 3,810 surface passes; SWE-Bench Pro violations 24%→73% Opus 4.7→Fable 5 then 11%/0% on Fable 5.1/GPT-6 Astra; repair needs replay + fresh eval.
Sep 28 Liu/Dai/Du/Yang: 10 revival + 13 reconstruction tasks behind hidden graders; 28 verifier defects (24 FN, 2 FP); CFD graded a self-reported zero; three checks—achievability, candidate consensus, never grade a self-report.
Sep 28 Damian Karzon, part 2 of building localagent-box: a loop mode that stops when the tree stops changing and ignores 'done' while tests are red, an auto-review agent queued on every agent PR, autofix agents for the findings, and a 40-file review that cost 3.7 million tokens and skipped the reviewer's own code.
Sep 28 Jeon companion: grade filter → undetermined; vocabulary 36%→98.4%; ingress refuses 3,070 writes — none of the three give the 32 decision rows admissible input; coverage ≠ reach.
Sep 28 IdeaHorizon: AfS encodes long-horizon research as iron laws — commitment before measurement, unforgeable freezing, hash-chained ledgers, compounding memory — so multi-day agent campaigns cannot forge freezes or invent discharges; two closed campaigns (50h assisted; 8.85h unattended Ising) illustrate the machinery.
Sep 28 Han et al.: replace Stage-1 LLM execution-free verifier with policy hidden-state filtering — EF-verifier tokens −66.6–81.0% and total verification tokens −49.1–62.1% at K=16 on SWE-bench Verified; DeepSWE hybrid Best@16 59.26%→60.06%.
Sep 28 Ghosh: hash-chained Claude Code hooks capture intent, model self-report, and executed tools; anchors catch re-chains; seeded-fault splicing shows rules+LLM reviewer catch concealed misalignment.
Sep 28 Du et al.: ExpOS regularizes post-training actions; Reviewer-Guided orchestration grows a research DAG — 54.49 vs 46.53 on PostTrainBench at 0.0% hacking; 35B beats Instruct on LiveCodeBench-v6 (41.21 vs 37.36).
Sep 28 Driven to Develop: a writer who teaches focus and slack describes checking an agent fleet compulsively because idle capacity feels like a running meter, and names the real shortage as well-formed tasks, not compute.
Sep 28 Das et al.: 48 longitudinal sequences; evolved agents 47.2% completion vs 35.7% baseline but 43.9% attributed safety failures vs 0%; tools/skills surface worst; CoT monitor cuts 70.9% harm at 9.7% FPR.
Sep 28 Luke Chadwick (Far Horizons): on a half-million-line portal the agent merges and deploys its own PRs. The trick is a build that takes up to two hours, strict AI reviewers that must be satisfied, hooks that redirect instead of block, and a rule that work isn't done until the deploy is green. A 1:15am brief became twenty merged PRs by nine.
Sep 28 Bobadilla-Suarez, Suh, Fortin: stationarity dichotomy — fixed reachable edit set ⇒ saturation; escape only if the scaffold expands. Best-of-k buys rate not ceiling; same-family vote fails 42% of tasks.
Sep 28 Aly/Assaf/Kobti: MARCH on CodeJudgeBench ties 78–95%; same questions on both solutions 76.6%; withhold non-discriminating comparisons → 20.7%→36.9% accuracy while answering half.
Sep 28, Matt Adams: a mini PC works his GitHub backlog from midnight to 7am on weekdays. 83% of 189 implementation slots never started the agent because nothing was eligible or three PRs already awaited review. Median implementation ran 17 minutes; browser verification about 39.
Sep 27 Wang et al.: 120 real multi-repo tasks across 41 ecosystems; best config (Codex CLI + GPT-5.6-sol) fully solves 42.5%, yet often greens one target while leaving another unfinished — joint workspaces help features, independent runs mainly recover omitted scope.
Sep 27 RaysonTech open-sources nomArmy. Of 21 implementation jobs that reported done, 5 (24%) failed independent evidence checks — including tests that still passed after the production change was reverted. Of the 15 that passed both checks, review still found real defects in 5. Worker, orchestrator, harness, and validator should not trust one another.
Sep 27 Parashar et al.: scoped harness edits from failure classes raise Terminal-Bench checks 63.7%→70.3% and UltraHorizon 47.6→59.8; intermediate evidence checks repair, answer-rebuild critics break more than they fix.
Sep 27 Mei et al.: persistent audited notes + hybrid triggers lift resolve up to +15.0 pp on SWE-Bench Pro; Opera-guided student rollouts match teacher distillation on held-out repos without TB2.1 collapse.
Sep 27 Li et al. (NeurIPS 2026): Counterfactual Rollout Replay lifts SWE-bench Verified 41.7% vs 36.4% GRPO (+5.3 pp); equal-wall-clock still +5.0 pp; Search-Select control shows credit assignment beats branch picking.
Sep 27 Kent Alstad (KandiForge): hundreds of agents → millions of lines across 20 repos; bottleneck is completing/hardening; post-integrate test gates arrive too late.
Sep 27 Elina: Maat validates agent-to-agent handoffs against a versioned contract with no model in the gate; on 522 trials, attributable halts lift rubric scores +7.7–29.1% in five of six workflows and cut model-call cost 17–53% when defects fire early — with a blunt audit that 37% of v1 halts were false alarms from validator misconfiguration.
Sep 27 Ariño & Pafka: 584 coding-agent runs on one ML task show within-pairing SD (0.0107 AUC) exceeding spread between pairing means (0.0095); ranking observed agent gaps needs ~39–113 runs each, while best-of-3 compliant artifacts lift median holdout +0.0081 — and rule-breakers occupy the top of the scoreboard.
Sep 26 Zhang et al.: dependency-centric async multi-agent SE bench — ADPR/DRS over 19 tasks, 52 deps; Test Pass ≫ ADPR (71.6% of Async-Manager-RO outcomes); coordination often closes in a mid-run hopping window.
Sep 26 Tian et al.: long-horizon SRE benchmark with overlapping faults, alert streams, and change-driven defects — best Marathon-Score 41.3/100 across ten methods; agents often correlate/locate but repair ≤2.4% while faults stay active.
Sep 26 Jung/Bilkhu et al. (Apple): open event scheduler + hybrid simulated clock compresses 30-workday MetaClaw to median 5.9h (162×); ten harness×memory configs × ten models; post-train through unmodified Claude Code/Hermes/Codex.
Sep 26 Michał Jaskólski, AI Developer Toolkit: score every ticket on oracle, blast radius and reversibility, and only pre-approved task classes go overnight. Overnight output has to be reviewed within one working day or the run didn't count, dispatch is capped at the next day's review capacity, and the first escaped defect demotes the class.
Sep 26 Patrick Hughes: false green after scanners read 0 of 5 repos — append-only event receipts so later no-op runs cannot bury a real fix; missing evidence is unknown, not green.
Sep 26 Hu et al.: mini-SWE vs OpenCode across 10 Qwen/DeepSeek models × 3 benches; NanoHarness (+tools, task-specific subagents) recovers most product-harness lift — complex scaffolding is not free.
Sep 26 Du et al.: Relic turns recurring multi-agent friction into organization-owned executable protocols — complete-contract delivery 14.06%→19.76% over matched structured teams; fresh-member transfer 41.2% with bindings vs 34.6% text vs 25.4% none.
Sep 26 Alenezi: discrete-event sim separates AGC from AGT across five DORA metrics — AGC raises CFR/rework under both capability profiles; AGT lifts DF +31–34% and FDRT −29–38%, but can raise escaped defects when autonomous CI repair masks failures.
Sep 25 Salas: OCC for institutional agents — reject-and-rerun on stale deps/components; complete mediation + outbox; 18/18 mediated effects, 8 bypasses detected only with independent journal.
Sep 25 Matt Pocock at AI Engineer Paris: agent-triggered work turns a factory into a slop cannon unless checks, a separate reviewer that commits its fixes, and risk-labelled PRs sit in front of humans. Every one-way door gets human review, not every two-way door, and a retro turns each review into checks for the next run.
Sep 25 Mukherjee & Tanjim (Adobe): MoMHa joint reward J=0.482 vs 0.198–0.422 baselines; safety 0.781; +2.7 overall vs two-phase at 95 fewer tokens/example; wins 5/7 held-out real-world columns.
Sep 25 Hu et al.: 1,200 trajectories — three behaviors hit 79–98% of tasks and up to 22.75% of cost; structure-aware retrieval can raise spend 28%; developer-designed skills cut cost up to 41.73%.
Sep 25 Díaz, Pérez & Gil-Borrás: 5,435 agentic repos; reweighted any-mechanism prevalence 21.7% (65.3% in >100★); mean breadth 1.15/5 even in popular stratum; rule files dominate, persistent knowledge ~1.9%.
Sep 25 Arman & Molybog: roundtrip optimizer hits fidelity 1.0; source-withheld resolution 0.08→0.71; with source present, compact/full/retrieved context never beat issue-only across models and ten repos.
Sep 25 Afoud & Wu: pair each spec level with an external verifier; deterministic gates + optional judges; unverified ReAct answers 0/10 (all hit caps), verified VG answers 8/10 and abstains with cause.
Sept 24 from Leonardo Piñeyro (Pento): Stripe merged 1,300+ no-human-code PRs a week; Ramp’s agent writes 3 of 4 merged PRs. Factories sell the queue and gate, not the model. Autonomy only as wide as cheap verification; the lit factory keeps a person on merge while the gate moves upstream to specs and risk.
Sep 24 Marmelab audit of 246 repos + 57 pubs: same model 68%→88% across harnesses; 60% of big harnesses untested; prose security rules ~4–16% enforced; Vercel tool-strip and OpenAI merge-throughput caveats.
Sep 24 Li: Limbo grades 25,930 episodes — frontier models nearly never duplicate lost acks (0.5%) but duplicate 56%/74% under late commits/redelivery; keys everywhere cut duplicates 28%→4%; agents overclaim success in 90% of duplicate episodes.
Sep 24 Leith: 21k-LOC wholly Claude-authored tool — 14.3% of generation events introduce errors AI tests catch pre-commit; checkable facts 94.3% accurate; design-fix proposals 79.4%; ~1 in 4–5 responses err.
Sep 24 İşcan: candidate generation ≠ release authority — 21/22 fabricated ready plans on unanswerable tasks all rejected; 0 false releases / 83 gated (U≤3.54%); ungated shell 9/18 false.
Jul–Sep Patel lab: spec interview + adversarial plan review before code; held-out sealed suites; exec-loop ledger. Three-arm result demoted blind merge gate (5.9× cost, same defect)—fresh adversarial eyes still pay.
Sep 24 Docker: Cloud Sandboxes put the same microVM agent isolation on always-on Docker compute — move local↔cloud with one command, run hours-long / parallel agents, review in the morning.
Sep 24 Unni: open-source engrams runs agents in your cloud — scheduled CVE/dep PRs, Datadog hotspot rewrites, error triage with dedupe, Linear+Slack end-to-end with microVM sub-agents.
Sep 23 Zhang et al.: closure-bound approval — 40→13 residual gaps across 111 pairs; effect-bound pre-auth predictions cut residual effect instances 10→3 on holdout; Vite install and MCP tool calls omit file/network effects.
Sep 23 Roelants evidence report v3.66: bounded agent pipelines exist; strict lights-out ownership of live software does not — productively complicates overnight-harness optimism with lifetime-cost and review-boundary gaps.
Sep 23 Jeon: production gate green for ~2 weeks while one subgate ran 0/2 and another 6/8 — violation-count keys treat absent data as pass; add unable-to-determine that fails closed.
Sep 22 TechSpot on voxium's viral post: an anonymous engineer says Claude Code now writes the specs, tests, tickets and reports at their company, people work 12 to 13 hours a day approving output, and there's no time to review or understand the code. The complaint is aimed at management measuring PRs and features shipped.
Sep 22 Unreal Labs: agent harness schedules tool work between model turns (in-progress then final event-log records); Harbor benches show comparable pass rates at lower $ than Codex/Pi.
Sep 22 Takerngsaksiri, Duong and Barnett followed 6,774 merged agent PRs in popular GitHub repos into the fixes that came after them. Agent merges drew verified fixes at 1.62 times the odds of human merges in the same repos, about 70% of those fixes came from the same agent, and nothing visible at merge time reliably flagged which ones would need it.
Sep 22 Srikanth et al. (Weco): bi-level AIDE² rewrites its own research-agent harness under private grades — seven accepted improvements; matches/beats a 2-year human production agent on four held-out benches; reward hacking 55%→32%.
Sep 22 Pratik Patel: multi-day UI agent runs need stop conditions, blast radius, surviving logs, budgets, kill switches; overnight is a deliberate mode — reproduce→trace→fix→verify on the real path.
Sep 22 (updated Oct 1) Tania Papazafeiropoulou, one of two developers on a children's sports app: grew from two to as many as 15 parallel coding sessions, added a non-coding hub session that keeps a merge board and hands out the single test slot, and found the real limits were laptop memory, false-green test runs, and her colleague's review day.
Sep 22 Lee: disposable machines + coding agents on frozen failures; harness reruns the exact command, fences the network, and refuses toolchain cheats — 17/33 fixed; six harness bugs found by the fixer.
Sep 22 Li et al.: invoke makes prompts, tools, and REPL history first-class REPL variables—StuLife far-recall 69.9% vs Letta 61.8% at ~half cost; AppWorld TGC 74.2% vs ACE 69.9% with lower meta spend.
Sep 22 Li et al.: failure-guided Growing Harness turns a strategy-free scaffold into reusable executable control — 76–92% fewer LLM calls, held-out gate rollback, success holds across 4B–120B on WebArena.
Sep 22 lezli01: commit/merge KPIs break under agents; independent review needs unlike contexts; cap pending PRs as inventory; vincent gates left-of-merge, WIP-on-PRs still open.
Sep 22 Herlein: five agents, one frozen Qwen, held-out 32-case suite — done-gates and independent oracles beat bigger quant; self-written tests can false-pass the agent's own blind spot.
Sep 22 Axon Active: harness = instructions, context, tools, loop, guardrails, verification; independent review catches float-money false greens; harness work compounds across model swaps.
Sep 21 Xia et al. (Google Cloud AI Research): annealed edit budgets, leakage critic, noise-adjusted floor, and cost-aware pruning — up to +14.1 evolve / +4.7 OOD with 30% fewer policy tokens than unregularized RSI.
Sep 21 Boughamoura: beautiful markdown is not evidence — provider-isolated validator + workspace content-hash gate (excluding the report dir); wb-flow’s Infra/Artifact/Oracle contract.
Sep 21 Tayebati et al.: Detect→Notice→Heal→Validate admission control — of 383 replay-rejected proposals, 211 (55%) fixed the trigger while degrading protected cases; task-completion higher under Harness in all 16 pairs.
Sep 21 StratoNext sf: local factory — GHA-style YAML pipelines (agent + shell stages), per-request git worktrees, parallel queue, optional daemon; reuses installed Claude/Codex CLIs; human-review stages when needed.
Sep 21 Lencevicius: 591 Claude Code sessions; 69% of completion claims lacked a post-edit passing test. isitdone blocks Stop until the repo’s own checks pass the current tree, with receipts and test-tamper scans.
Sep 21 Park/KAIST et al.: reconfigure Sheridan supervisory control into Plan/Monitor/Wait/Review/Teach/Manual Fix/Update Assets from 19 developers; Reddit study shows concentrate planning, delegate review to other agents, externalize recurring guidance into assets.
Sep 21 Lundberg: correlated model blind spots ≠ two humans; trust from facts outside the diff (paths, history, tests moved, CI) — 58-PR queue ranked billing+migration risk over tidy size.
Sep 21 Ayyad et al.: outcome-group + centroid-in-embedding selection over sanitized trajectories — 5–10% subsets cut avg resolve-rate error 3–11% and worst-case 38–46% vs strongest random/stratified baselines; median error <5% at 10%.
Sep 20 Fan et al.: 111 SWE-rebench targets, 3,634 trajectories — frozen verified experience lifts 4/5 solvers +1.1–4.5 pp and cuts steps; 11/12 Mem0/SimpleMem/MemoryOS/A-MEM pairings stay at or below memory-off.
Sep 20 Ma: ceiling effects hide regressions once easy golden cases saturate; stratify easy/medium/stretch, detect when stretch is too green, and treat a yardstick that can still fail as the asset.
Sep 20 Gredeskoul: .plans emoji state you can prove from files, specialist agents that cannot publish, alock across worktrees, and a tea-break run that advances a dozen specs until a session limit — continuous factory with human merge.
Sep 20 Dottin: durable board + isolated profiles + Discord control room; concurrency caps before swarm theater; notify≠wake; review in another context — factory continuity without forty agents.
Sep 20 Zhou: split agent patches into Lane A (generated, report-only) and Lane B (human-owned oracles on sealed fixtures) — forbid oracle-path edits in the same diff, leave coverage in the summary.
Sep 19 rmax-ai: coding as constrained transitions over repo states—goal contracts, Docker sandbox, hidden tests, git checkpoints, SHA-256 evidence chain; model never makes the authoritative completion decision.
Sep 19 Chalupa: four-piece harness checklist; ACP mediation probe that caught an engine writing outside the project without asking; eval gates before any overnight autonomy.
Sep 19 AsyncDot: coordinator≠implementer, durable Plan Desk state, red-gate-first, re-run claimed commands — exit codes decide; reports are evidence, not instruction.
Sep 18 Tess Ainsley: measure reviewer-hours per merged production change, not per submission — Salesforce plateau + abandoned agent PRs hide the real verification tax.
SavvyCoding’s experimental factory: ~60s orchestrator cycle, sandboxed Claude personas, lint/drift/test/verify recipes before PR, reviewer + auto-merge, supervisors that break cascade loops.
Sep 18 Ma/Mikek/Li et al.: Benchproofer → SWE-Proof (500 Verified issues + formal oracles); audit overturns 26.8% of test-green patches; ground-truth formal spec lifts Opus 4.8 85%→~95%; self-written specs mostly fail faithfulness.
Sep 18 Kiro team: three full-time engineers merged 1,000 PRs in seven days, each through CI and review, by climbing five stages from one hand-driven session to a supervisor agent running 30 to 50 sessions across six to ten machines. Past 50 sessions the supervisor itself stopped scaling.
Sep 18 Ankit Jain: AI volume turns seniors into intent reverse-engineers; codify recurring feedback, preserve prompt/session acceptance criteria, measure verification work — not LOC throughput.
Sep 18 Firu: autonomous-sdlc-harness daemon → worktree → plan/build/review/browser-test loop; state in git checkboxes; four review gates including a skeptic; stops at a pushed branch, human owns the merge.
Faros's second AI engineering report, built on 12 months of telemetry from 22,000 developers across 4,000 teams, finds the worst early-adoption damage easing per change while the verification load moves downstream: unreviewed merges up 76.3%, time in QA up 300.6%, monthly incidents up 125.4%, and restarted work up 66.7%. Heavy agentic review correlates with better downstream numbers.
Sep 18 Codacy: advisory LLM reviewers vs rule/threshold gates; Copilot/Claude/Gemini have no native required check; fail-closed Diff Coverage and Verity prompt-conformance as the governance layer.
Sep 17 Tunguz on HarnessTax: same model 71% cheaper on Pi than Claude Code with no significant quality gap; harness compression as margin, moat, and the difference between rationing overnight work and saying yes.
Sep 17 Steve Brownlee: pull decision latitude out of the coding agent before the first token — assertable Given/When/Then, JSON phase contracts, convention profiles — so runtime review loops have less hallucination to catch.
Sep 17 NVLabs SoL-Pi: auto-research at the harness layer cuts token traffic ~45–49% and API cost ~⅓ vs native Codex/Claude Code while matching Pi on EdgeBench — four surviving mechanisms for unattended runs.
Kenn.io roborev: post-commit background reviews + agent hooks feed findings back while context is fresh; fix/refine loops in worktrees; compact verifies findings against current code before the PR flood.
Sep 17 Patodiya: differential tests on 18 fixtures across four pinned Python MCP integrations find 13 fixture–task divergences—structured slots, error flags, and rich content change after decode, by interface and consumer contract.
Sep 17 Prince Panchani on no_human: tamper guard, repro gate (fail at base / pass at head), fresh-context reviewer with edit tools refused, merge policy → awaiting_approval — confidence is a claim; evidence is re-runnable.
Sep 17 Nijkamp et al.: levels + clocked ticks + cascaded intelligence — a ten-day campaign with once-a-day human attention kept thread across every reset; continual learning belongs in harness checks.
Sep 17 Fan et al.: 176 matched settings on SWE-Bench Verified and Terminal-Bench — context management mostly prevents overflow; planning saves cost for strong models; bash-capable models often win cheaper without predefined tools.
Yu-kai Chou, creator of the Octalysis motivation framework, argues that supervising a swarm of agents burns people out because of the number of unfinished tasks one person holds open, not the number of agents. An approval queue gives the reviewer all pressure and no accomplishment. His fix: one main task carried to done per session, review with a visible finish, and agents that earn autonomy through clean streaks.
Sep 17 Augment/Hingel: unattended merge needs owned gates, escape-rate telemetry, provenance, incident owners, drilled halt — vendors still keep humans in the merge path; dark ≠ unowned.
Sep 17 Anthropic: projects as long-running factories — shared memory, per-thread branches, PRs/tests while you step away (even from your phone); merge conflicts stay ordinary PR work.
Sep 17 Air Security: zero-click RCE via SHA-pinning bypass in Claude Code, Codex, Copilot, Gemini — auto-update swaps marketplace plugins when git prefers a hash-named branch over the commit.
Sep 16 from Sinatra: nineteen weeks on two production repos, 306 agent-opened PRs, 289 merged (94%), 14 closed unmerged, 3 still open as of Sep 9. No auto-merge; every merge is a human button. Tickets written for agents, sandbox tests before ready-for-review, and agent self-review before a person looks.
Sep 16 Signadot: loop cost = iterations × cost-per-iteration; guardrails cap waste, only high-fidelity feedback reachable inside the session shrinks both; platform owns the verification surface.
Sep 16 Sara Mo: OpenAI’s factory can auto-approve low-risk changes — CI proves the checks; the classifier’s skip-human decision needs its own record, replay, and post-deploy score.
Sep 16 field receipt: 13h unattended fix of 12 design findings — 100/100 tests + byte-identical release via independent verifier; 95% of tokens were re-reads; idle-parent polling burned 6.5% on “not yet.”
Lin and Zhang (HKUST) model agent PRs as a queue with a mandatory human merge gate and an optional AI reviewer in front. Optimal review passes through four phases as maintainer capacity grows, including one where a useful reviewer is deliberately left idle because its false rejections cost more agent runs than its catches save. Calibrated on SWE-Review data, adaptive routing beats review-everything or review-nothing by up to 37%.
Peak Claude sessions are calendar-bounded. A harness that keeps modest verified momentum through the night compounds past glory bursts — but only if morning inventory is judged, not just generated.
Sep 16 HarnessTax (Pan et al., UC Berkeley/Arena): 21 model×harness pairs on SWE-bench Lite and Terminal-Bench — success moves a few points; cost up to 5×. Claude Code ~2× Pi on shared models.
Sep 16 Gwang-Jin Kim on Tan’s SpaceX/Cursor talk: ~1,000 PRs/month and merges while asleep — not a magic model; make verify, evidence-first bugs, Skills, CI-hardened review comments, fresh-agent reviewers.
Sep 16 Maxim Berenshtein: sixteen repo gates + edit-time hook + gate-inventory; prose in AGENTS.md fails silently — conventions become unignorable when the check refuses before CI.
Sep 16 Kapoor/Paca/Zemgulys: specs, constitution, control panel, context hub, and four gate types (automated / critic / human stage / regulatory) — maturity ladder for when humans still must say yes.
Sep 15 Zach Lloyd: Trigger→Agent patches hit context/metrics/security limits; walk = cloud default + shared context + triage→spec→implement→review→verify→monitor — Warp’s marketing site ~75% factory-shipped.
Sep 15 TechNode: Cognition’s Kevin Lee — more agent sessions are async than human-triggered; 89% of eng commits via Devin; judge by first-pass verification, not PR volume.
newsletter.pragmaticengineer.com
Sep 15 Pragmatic Engineer: inside OpenAI, humans define outcomes while Codex gathers context, implements, babysits CI, runs specialist reviews, deploys with per-change agents, and feeds Perf Factory — 10× CI load in six months.
Sep 15 Lugiax: Ghostty fork + MCP — supervisor tab reads screens, restarts quiet workers, group chat + task panel; won't answer permissions; overnight FOMO as unfinished runs.
Sep 15 Robert Glaser: encode judgment as backpressure and Outcome Grading; keep humans collaborating on intent; improve the factory while it builds — review FOMO is the wrong bottleneck.
Sep 15 BlueLupin census: code was ~8% of model output; review/verify ~40%; 15/39 Playwright runs were already-green reruns — cut avoidable proof without deleting the toll booth.
Sep 15 The Register (AWS OSS last week): Pizza Bot — local-first email-style inbox for background agents; Unread vs Action threads, LangGraph checkpoints, always-on host; FOMO UI for overnight work.
Sep 14 Wu et al.: contrastive trajectory RSI over Agent Loop / Tool Use / Observation / Context / Completion modules on 2,000 external tasks — lifts TerminalBench 2.0 Acc 47.57→52.43 and transfers across models; joint/non-modular evolution regresses.
Sam Armstrong on the software factory behind Vigil SOC: every PR ends with what it ran, what it skipped, and what the sandbox blocked; a separate environment agent repairs those gaps each day; and a single job cap, freed only by a human merge, keeps the factory from outrunning review.
Sep 14 Tejas Kumar walkthrough: six harness parts, deterministic verify on traces, secrets out of the model — GPT-3.5 Turbo upvotes HN only after harness engineering, not prompt heroics.
Sep 14 Agentic Batch Changes GA: one plain-language change scoped across the indexed fleet, CI-repair loops until PRs are reviewable — ~1k beta merges; Mercari ~80 repos; Canva 50+ migration PRs.
Sep 14 Specific Labs bench: Fable 5.1/Claude Code tops at 38.8% on proprietary multi-file tasks; six tasks <15% success; missed requirements and integration errors dominate.
Sep 14 OTF: Claude Code ≥2.1.269 runs WITH vs W/OUT plugin evals; CI --threshold on contribution lift so skill merges stop being demo folklore.
Sep 14 os-factory/har v1.14.2: agent-agnostic CLI+MCP — worktree slots, .har/ contract, deterministic verify + validated tree hash, Mission Control; Claude/Cursor/Codex portable.
Sep 14: same Mouse harness on GLM-5.3-Flash hits 23/30 ($0.29/pass, 97% cache) — within 1–2 of Claude Code/ZCode on full GLM at ~1/7–1/9 the $ per pass.
Sep 14 on Maintainability Gap (623M changes, 2023–2026): heavy AI users +25% vs own baseline; block duplication 40.3→73.0/MLOC; moved/refactor code 21%→3.8%.
Sep 14 industrial DSR study: specialized agents + context-specific knowledge across readability/maintainability/reliability/performance; >200 issues, ~69% rated important.
Sep 14 Bonomo build log: CLAUDE.md operating loop, sprint contracts, implementer≠evaluator, session-handoff.md — Session B finishes after /clear with zero naming drift from Session A.
Sep 14 Malhotra: 8× code, 10× tests, 25× CI jobs in six months; patches lasted 70d/29d/<1d before a journaled, horizontally scaled test-impact service — overnight agents raise the floor.
Sep 13 from Russell Winslow (Automater): six branch-protection gates for unattended PRs — tests with count floor, diff ceiling, secret scan, path rules, eval threshold with margin, human approval on the latest push — plus a morning queue that treats blocked PRs first.
Overmier skipped his autonomous coding pipeline's status dashboard and queried its task table directly. 555 starts in March and April fell to zero by September, approval gates cancelled more work than agents failed, and even the latest 'completed' task had only added an import.
Sep 12 Kraishan: 37k provenance-labeled PRs across five commercial agents — Codex reverts at half the human rate, Devin at 1.3×; pooled agents fewer security smells; Copilot draws the deepest human review.
Sep 12 Gabo Esquivel: an agent that edits and opens PRs is not a factory — stages need artifacts and decisions, long runs need handoff records, and acceptance stays stable while implementation iterates.
Sep 12 field report: a multi-repo agent takes a Jira story to stacked reviewed MRs — prose workflow contract, linear ticket chain, independent model review of tests then code, two human gates.
Sep 12 HydraFlow field note: standards become manufacturing only when violations turn into remediating work, the same gate judges the repair, and floors can tighten themselves.
Sep 11 Relux: across 3808 Codex rollouts, 52 goal sessions ate half of 59.2B input tokens; idle spin at 0.03s restart + poll-proof-of-wait; Claude Code awaits events instead.
Sep 11 Prokop Simek: agent PRs finish fast then wait — LinearB 5.3× pickup lag, Faros +91% review time; three attribution buckets and four saved views keep the queue moving.
Sep 11 field note: Abouwarda forks Claude Code review into five subagents that disagree on what is authoritative — ADR, spec, live HTTP, config — so the writer’s session never grades its own diff.
Sep 11 Firecrawl field guide collates Stripe/Spotify/Shopify/Ramp/Faire: intake, isolation, tools, verification, merge policy — plus Microsoft’s merge-rate gap and Faire’s failed confidence-score filter.
Sep 11 GitHub changelog: Copilot code review auto-resolves addressed threads, runs shell tools behind the agent firewall, and switches Lite to a multi-agent ensemble with higher addressed-comment rates.
Sep 11 Willison quote of Boris Cherny: Anthropic’s continuous verification stack — lint, tests, Claude e2e, daily fuzzers, automated review/security/refactor — is the price of agent labor at volume.
Sep 11 Akshay Utture, Augment Code: across 17,200 merged PRs from November 2025 to July 2026, size-adjusted output per developer rose 4.5× while median merge time fell from 11.2 to 3.1 hours and 14-day reverts fell from 1.9% to 0.4%. Review was automated first because its backlog was growing, and a human still makes every merge call.
Sep 10 Waydev: five of six links from capability to shipped value are harness work — verification is the new bottleneck, autonomy is a blast-radius grid, and flow metrics beat adoption dashboards.
Sep 10: VP-CONTROL shows cross-model votes over shared evidence still approve 62.9% of unsafe proposals vs 22.9% with an independent source — source diversity beats model diversity for agent commit gates.
Sep 10 field note: after 30 days of agent-primary production code, Tamiz argues the craft moved upstream — specify interfaces, manage context, verify fit — and treating agents as juniors you only review leaves architectural debt.
Sep 10 field report: design-first + adversarial plan review + parallel subagent diff review + browser gates — including an IDOR caught by another instance of the same model — before a human merge decision.
Sep 10: a referee-gated protocol (proof kernel, verifier, or withheld tests) walls the agent off, probe-tests isolation, freezes predictions, and treats budget stop as halt — so method effects on coding agents are measurable, not narrated.
Rod Rivera's paired experiment on 180 known-label implementations: telling a small-model reviewer that a prior reviewer passed the code nearly halved the bugs it caught, and a FAIL line raised false alarms from 4% to 46%. Independence is a property of the request, not the call.
Sep 10 Krentsel/Zaharia/Stoica et al.: requirement gap + model gap unify reward hacking and hallucination; an assurance-revision loop uses deployment evidence because predeployment evals stay proxies.
Sep 10 poornerd: describe plan→execute→review stages; Claude Code ships a worktree-isolated orchestrator plugin — and surfaces silent agent-teams traps before the first hang.
Sep 10 public beta: OpenAI hosts the Codex harness — durable sessions, compaction, subagents, sandboxes — so teams stop re-homing long-running agent infrastructure every model launch.
Sep 10 field note: Microsoft’s attributed PRAssistant result (5k repos, 90%+ coverage, 600K+ PRs/month, 10–20% median PR completion time) measures when review starts — not whether what merged got better.
LeadDev’s Sep 10 field guide: individual coding agents stall at 20–30% org gains; step-function lifts come from separated multi-agent loops—starting with PR→merge.
Sep 10 v2: 17 harness-evolution streams (802 tasks) — harness-induced forgetting, inconsistent self-evolving adaptation, and a retention↔adaptation tradeoff when tools/skills/agents keep changing.
Sep 10: harness evolution that separates model-specific accommodation from systematic harness repair via cross-task failure patterns — up to 1.84× faster training and +18.56% reasoning accuracy.
Sep 10 Alex Zhu: four named roles, a one-page session card, and a decision table — if nobody answers Slack, park or kill; never hope the agent finishes alone.
Sep 10 Decoding AI: Paul Iusztin ships Decode headless on Modal — CLI, webhook, or nightly cron; parallel attempts push decode/ branches so morning review starts from finished work, not five open terminals.
Sep 10 launch: cloud coordinator + shared context + subscriptions (Slack/PRs/schedule) so fleets keep shipping while you sleep — FOMO as product, review still the human gate.
Sep 10 CodeRabbit field note: green harness checks prove covered behavior; merge still asks whether the change should become part of the system — with review depth following cost of error and blast radius.
Sep 10 Cedric Teyton survey of Stripe/DoorDash/Ramp/Monzo/Checkout/etc.: durable runs + ephemeral workers, determinism inside the loop, fresh-context review gates — attention scarcity, not typing speed.
Sep 10 BlueLupin: ruleset, map, broker, human-on-diff before sessions; intake→verify→named reviewer; launching more agents fills the review queue — capacity is who drains it.
Sep 10 paper: lifecycle-model instrumentation for agent evals; full-chain recall 77–100% vs 23–94% hackability scanner; 96% runtime accuracy on labeled trajectories.
CapScope’s lesson isn’t “spot the injection.” It’s freeze what an agent may do outside the model, then put an owner on that inventory. Fleet management of ceilings across a software factory is the natural next layer — and it’s exactly the kind of platform-dependency objective stewards are good at enforcing.
Atlassian’s Sep 10 ship note: always-on Jira agent loops plus org Standards and a dedicated AI Review agent — humans keep the merge button, not the bottleneck before it.
Sep 10 TNS field note: functionally correct agent PRs that cross domain boundaries slip past human review — Executable Architecture (pytest-archon / ArchUnit) makes boundary violations fail CI instead.
Sep 9 XAgent: differential analysis of failing vs passing executions for localization, plus context-aware test augmentation so validation doesn’t overfit the ticket — 62% resolve / 72.8% function loc on SWE-bench-lite.
Sep 9 TrajMark: ownership + segment-level tamper localization for coding-agent trajectories — because patch watermarks miss process provenance and local edits can keep a global ownership signal.
Sep 9 field note: when agents flip the write/read ratio, the real review target becomes permissions, tools, and who sits between output and production — not every generated line.
Sep 9 paper: of 654 Bandit–Semgrep-clean Python samples, dynamic exploit verification confirmed ~1 in 7; several frequent CWEs flagged by neither scanner.
Sep 9 Part 2: scaling agentic coding across ~15k engineers — nine-stage maturity language for managers, token discipline as quality, and a model-independent harness so routing/skills/context survive provider churn.
Qodo’s Agentic Toolbox (Sep 9) packages context, standards, and an independent review pass as skills coding agents can call before a PR exists — and again to resolve structured findings after.
Phil Booth’s Opair makes shared ownership the product goal — no shell, scoped tools, driver/navigator role swap — so the human stays in the loop before the 5k-line agent PR arrives for review.
Sep 9 Mouse: on Runta’s FrontierHarness (Kimi K3 held fixed), OpenCode scored 15/30; Mouse’s deterministic post-turn verify loop hit 25/30 — checks, deleted-test block, MOUSE_AUDIT evidence, and cache-friendly retries.
Sep 9 mikeroySoft: label-driven always-on factory (triage→worktree→CI gate→evidence-bound review→merge); self-declared done is unreliable; 14 self-merged tickets; events.jsonl accountability, not autonomy theater.
Sep 9 LeadDev field note: authorship, memory, and comprehension fail under agent volume; gate on detection and blast-radius containment, not per-line human recall.
Sep 9 arXiv note: iterative LLM bug-fixers invent bugs in correct programs faster than they heal real ones, then oscillate forever — a stopping-condition problem for review-time APR.
Sep 9 benchmark: 660 evidence-grounded gaps in research-method specs — defect localization is the bottleneck; gold clarification lifts codification-ready rate from 14% to 98%.
Harness CLI 3.0 (Sep 9) puts risk scores, review groups, and check→pipeline log streaming in the terminal so agents can triage and close the review loop without the 12-tab dance.
Harness field CTO on Sep 9: PR floods at 1.5–10× (sometimes claimed 50×), reviewers who should see the real change not the dependency scaffolding, and a delivery knowledge graph so review context is cheap.
Google’s Sep 9 harness note: end-to-end benches won’t tell you why the score moved — unit-style behavioral assertions on tool use and intermediate steps let you iterate prompts and models without silent regressions.
A six-person team’s field numbers after agents arrived — PRs doubled, median time-to-merge went 4h→14h — and the social fixes that worked better than an AI review bot.
Hartman’s Consort (Sep 9) sorts agent frameworks by persuasion vs structure vs hard controls, then argues for immutable tests and gates on a live DB branch the worker cannot bypass.
Field note from Black Box via Codacy — ~8 coding agents in the loop, every PR through five independent reviewers, plus Verity’s markdown knowledge graph so later turns inherit what got rejected.
Sep 9 Marron & Barr: treat incomplete code as a first-class hole, synthesize from traces and typed IO pairs inside a harness+agent assembly — JIT for software construction.
Observer → tech lead → architect → engineering manager (software factory) → product manager — pick deliberately by trust, blast radius, and how far you are from the code.
Sep 8 paper: anti-hacking + task refinement cut inflated SE-agent scores (GLM-5.2 78.80%→57.32%); gold-patch leakage via git, FS, network, and metadata.
newsletter.pragmaticengineer.com
Gergely Orosz surveys how labs and startups actually handle the review flood — from reviewing the AI review, to blast-radius triage, to reviewing plans and schemas instead of diffs.
Sep 8 Pini Shvartsman series finale: agentic factory as nine human control surfaces (readiness → halt); five failure modes where metrics improve while judgment dies; F0–F4 maturity without buying a platform.
Sam Bhagwat on Mastra Factory, which now writes 25-35% of Mastra's PRs and closes 50-60% of its issues: full-auto mode flooded review when backlogs were imported, wrong upstream assumptions produced PRs that had to be closed, and the project now asks the community for detailed repros instead of small PRs.
Sep 8 Allen Hutchison: scheduled auto-dev ticks triage→plan→build→address review; harness forbids merge — morning work is decide-to-ship, not reverse-engineer intent.
Sep 8 Arjmandi: paired same-model harness contrasts on a private 256-task suite — averages near zero (−1.25 / +1.25 pp); Opus strata reverse by task type; 22/81 wall-clock cancels already had a passing patch.
ExecCritic separates Test and Repair agents, freezes fail-closed tests, and shows test quality decides whether execution feedback helps — composing trained roles hits 72.6% on SWE-bench Verified.
Sep 8 Brian Curry: six-layer fleet architecture; review-queue bifurcation; sustainable agents scale as the reciprocal of the human-routed fraction — gates, not headcount, buy overnight capacity.
Sep 8: cross-substrate authority gap — identical files can be safe or unsafe; planner observation fails, a deterministic execution guard at the mutation boundary catches unsafe publish intents.
CapScope freezes a task-wide authority ceiling and per-agent typed capabilities outside the model — injected effects drop from ~33–47/75 to 3/75 without tanking repair success.
Sep 7 Farchi case study: natural-math specs + invariant proofs + runtime coverage; code is regenerable output the factory need not read line by line.
Sep 7: serverless coding harness on AgentCore microVMs — handoff/task modes, S3 checkpoints, Telegram approve; ~$3.45 compute over 12 days vs always-on EC2.
Across 3,171 public repos, 16% of multi-component agent setups carry a confirmed security defect — unpinned MCP servers, scoped-looking shell grants, skills that pre-approve Bash for whoever installs them.
Sep 7 survey of 281 OSS AI policies: most allow AI code, but demand human involvement, accountability, and increasingly disclosure — plus ten countermeasures against AI slop at the PR gate.
Sep 7 Camargo research note: authorship ≠ responsibility after AI-generated defects; acceptance evidence should start from the demand, not the diff.
Sep 6 Hodalog: Claude Code Routines grew 2→8; 119 Issues, 78% adopted, 95% PR merge — plus a weekly PR-care bot; adoption decision and Issue readability become the bottleneck.
Yu et al. track 400 repeat reviewers of agent PRs: approvals climb while hand-crafted comment features stay flat — sentence-embedding centroids move, and language shifts lag the approval change.
Sep 5 field build: cron+claude -p, hooks to phone, CLAUDE.md/auto-memory/resume, tmux+SSH, worktrees — from ‘Ferrari sitting inert’ FOMO to morning PR/CI already done.
Wang and Liu’s Sep 4 theory note: Trustworthy Change as the engineering object, Human-Agent Cells that produce evidence without acceptance rights, and Responsibility Topology for who can absorb residual risk.
Sep 4 Rootly field note: Flare, Airbyte, CircleCI, Moloco, Twingate leaders on agent PR volume — spec-first review, risk tiers that skip low, earlier machine verification, and a named human who owns the merge.
Paul Iusztin maps factory stages onto the SDLC, puts humans on brainstorm/plan/final check, and warns that first-version autonomy theater often fails.
Sep 4 Frisinger at 8th Light: verification is the bottleneck; fail-closed gates, evidence ledgers, and provenance turn overnight agent output into accountable inventory — not plausible debt.
Lessons from an autonomous GitHub team: same-model author/reviewer blindness, dry-run evidence that lied, roles-as-data, structured envelopes, and refuse-to-guess gates.
Bhati’s Sep 2026 synthesis names the attenuation curve from agent commits to releases, and reframes output as PQC — changes that cleared review, test, security, and ops gates.
He et al. build a 303-instance benchmark where review constraints from real PR comments are tested separately from functional tests — and 221 of 644 functionally green repairs still fail the review bar.
Sep 3 RuntimeWire: OpenAI desktop Codex attaches a recurring heartbeat to a PR — re-reads live GitHub state, repairs CI and conflicts in a worktree, optionally merges; continuous ownership instead of one-shot heroics.
Sep 3 case study (N=21): a year into AI adoption, engineers report accountability anxiety, craft-identity disruption, meaning erosion, and workload intensification — including around review.
Guild’s Sep 3 Software Factory receipts — 34% of merged PRs, separate review agent before humans, maintenance agents that keep the loop alive, ~$12 model spend per merged PR.
Atlassian research: 74% see faster generation but 78% still use traditional peer review; leaders want a governed system of work agents can write back into.
Sep 3, Ilpo Niva at A-CX: unattended parallel agents closed 204 issues and merged 152 PRs in a week, but only against issues with an objective, design, checkable criteria, and dependency order. Worktrees, an atomic claim registry, and a fail-closed merge checkpoint keep it safe; about one in twenty machine findings is rejected on review.
Eleven failure modes from production self-improving loops, and PROCTOR’s Teacher-Student design with hermetic sandboxes and frozen holdouts.
Thoughtworks CTO Rachel Laycock answers Houck — same Meta/DX crush numbers, different fix: shift judgment left via pairing and design, review by exception, don’t preserve PR ceremony with an AI pretend-reviewer.
Sep 2 Maya Shenoi methodology note: HELIX rebuilds the SDLC around verification contracts — R0–R3 autonomy, machine gate then adversarial AI then human intent review, and no merge without a verification record.
Sep 2 Endor Labs field note: agent PRs break the one-reviewer-per-change ratio; put security context at generation, architecture-aware review on every PR, and reachability before remediation.
A practitioner template that forces intent, surface area, risk, evidence, edge cases, rollback, and review focus onto every agent-written pull request.
Sep 2026 Brett Ridenour: draft PRs only, P2/P3-only autofix, never push main, halt on broken baseline or three consecutive fails, hard wall-clock cap — scar tissue as the safety model.
Sep 1 Jiang/Chu/Tian et al. (Huawei): reference-guided root-cause vs failed runs; quality gate (leak/bloat) + performance gate (batch↑, recent↘ε); CloudCoreNetwork-QA 43.4%→86.9% (+21.6pp over GEPA).
Sep 1 benchmark: can models create and evolve their own agent harness? Generated code/search harnesses lag human ones; evolution gains are unstable and model-tied.
Sep 1 HoH wraps existing coding harnesses in planning–coding–testing iterations that force new capability each loop, schema-constrained artifacts, and an independent tester — +52% avg relative vs standalone harnesses.
Sep 1 Faros field note: software factories coordinate agents, harnesses, and verification — and the 2026 Engineering Report shows AI adoption lifting throughput while PR review time jumped 441.5%.
GitHub’s Sep 1 changelog adds approval assessments on every Copilot review, plus an off-by-default path for Copilot approvals to count toward required reviews, scoped by path.
Sep 1 field report: 92M msg/day Node→Go with Claude Code overnight — golden-file black-box harness defined correctness; agents shipped ~13k lines; zero incidents after a queue-topology miss taught boundary fidelity.
Sep 1 Allstacks / CircleCI field note: 28M workflows, main success 70.8%, merge efficiency 3.9→1.3; put checks in the agent loop and adversarial review in the spec.
Aug 31 Pini Shvartsman: continuous integration → continuous verification — comprehend, choose risk-based strategy, investigate failures (no blind retry), emit an evidence package; escalate up, never auto-approve consequential deploys.
Aug 31 Ke/Wu/Li: agent proposes specs/lemmas; checker owns authority via SMT discharge + replay; 1367/1475 passes; local lemmas ablation −221 tasks; SV-COMP ReachSafety 91.5% vs CPAchecker 60.1%.
Augment’s org-design guide: agents absorb implementation, so scarce human work is spec, verification, halt authority, and line ownership — with receipts on review share collapsing under PR flood.
Signadot/TNS on GitHub’s 2.9B monthly commits: generation is machine-paced while runtime verification still queues through staging—and how per-change isolation closes the gap.
Controlled study of 1,116 agent-built web apps: boot probes, shells, and screenshots help only for the failure modes they can observe.
Aug 28 openJiuwen team: Rail-composed Inner/Outer loops + runtime adaptivity — SWE-bench Verified 82.6% (+3.4 pp) and Terminal-Bench 2.1 87.19% (+3.39 pp) over strongest selected leaderboard points.
Aug 28 field note: unattended coding needs verify-outside-worker, admissible evidence (not narration), budget reservation, resumable controller, and stop-at-merge — plus inject/copy/grade playbooks.
Aug 27 Xu et al.: budget-aware harness evolution with behavior-aware verification + attributable-evidence gate — 7.6–13.6% held-out gains at ≤200 interaction units vs thousands of fixed-set rollouts.
Aug 27 Xiao et al.: supervisor–worker harness redirects active workers and distils skills/memory live; +14.6/+12.4 pp Terminal-Bench self-improvement, tokens down ~43–47%.
Turns scattered optimization history into persistent skill knowledge so agents can reuse what they learned across tasks.
More than 70% of Uber PRs are attributed to agents. Their new post is a field manual for measuring and cutting agent spend without starving quality, including how they bench uReview for code review.
2,269 real multi-round review tasks with defect lifecycle labels—LLMs degrade as rounds grow and miss low-salience defects.
Aug 27 Harness: Code Repository scaled for agent PR volume + AI Code Review with risk-grouped diffs and required AI Checks — scoped agent identities so continuous factories don’t overwhelm human review.
Aug 27 Travis Frisinger: six months, ~1,700 merges — throughput is the honeymoon; fail-closed gates, calibrated seats, institutional memory, and scarce Purpose form PAAA.
Aug 27 Woodhead/Foremerge: parallel agents in worktrees can merge clean and still break design — declare semantic scopes before edits; acceptance gated on harness-run checks, not agent claims.
Aug 27 Ralph Duin, Inspired by Frustration: a lead agent that never edited a file ran 37 subagents across seven worktrees, finished 15 of 16 tasks for about $49, and merged nothing to main. The two bugs worth reporting got past 1,374 passing tests and were found by fresh-context reviewers who opened the rendered PDF and reproduced a stale cache.
Aug 26 WonderLab / MyCodeAgent: single immutable loop + TransitionReason traces; completion gate PASS/UNVERIFIED/FAIL with feedback injection; terminal reasons exhaustively named.
Warp’s self-improvement loop: scorers for efficiency/quality/compliance sample runs, then a scheduled agent proposes skill/config diffs from failures.
Aug 26 Sumant Thakur: Spotify Xirp, AWS Kiro Crew, Warp Factories — durable state, isolation, event starts, and verification outside any one agent session.
Aug 26 Stolze & Strässle (ESEM 2026): five-practitioner study — preventive + executable + human layers; review 3–4× generation; promote recurring findings into lint.
Aug 26 paper: MalPR-Bench scores attributable PR security review; CodeRabbit blocks almost as often as PRGuard but identifies fewer target vulnerabilities — especially when evidence sits outside the touched files.
Aug 26 Lewis (Yuj): paired control vs closed-loop harness on fixed weights — tight 20k Verified F2PF 28%→49%, solutions 43→72; frozen treatment transfers across four model designs; wide window mostly erases the gap.
Aug 26 DoneGate: worktree + path leases + independent validation — commit claim must match HEAD, allowlist/denylist, required checks, and execution evidence before completion.
Cline walks through a two-pass PR reviewer on their SDK—record findings, then a stricter judge—plus deterministic tool guards so the reviewer stays read-only.
Aug 25 Vercel guide: eve software factory moves labeled work through triage → plan → implement → independent review on a different model vendor, with bounded request_changes loops and evidence against acceptance criteria.
Scott Spence’s practical definition: agents write; engineers own context, scope, validation, evidence, and review—with harness numbers from real projects.
Aug 25, Maurice Kleine. He used to be the message bus between 4 to 10 agent tabs and felt stressed whenever his laptop lid was closed. He moved the fleet to a €100/month server with supervisor agents that route tickets around the clock. 468 PRs merged and 537 tickets completed in 23 days. His first AI review setup nearly froze the fleet, so review cycles are now capped at two before a ticket blocks and comes to him.
Aug 25 Esakkiraja et al. (ServiceNow/Mila): evolve prompts/tools/skills/MCP/subagents with failure-mode strata + hidden selection; +20–35 pp on ITBench / EnterpriseOps-Gym / AutomationBench, frozen transfer across GPT and Qwen.
Aug 24 Genovese/Regolo: OpenCode in GitHub Actions with a bash verification gate — pytest, forbidden paths, secret scan — PR opens only if success(); prompt alone is not enough.
Aug 23 Quiroga TaskKeeper: scheduled Claude/Codex runs in a fresh git worktree; VS Code closed; local review inbox with cost + diff; accept merges locally, discard deletes the worktree — never auto-push.
Augment’s metrics guide treats the agent fleet plus human reviewers as one production unit — Faros/LinearB receipts show PR volume up while review waits explode, and DORA’s verification tax names the spend.
Aug 22 Lulla et al.: gray-lit definition + mine of 36,710 repos; 217 confirmed autonomous loops (mostly event PR-review/cron triage); configs committed, runtime state almost never versioned.
Dan McAteer on Latent Space traces harness eras from AutoGPT to Claude Code, then argues what’s left after models absorb scaffolding is an interface to human attention.
Aug 21 Augment field note: agent PR volume turns merge-only review into terminal inspection — station-level quality gates (intake→spec→build→verify→ship) with named owners stop defects where generation is still cheap.
Selvanayagam and Ghaleb link 248k agent-authored PRs that got at least one AI-attributed review on CodAGE — cross-product review is ~1.6% but grew two orders of magnitude from 2025-Q1 to Q3.
Osmani on lights-on factories: encode taste in the environment, budget verification like performance, and keep ownership on what ships.
Aug 20 Watson Labs / Swamp: Gymwasp’s Mandible factory — 372 issues, 1,801 review rounds, 11,967 verdicts. 65% mergeable after one round; after round 4 the loop oscillates; 29% shipped on warn; volume 3× with median rounds flat.
Tessl’s Aug 20 Code Review launch: team standards live as versioned skills you can inspect, eval, and point at another repo — whole-PR context, stateful re-review, and a path toward an autonomous merge gate.
Aug 20 field note: verification debt as the queue behind cheap AI implementation; independent layers beat same-model self-checks; measure verified delivery throughput.
Aug 20 Bhardwaj et al.: worker + independent gate + declared budget; prove termination under repair and ledger-supported done; instrument finds 47 vacuous gates in shipped loops.
Aug 19 arXiv 2608.18565: verification-gated PLC harness — spec check, real toolchain compile, live execution. Project-context tasks: 89.4% compile vs 52.2 dynamic behaviour — the gap is the argument.
Aug 19 Chen/Shi (Paritok): instrumented Claude Code/Codex sessions — tool filtering saves ~21–57K tokens/turn (linear); content compression ~2%/turn but compounds ~3,350 N²; single-shot SWE-bench quality ≠ multi-turn cost.
Aug 19 Addy Osmani, advice file in his reference software factory for Claude Code and Codex: a plain harness plus a clear issue gets surprisingly far, the factory layer earns its place through back-pressure and independent verification, and ten features with an independent verifier cost at least 160 minutes of suite time before any human looks.
Google Mandiant details an agentic vulnerability discovery harness—threat model, entry points, hypothesis generation, multi-temperature validation, then human PoC.
Aug 18, Julien Danjou (who runs a merge-queue company) grants that one-commit-per-green-build can't survive agent commit rates, but argues that landing 150 commits on a red main throws away the link from each failure back to the agent and intent that caused it.
A small studio on what changed when agents started running for hours unattended: every mid-task judgement moves into the brief, and the morning review queue, not launch capacity, sets how many runs are worth starting.
Yohei Nakamura audited his attended overnight coding-agent loop against its records: 72 tickets, 56 PRs, 54 merged, 0 reverted. Ticket defects outnumbered code failures 4.5 to 1, 18 of 38 constraints were learned only by failing, and verification needed a layer that distrusts the evidence itself.
Goat Squad Studios: plan by day, agent loops by night, review in the morning. Every ticket carries an autonomy field, a missing field means it waits, and only work a test can prove is allowed to reach done without a human. Production still moves through one PR a day that the human reads.
Aug 16 yureki_lab: Playwright capture harness + report.json (overflow, console errors, failed requests) + stale-hash gate — multimodal screenshots close the UI loop agents already have for pytest.
Cristiano Mazzella on his own overnight harness: a read-only verifier that records the command it ran, three deliberate ways for a run to stop, and why a rigorous, green, reviewable workaround can still be the wrong change.
A new paper finds naive multi-agent review collapses into false consensus, and that a smaller setup with an explicit critic does better.
ZenBusiness (Witt, Aug 14): nightly Dependabot Janitor — deterministic semver/scope filter, then Claude Managed Agent skills; ~250 auto-merges / 800+ parked with receipts across ~30 repos in seven weeks.
Aug 2026 Paul Kuo: Cherny’s 388→180 maintenance PRs vs 150 days of launchd/Cowork jobs — silent failure when green logs lie; verification must sit outside the executor.
Aug 14 coleam00: Archon cron factory — dumb label dispatcher, independent validate-PR holdout, governance files auto-reject; humans file issues and promote releases only.
Aug 13 Qodo field note: independent review re-enters judgment under agent volume, but lifecycle authority still decides whether the task may advance after commits, CI, repairs, and stale approvals.
Aug 13 Claude Code #86278: ~12h autonomous session merged ~24 PRs with green required checks; user hit ≥9 live regressions the same evening — green CI ≠ production-qualified.
Aug 13 Augment: generation is cheap; review/verify/repair/decision are the drag. Cosmos Experts (Risk, Deep, Pair, Verifier, Fixer) keep working until a human can merge with evidence — loop engineering, not a better comment bot.
Aug 13 Winston/Voss (Arize): long-running agents fail where chatbots never go — separate eval vs guardrail vs harness, give judges the trajectory, and feed failure explanations back as the next engineering brief.
Aug 12 Grammel/Dodds: issue→classify→analyze→implement→review→backport in sandboxes; 70–80% of closed issues via factory; reviewer efficiency is the product, not more agents.
Aug 12 Denis Morozov: 37 packets, 37 merges, $3.89 — planted tests, named invariants, harness-run gates; the implementer’s ‘verification’ block is recorded and ignored.
Aug 11 Conveo: /nightwatch skill — Helm plans, remote Field Orchestrator spawns worktree workers, cheap per-ticket verify loops, one green master PR after 7.5h on 3 agents.
Aug 11 Ascii Box template: continuous factory Box queues GitHub/Linear/Sentry issues; build Box plans/commits/tests; fresh verification Box gets the branch as a Git bundle — humans merge, never the factory.
Aug 10 Wang/Li/Li/Li: externalize obs/plan/memory/act/verify/recover/stop/budget as Loop Policy Bundles; online+offline evolution with paired eval, robust gate, task-boundary activation, rollback—YC-Bench lifts returns and survival vs fixed π₀.
Aug 10 steezr (Johnny Unar): an inherited client repo had 300+ open agent PRs and two exhausted senior reviewers. Path-based risk tiers took about 60% of agent PRs off human review; commit trailers recording provenance showed a 1.8% revert rate on auto-merged low-risk PRs against a 2.3% human baseline, and 11% on high-risk surfaces.
Rule-guided dispatch, grounded file tools, and an asymmetric reflector that filters hallucinations—higher SEM-F1 at far lower token cost on AACR-Bench.
Aug 10 Jeff Kazzee: stage-by-stage SDLC switches; verification buys autonomy; deterministic checks + independent judgment + real surface; a pilot of seven image calls with preservation and browser gates.
Open-source Open Code Review tooling from Alibaba’s internal assistant lineage, with Claude/Codex/Cursor plugins.
Aug 9 Li et al. (Concordia): training-free bidirectional reconstruct-and-verify after agent patches; +7.0 / +3.6 Pass@1 on SWE-bench Verified vs mini-SWE-agent; Self-Refine with similar budget loses.
Aug 8 Razzhigaev et al.: self-developing coding harness with free + experience-driven core evolution under a blocking diff gate; Opus 5 Terminal-Bench 2.1 86.74% audited; Hope—161 days, 1,085 self-mod commits, 63.5% recent review block rate.
Aug 7 Assembled: agent PRs left humans behind (TTF review 3.5h→16h); dual-model auto-approver + policy ridge hit 2.4× merges, 3.5× large-PR throughput, 43% auto-approvals — without raising reverts.
Aug 7 Anthropic: Nuro 10pm–5am hill-climb → 3 PRs; classifier vs permission fatigue; FOMO answered only when eval signal + deny rules + PR-as-boundary exist.
DVx Field Manual 002 splits unattended agent work into three shifts (human decides, agent executes, human accepts), with green/amber/red autonomy and a morning review packet read tests-first.
DX’s Brian Houck on Meta’s review crush under agents — RADAR’s risk-stratified automation works, but automating diffs without protecting knowledge transfer and intent debt just hollows the team.
Aug 4 Pipeline Mag: four 2026 merge-rate studies disagree; rejections are mostly silence; Faros shows review time +441% and churn +861% under heavy AI — throughput ≠ what survived.
Aug 4 field guide (updated Sep 14): task queue → worktree-isolated Codex exec/Goal Mode → watchdog → morning triage — atomic verifiable tasks, AGENTS.md constitution, and when not to go lights-off.
Aug 4 Hsu/Chi/Everett: sub-path anchors + freshness≠evidence; after upstream drift Haiku ANCHOR F1 median 1.0 vs PROSE 0.27; refuse when support withheld—never guess.
Aug 4 HILLS Lab field note on how one engineer runs AI-assisted delivery overnight at Tilt: a written definition of done before the run, a fresh agent session per ticket, isolated worktrees, explicit stop conditions, and a morning manifest of checks, blockers and the decisions still needed from a person. Merging and deploying are separate outcomes that need their own authorisation.
Cloudflare/Astro cut open issues from 200+ toward zero with isolated subagents that reproduce, diagnose, and ship preview fixes inside GitHub Actions.
Aug 4 Cloudflare: replace human-paced SDLC with Agent Development Lifecycle — programmatic, preview-per-agent, event-driven Workflows that spawn heal/review agents; @cloudflare/ci self-heals and stages fixes for merge.
Aug 3 Zhang et al.: constrained + proactive-feedback + component-wise harness evolution — SWE-bench Verified 54%→66% Pass@1 in 5 turns; held-out 60.4% vs AHE 54.7%; frozen harness lifts Claude Sonnet 4.6.
Aug 2026, Steve Yegge: running 40-plus agents around the clock against a half-hour build, his bisecting merge queue grew past 100 requests and stalled. His fix is to land 120 to 150 commits on main at once and have agents repair whatever goes red.
TrueFoundry’s field guide: factories aren’t new; agent-native ones need model supply, credentials, QC, telemetry, and spend metering—not just more agents.
Aug 3 Tony Alaniz (OpenAI Cookbook): phased harness/build files, human gates, and an append-only build-log that records observed verification — so long-running agent work cannot mark a phase complete on narration alone.
Aug 3 Ma et al.: Manage–Execute–Audit keeps task state outside the growing session; fresh-context executors, read-only auditors, and Claude Code/Codex backends — WeaveBench 51.8%→80.7% with the same model.
Aug 2 Nikolaev confirmatory: static Claim Plane lifts pair pass 23.3%→50% and integration 65.6%→96.7% on 30 CooperBench pairs—but serializes 96.7% of runs; dynamic scope fails closed on undeclared regions.
Aug 1 Sudarshan Shenvi: testable requirements, a separate critic that refuses partial passes, always-on sandbox with permissions off — review the output, not the babysitting.
Jul 31 Li et al. (Microsoft): 112 dependency-DAG tasks, 5,300+ units, flow-aware test release + live regressions; Opus-4.7 + Claude Code + outer continuation tops out at 25% resolve.
Jul 30 Xu et al. (Concordia): evidence-conditioned execution layer gates premature commitment on SWE-bench Verified; +4.8–11.8 Pass@1 across two models and two scaffolds while cutting tokens up to 12.1%.
Jul 30 Swail Tycho: completion-hook ledger that checks files, commands, exit codes, and claims against git/FS — no LLM in the trust path; VERIFIED/FAILED/STALE for overnight review.
Jul 30 funador: FIFO land + checkCommand for multi-worktree Claude Code; promote is human-only; slow gates cap landings/hour; checkCommand≠human review (on purpose).
Jul 29 Toolshed Labs Nightshift: four bash habits — rewrite handoff, fail-closed verify-gate, runaway caps, morning brief — so unattended agents stop producing confident unverified inventory.
AppSec’s Dispatch pipeline — partitioned hunters, cross-model verifiers with real integration-test oracles, then fix PRs — so better models don’t just dump confident noise on developers.
Jul 2026 noktohq/nokto-agent-orchestrator (still shipping Sep 21): Claude/Codex/Gemini delivery loop with allowlisted verify, secondary review, JSONL audit, resume; human merge is hard — PRs carry the receipt, not a silent land.
Jul 28 LinearB 2.7M-PR benchmarks: elite orgs merge agent PRs at 79% vs 92% human-only (37% at top 60%); ownership — not model skill — is the limiter; AI code review lifts PR yield up to +5–7pts while generation moves the bottleneck downstream.
Jul 28 Fu et al.: causal CIEL+ARCO orchestration — GAIA 71.3% at 28.4% fewer tokens; SWE-Verified 65.6% at 13.1%↓ tokens / 25.9%↓ time; Always-Full ablation collapses GAIA to 27.7%.
Jul 27 Lukas Grigis ralphctl 0.17: generator–evaluator ralph harness; evaluator must audit diffs for verification tampering and cite hunks — FAIL and PASS both need evidence.
Jul 27 John Young: inventory vendor defaults for permissions, hooks, sandbox, telemetry — exit 1 ≠ block; Cursor failClosed defaults false; sell blast-radius control, not quality.
Jul 27 MalyStern: local Rust supervisor for Claude/Codex/ACP — block forbidden tools before they run, gate done on real tests, undoable checkpoints, budget kill-switch, VERIFIED/RECORDED/ESTIMATED report lines.
Jul 25 lmorchard: unattended issue→PR harness — acceptance criteria frozen before work; separate read-only grader; auto-ok vs needs-review tiers; eligible-for-auto-merge is a finding, never a machine merge.
July field note: Elstner’s four-rung promotion architecture for AI code review — shadow → advisory → soft-gate → hard-gate — on trailing precision, availability, latency, and false-block rate, with silence treated as failure.
Jul 22: AIDev shows 46.41% agent fix-PR rejection; instructions alone don’t lift merge rate — put rules/tests/human gates inside the harness before more tools.
Jul 22, Andrew Thompson (CTO, Orbital Witness). Scored against Ryan Lopopolo's twelve harness-engineering theses, Orbital's harness roughly doubled weekly merged PRs per contributor on a legacy repo, to mostly 6 to 7, at the same review and CI bar. A nightly bug hunt has sceptic agents try to refute each finding before a fix ships for morning review. Three theses are only half-built: turning feedback into infrastructure, retiring nightly loops, and measuring attention rather than cost.
Jul 22 Nightgauge (open-core): local issue→PR pipeline with isolated stage handoffs, build-as-hard-gate validation, batch overnight runs, cascade circuit breaker — nothing ships unmeasured.
Jul 22 Magrathea field note: labeled-issue dispatcher, per-issue worktrees, test/build/cdk gates, and a hard no-push boundary—defer colliding work, report empty queues honestly, leave closeable bugs for humans.
Controlled pairings of Claude and Codex as writer/reviewer show one direction lifts pass rates; the reverse can hurt.
Jul 22 Anthropic (Delba de Oliveira): turn the manual checks you keep repeating into Claude Code skills — standalone, embedded, chained, or on every PR — so modest continuous verify runs without relying on memory.
Jul 22 Addy Osmani: loop → harness → factory; dark ships unread diffs; lit keeps judgment on the outer loop — autonomy only as wide as cheap, hard-to-fake checks.
Ker102 Harneloop (v0.0.2): self-evolving harness units — capture artifacts/traces, diagnose, candidate patches, promote only with evidence; ViperMesh case study: 2.53× speedup, +8.19 visual, −90.9% tokens vs Blender MCP baseline.
Deputy CISO Jason Clinton on securing an SDLC where Claude authors ~80% of merged code — multi-agent PR review, risk-tiered automation, shadow mode, and identity boundaries instead of prompt trust.
July 2026 developer blog: repository rules in AGENTS.md that Codex Code Review can cite on findings.
Jul 20 Solar/Favur: six-session 2048 SOW run — false completion, oscillation, context exhaustion, review cost; validators cite docs/hashes; play-the-game signal beats green suite; task drift still open.
Jul 20 Verdi/Gilhuly (Arize Observe): video demos as trust units; ~30–40% ecosystem PRs merge without human review; risk-scored auto-merge; Bugbot learns from corrections; self-driving PRs.
Jul 19, Xinwei Xiong. Ten parallel overnight tasks produced ten convincing PRs; by lunch he had reviewed three, the eighth he just merged, and that night he queued ten more. A week of about three times his usual merged code shipped no extra features. His fix is raising first-pass correctness, not adding agents.
Jul 2026 Orwa Mahmoud: Claude/Codex/Cursor shifts with persistent punch list, park-don’t-ask, watchman recovery, morning receipts — AdaptTable overnight: 9 items, gate-green commits, local until human push.
Jul 19 Warden: worktree-per-unit agents, clean-room verify + fail-closed AUDIT PASS/FAIL, then digest ranks survivors and integrate re-runs every gate after each merge — base branch untouched.
Jul 18 Stanley Yang: vendor-neutral coding-agent harness — disposable workspace pinned to a commit, typed tools, external budgets, independently captured patch, trusted verification; model never owns shell, credentials, or success.
Jul 18 Latch: independent reviewer≠fixer identities, recursion-guarded fix pushes, 3-cycle cap, anti-tamper on workflows/policy — clears the agent-PR firehose without ever pressing merge.
Factory.ai’s definitional guide: standardized inputs/tooling, measurable output, replayability—and how agent-native factories change the throughput bound.
Melbourne, Monash and Atlassian researchers ran Claude Code, Gemini CLI, Rovo Dev and CodeRabbit on 362 human-verified review tasks. Repo-grounded agents roughly doubled precise, human-aligned comments over static LLMs, but exploration and validation dominated cost, and failed reviews carried more validation, not less.
Jul 2026 jooray/night-agent: reset-anchored unattended missions in an isolated worktree; separate verify+commit per fix; never pushes — FOMO operationalized as using the subscription before it evaporates.
Jul 16, Zach Wills on a year of running agent fleets: an overnight run merged 10 PRs in dependency order, proof-carrying PRs pushed his review ceiling to about 40 a day, and the next wall was a 32GB machine with 27GB of swap.
July 2026 white paper from David Reed, PhD: a Claude Code harness where done is an Evidence Ledger, the maker never grades the diff, and autonomy tops out below merge — zero auto-merges by design.
Jul 16 Huang et al.: treat reviewed/tested/DONE as claims; admit transitions only with source-bound, mechanically verifiable receipts — zero false-DONE in 10/10; hidden-fail amplification 31→2 per 1,800.
Jul 16 (updated Sep 10) Łukasz Graliński, Pirxey: a one-person software factory ran 50 tickets in a week for about $2.90 each, including a 01:12–05:47 night run that finished 18 of 20. Humans approved every plan and merged every PR. The four things that broke were fixed in deterministic code, not prompts.
Harness field note: an agent holds no standing privilege — scoped subset of the triggerer’s access, server-side RBAC/OPA, and connector∩agent tool allow-lists on every call.
Companion Warp engineering note: how cloud software factories close the loop specifically on review quality and agent compliance.
Jul 15 Luo et al.: semantic Harness Gene Bank + validity/activation/significance/gain gates — +5.1% to +15.4% Pass@1 across seven benchmarks; beats GEPA/DGM under the same paired-2σ ruler.
Jul 2026 AgentLoop: harness-agnostic verify-gate wrapper; sealed held-out grading; claimed +16–22% SWE-bench-Verified lift over raw Claude Code / OpenCode by killing false-greens.
Jul 15 AgentLoop: harness-agnostic verify gate with held-out sealed cases — +16–22% SWE-bench-Verified lift over raw Claude Code/OpenCode by killing false-greens.
Jul 15 Barbaste et al.: source anatomy of Claude Code, Codex, Gemini CLI, OpenHands, Pi, OpenCode, and peers — seven subsystems, 29 patterns, and the harness→platform turn.
Jul 15 hamish-mackie/sloop: Rust daemon schedules Claude/Codex/OpenCode tickets into isolated worktrees inside configured running hours (e.g. 22:00–06:00), judges from observed exits/commits/tests — not agent claims — with build→review→merge flows and an opt-in verify-before-FF merge train.
Jul 15 Cortés / VichuFlow: agent self-report is not evidence — durable state outside the chat, kernel-run gates, stage advances only on artifacts; worker grades nothing.
Jul 15 Camplight/Kacharov: METR harness ≈ coin-flip vs naive loop; self-review can degrade accuracy; dark only where checking is cheaper than doing — and gates still rot if uninspected.
Jul 14 Wang et al.: under matched budgets on Terminal-Bench 2.1, automatic harness evolution fails to beat parallel sampling / sequential refinement; held-out transfer only +0.6 pass@1.
Study of 1.02M reviewed PRs across human, LLM-assisted, and agentic eras—efficiency gains without matching quality gains.
Closed-loop framework that codifies review feedback into versioned rules and a self-review checklist—0% recurrence for ruled-against errors in deployment.
Jul 12 berenddeboer/ready-for-agent: local clanker harness turns ready-for-agent issues into PRs across Claude/Codex/OpenCode/Grok — Merge Policy Off/Classify/Always, forge-agnostic, aimed at 150+ PRs/week without session babysitting.
Excubitor (Shepherd): mechanical PreToolUse guards + falsifiable intent records so a headless coding loop cannot merge, push, or delete on evidence it authored itself; frozen-oracle verification; candidate branch only.
Jul 10 Xia and Miller: a year of commits across 182 repos shows agent-written lines get corrective fixes at a 49% higher rate than human lines, a rising agentic share predicts more human bug-fixing the next month, and projects that merge more without review carry a heavier agentic maintenance load.
getprusik/prusik: independent verify outside the agent — prove parses runner output (executed ≠ skipped), prove-red requires acceptance tests fail without the change; 1,300+ tests, 43 field findings verified-closed.
Greptile argues PR review is one surface of a larger goal—codebase knowledge bases that catch second-order bugs beyond the diff.
Extract NL specs from vibe-coding sessions and validate them by blind regeneration—specs as the human-review artifact when AI code outpaces review.
Jul 9 Monkeyattack MCP: criteria-at-birth, claim_done with evidence, verify hard-rejects E_SELF_VERIFY; pending→claimed→verified|rejected is forward-only in SQLite.
Jul 9 Andrew Joia (LangWatch): after Claude Code pushed more code in three months than the prior year, PRs rotted unreviewed. A twice-daily agent that gives every PR exactly one owner and one reviewer took human merges from about 10 to 18.5 a workday.
OpenAI’s July 2026 analysis of evaluation flaws in SWE-Bench Pro—and why noisy tasks poison agent comparisons.
Adapt’s recurring Code Janitor: hourly/daily scan → one small CI-gated cleanup PR at a time, memory-repo dedupe, human merge only — modest verified throughput as the product.
Agarwal et al. code 3,100 practitioner docs into a causal model: agent impact on software is decided through how the team structures review and what expertise humans still bring.
July 2026 paper introducing agentic accept/reject review with SWE-Review-Bench and trajectory datasets.
Jul 7 Kan/Kan/Ertel: Claude Code + declarative HHL harness (kernel verify, no-Admitted, timeout, coverage) — full Iris core, 217 RustBelt, 318 reglang, 72 iris-lean; ~380 model-hours unattended.
Jul 2026 TokonoMix ANS: unattended ASK→PARK autonomy contract, deterministic test gates + git revert, watchdog resume, morning report that separates trusted done from daylight review.
Morgan Stanley’s OpenRewrite year at Code Remix — review is the bottleneck, risk-based PR routing via DDRA-style scoring, and agents that invent recipes while CPU tools run the blast-radius map.
Jul 5 trimmdev: workers/agents open PRs; a GitHub Actions coordinator arms, serializes risky lanes, revalidates stale heads, re-arms when GitHub drops auto-merge — and cannot merge past required checks.
Carreira’s feature-factory: durable run.json transitions, parallel build / serial merge, reviews bound to the judged commit, observed verify after each slice — autonomy without self-reported done.
Jul 4 Eftimie: harness = list of things you stopped trusting the model to do; mechanical/prompted/structural/instructional ladder; scope hooks, completion gates, fixture-tested rails.
Jul 2026 Gus IT defensive publication: sandboxed authors cannot push main; an external train merges only on fail-closed ledger verdicts from separate reviewer and tester personas.
Empirical mining of developer reactions to CodeRabbit reviews—field evidence on when agentic PR comments help or annoy.
Jul 2 Tian Pan: unattended agents move the bottleneck to 9 a.m.; assign trust tiers at dispatch, demand a provenance receipt with a deferred list, and batch the night's questions into one decision queue.
Jul 2 nightcrew: overnight control plane around Codex — plan from markdown backlog, worktree isolation, deterministic verify, independent merge review, streak guards, morning report.
Jul 2 CMU/Stanford longitudinal case: 802 developers, 196k PRs. Per-developer throughput hit 2.09×; reviewer load ~2×; human review coverage fell 89%→68% while automated review rose to ~84%. Merge and revert stayed flat on coarse proxies.
Jul 2026 Falco: zero-runtime CLI that never runs agents; ledger slices + verity claims reconciled to filesystem, git HEAD, and exit codes — exit 0/1/2; dogfooded 18+ green claims that caught its author.
Latent Space interview/dispatch energy around Warp’s CEO on factories as integrations into existing workflows—not a new IDE religion.
Jul 1, Takayuki Kawazoe on Codens' round-the-clock agent pipeline: every PR approved and green, zero merges for three days, caused by a review freshness check that recovery re-armed past existing approvals. The fix was to read the effective review state, not new events.
A 12-week case study and middle-range theory of “governance conversion”—turning agentic failure modes into durable controls.
Jun 30 Temrel: agent PRs arrive in ~90s, then burn ~90 minutes of trust-building — LinearB’s 4.6× review lag, graded verification (deterministic → review subagent → human), and evals that make the tax intentional.
Jun 30 sxivansx skill: scaffold LOOP.md + maker/checker/monitor agents + stop-check.sh — one on-disk cycle every trigger; human gate on merge; unattended means isolated.
June 2026: Vortex guides/verifies coding agents in-loop; Remediation Agent GA clears backlog with verified fix PRs.
Jun 30 2026 Ramón Chancay: each task gets a git worktree + branch; reproduce with a red test, require the whole suite and that the new test actually ran, ship a PR — never merge unsupervised.
Jun 28 Macedo: defines the external loop spec (trigger, goal, verify, stop, memory); codes 50 Loop Library entries—70% autonomous verify, but triggers and durable memory still mostly manual.
Jun 28 2026 Samuel Fajreldines: a verification harness turns agent attempts into reviewable PRs — scope, required proof, risk inventory, stop contracts, and gates that block before human audit.
Jun 26 Yuvrajsinh (Trevion build log): an always-on daemon dispatches headless agents into worktrees around the clock, about four commits a day including nights. Two things made it work: a per-area regression gate that auto-reverts before a PR opens, and a MAX_OPEN_REVIEW ceiling added after 50 open PRs piled up and started to rot.
Jun 26: Mnemosyne/Prometheus/Themis/… cron fleet on Curator — review, docs, triage, merge shepherd; deterministic-first Hyperion; shared lessons file; support tickets −50%.
Jun 25 paper: across 1,030 agent repair traces, 170 confirmed silent failures (omit / introduce / inadequate defenses); test-pass gates and LLM reviewer roles did not catch them.
Jun 23 (cross-posted Oct 6) Ahmet Zeybek: after agent PRs passed half of one service's changes and a two-approval PR dropped a schema default in production, the team moved review into failing CI checks. Median time to first review fell from 5 hours to about 2, median PR size from 310 lines to 160, and reverts within a fortnight from 6 a quarter to 1.
June 2026 release expanding governance: cross-repo blast radius, standards mined from PR history, and managed skill standards.
Jun 23 Alfredvc Show HN: Codex workflows as TypeScript FSMs — typed evidence, approval gates, repair paths, inspectable run logs; Codeflow slices roadmaps into verified commits.
Jun 22 Yan: production AI engineering harness — Shadow Explorer + multi-agent review + backtrack loops; first-pass review ~40%→70%; Agent = Model + Harness.
Haoran Yu and co-authors follow 400 repeat reviewers through 11,429 reviews of agent PRs in the AIDev dataset. Approval rises from 27.9% to 42.4% between a reviewer's first and latest tenth of reviews, inline comments fall 22%, and median wait grows from 3.9 to 13.5 hours. PR size stays flat, and approval of human PRs in the same repos falls over the same months.
Jun 2026 VisionForge Foreman: gated plan→ADR/PRD→issues→TDD→e2e; PreToolUse deny + read-only evaluator agent; Foreman alone flips pass; retro patches need bench reports.
Jun 21 Jirák: eight unattended hours — coverage 41%→86%, four real bugs in BUGS_FOUND.md; ~30% reverted; branch isolation + suite gates; review moved to a civilized hour.
Jun 21 Nightcore: local Claude Kanban studio — worktree builds, anti-gaming + Structure-Lock gates, independent reviewer PASS/CHANGES/FAIL, flight-recorder ledger; you approve merge.
Jun 2026 attunehq/bastion (still shipping Sep): agent volume ×100 breaks human diff review; narrow one-property reviewers run locally then in CI; verdicts aggregate to one required check — humans curate reviewers, not every line.
Jun 18, Cloudflare open-sources the security-audit skill that seeded its fleet vulnerability harness: multi-phase recon and hunt, adversarial validators, schema-checked findings, and a fresh agent that re-verifies every survivor — then a second model system for triage and human-gated fixes.
Jun 2026 yotamleo/Himmel: worktree-isolated Claude Code harness — /overnight-shift opens PRs while you sleep; hooks block main edits and unreviewed PRs; /pr-check multi-agent gate; handover state survives sessions.
Jun 16 Meela Dheeraj: hourly fresh sessions 11pm–7am, BUILD_SPEC + PROGRESS.md + git as memory — commit = checkpoint; irreversible ops deferred to ‘needs human’.
Jun 16 Banik et al.: 86,156 agent test-file patches across 33,596 PRs; 80.2% weak/no oracle; strong multi-signal oracles raise merge odds after controls (OR 1.28).
Jun 15 Sung: CI firewall for AI PRs — declared scope, workflow permission escalation, agent control-plane drift, matching test-file evidence — policy loaded from base, never from the untrusted head.
Jun 15: 5h Claude windows reset while you sleep — caffeinate + claude -p queue; FOMO as wasted capacity; productive complication: weekly ceiling still taxes daytime.
Factory.ai’s manifesto: model-independent, sovereign, self-improving SDLC loops where engineers build the factory that builds the software.
Addy Osmani’s 2026 map: Faros/GitClear/CodeRabbit data, blast-radius tiering, heterogeneous AI reviewers, and humans on the loop—not reading every line.
Jun 14 Qi et al.: Agentic Programming — code owns looping/branching/stop; LLM only at judgment leaves; OSWorld 86.8% in 15 steps vs 80.4% prior in 100; DAG context O(depth) not O(steps).
Jun 14 apsolut: Claude Code overnight wrapper — Night Contract, question firewall, PreToolUse outside permissions, Stop-hook ratchet vs model claims, SHA-256 event tape, tamper-evident Morning Receipt.
Jun 12 Darwin Agent Team: typed harness foundry + AEGIS Digester→Planner→Evolver→Critic — avg +14.5% (up to +44%) across five benches; variant isolation stops GAIA collapse; co-evolution +4.7% over harness-only.
Jun 11, Jason Vertrees (Heavy Chain). Five more failure modes from a year of a client-facing agent harness, this time where the signals mislead: an agent saves itself thirty seconds and spends the operator's afternoon, a config flag that reviews clean and does nothing, a gate with a predictable false positive that everyone learns to ignore, automated review that stops at style nits, and a prototype implemented as if it were the spec.
Jun 11 paper: reverse real vulnerability fixes, wrap them in 15 social-engineering PR framings, and watch review agents approve under prior-approval and dependency-compatibility narratives they cannot verify.
Jun 11 MakerChecker: gateway + scan + embedded governor — grant-pinned skills, SoD so maker ≠ checker, Ed25519 hash-chained audit anyone verifies offline.
Jun 10 Štefan Moravík, FlowHunt: across 1,409 merged PRs in ten months at QualityUnit, agent work tracked individual enthusiasm until a risk-tier policy file and a staged, bounded pipeline arrived. Then it jumped and stayed, with a human still making every merge.
Jun 10 Camus: deterministic local control plane — independent maker≠reviewer seats, repo tests as final word, bounded review loops, crash-safe resume; verifies overnight work without auto-merge.
Jun 10 loop-harness: cadence-driven Claude loops in worktrees — primary stages output, second agent must print VERDICT: PASS before PR/comments/Slack ship.
Jun 10 Deng: durable gated FSM + No-False-Success floor — 3,150-cell paired eval cuts fabrication to 0.95% vs StateFlow 25%; SWE-bench Lite −33 pp; honest stall over confident wrong patches.
Jun 10 BeamusWayne NightWatch: hash-chained black-box ledger for overnight agents; morning debrief re-runs recorded commands and diffs git vs claims; attest CI gate.
Jun 8 Zhang et al.: Weakness Mining → bounded proposal → promote only if held-in and held-out both non-regress; up to +132% relative Pass across Terminal-Bench-2 / SWE-bench Verified / AppWorld.
Jun 8, Jason Vertrees (Heavy Chain). From a year of a harness that writes, reviews and ships client software, five failure modes where the system said done and was wrong: a 738-test green board that wrote no rows, lessons that never became gates, one suppressed warning copied by every later agent, a fabricated API client built on by seven agents, and five gates that each assumed another had checked.
Jun 8 Dan Mercede / OrionArchitekton: fail-closed merge admission — parse LLM review, block on unparseable/schema-invalid/contradictory output; receipts over logs; reviewer is a command seam, not the product.
Jun 8 Cognition: first mergeability benchmark — 20+ OSS maintainers, 40h/task, blockers + quality rubrics; Opus 4.8 ~13% Diamond / 34% Main; 81% fewer misclassifications vs SWE-Bench Pro.
Jun 2026 ba1lly/nocturne: daemon picks agent-labelled issues across repos, isolated worktree + verify_cmd + review/fix budget, opens a PR; optional CI/review shepherding — never merges, force-pushes, or touches main.
Jun 8 Addy Osmani: automations, worktrees, skills, connectors, maker/checker sub-agents, plus disk memory — /goal stops on a separate model’s done-check, not the implementer’s optimism.
Stratton / godark: humans author architecture + scenario specs; implementer in Docker; quality + functional reviewers isolated (no file write); retry then squash-merge or needs-human-review.
Jun 5 Vanguard (still pushing Sep 23): watch loop claims tickets from GitHub/GitLab/Linear, plan→implement→review→adversary→repair in Docker, Proof-of-Work verify, then draft PR only — humans own merge.
Jun 3 Augment guide: Coordinator–Implementor–Verifier splits planning, execution, and spec-grounded validation so generation errors stop grading themselves — and names when single-file bursts still win.
Jun 1 zhjai agent-completion-gate: worker proposes candidate_complete; external check on real files + human-confirmed acceptance manifest grants complete — fills Claude/Codex /goal’s transcript-only stop gap.
Spotify’s Code with Claude talk write-up: 99%+ weekly AI usage, Honk in K8s with CI verification, Backstage golden paths as agent guardrails.
Ben Nevile on why agent-side filtering flips review etiquette from “only raise sure things” to “raise everything potentially relevant.”
June 2026 org field note: AI-coauthored PRs carry ~1.7× more review findings — rebuild around four-person pods (lead, two directors, quality engineer) so judgment and verification scale with agent output.
Demirer, Musolff and Yang match AI usage telemetry to more than 500,000 GitHub developers. Each generation of tool adds more coding activity, but the gains shrink at every step from commits to projects to releases, consistent with human review and judgment being hard to substitute.
Yefremov’s fork-and-own software-factory: four Claude roles on a GitHub label state machine, bounded retries, CI gate, auto-merger with guardrails — continuous verified throughput as a template.
May 30 2026 Ralph Workflow: overnight pattern = time-decoupled, structured plan→implement→verify→fix, verifiable receipts, reviewable git diffs — not a chat with a sleep timer.
Huiyan Wan's overnight-workflows plugins for Claude Code run multi-agent review panels on stacks of PRs while nobody is watching. One failure gets its own plugin: review subagents without a shell can't fetch the PR, return a BLOCKED report or review main instead, and by morning that empty report reads as clean. Re-dispatched with the diff on disk, one such reviewer found a data-loss bug that 179 green tests had missed.
May 29 42nights Otis: issue→PR overnight agent whose centerpiece is an 8-check harness (typecheck, suite, plan-tests, mutation-light, lint, diff cap, banned patterns, critic); never auto-merges.
Industrial study of RADAR at Meta—risk calibration and selective automation to reclaim reviewer time on low-risk changes.
May 25 dshakes compass: budget-capped Claude/Codex/Gemini harness, eval-gated guardrails, self-fixing PR loop with cross-model audit; fleet overnight; human merge permanent.
May 25 Anthropic: 93% approve rate → approval fatigue; environment boundaries (sandbox/VM/egress) cap blast radius so always-on loops can run without a human watching every tool call.
Makson179 Bello (May 2026; still shipping Sep 24): Codex/Claude/API coder in a disposable workspace; fresh-context supervisor for risky actions; completion review + adversary refuse 'done' on claims; ProgramBench C+A +26% mean vs raw; Efficient Budget −66% Codex weekly usage at same quality.
Ona's engineering-leader guide, built from its May 2026 Background Agents Summit with Stripe, Ramp, Spotify, Uber and Harvey, argues that faster individual agent sessions don't move org throughput; triggered, sandboxed agents that can verify their own work do.
May 22 Neely Dunlap (Faros): Agent = Model + Harness — tool orchestration, verification loops, memory, guardrails, observability; baseline cost-per-merged-PR and review velocity before buying more peak-session model.
May 21 Peralta et al. (MSR ’26): of 717 inspected AIDev Agentic-PRs, only 35.7% of rejections were clear agent failures; 31.2% workflow, 33.1% no rationale; 15.4% of merges needed reviewer feedback or commits—outcomes ≠ capability.
May 20 Pickuma: ingest→plan→execute→verify→package; verify-loop with retry bound; draft PR + agent-generated label; morning review non-negotiable; thin tests = no safety.
May 20 AIFactory PARR loop: plan→act→deploy→reflect→review — worktree isolation, live HTTPS verify, bounded auto-feedback, human-stop defaults; auto-merge only after reviewer gate.
May 18 SoloTerm/Aaron Francis: models → harnesses → metaharness — shared process graph, scratchpads, todos, timers, and cross-harness spawn so agents coordinate without you babysitting every tab.
Position/vision piece on how code review should change when AI agents author and critique changes across the SDLC.
May 17 JDonaghy code-coordinator: Claude CLI fleet across worktrees/machines; Test gates Review; independent review; merge queue + overnight drive-queue — human approves dispatch, harness drains to merge.
May 15 field note: Atlas/Ralph/Nitro/Sentinel each run inside a verification harness — live local stack, A/B benchmarks, plan-then-approve — because agents generate faster than humans can review.
May 14 Liu et al.: full-trajectory harness safety (boundary/fidelity/stability) on 210 tasks — best overall ~0.32; completion misaligned with SAR; multi-agent amplifies resource + info-flow violations.
May 13 Zhong & Zhu: eleven harness responsibilities, H0–H3 visibility ladder, and episode packages that adjudicate verification autonomy — reframing autonomous SE as a runtime-systems problem.
May 13 DeadReckon: harness around Claude/Codex/Gemini/Copilot/OpenCode — detachable Job, sandbox checks + fresh semantic judge, HMAC receipt; NEEDS_REVIEW when either key fails.
Process-level SWE-agent eval showing ~10.7% of passing trajectories are Lucky Passes—missing verification, blind retries, disordered stages.
May 12 Toni Antunovic: overnight diffs lack mid-flight correction — aggregate review, security triage, reasoning traces, semantic not line-by-line; constrain batch size or automate triage.
May 11 (edited Jul 28) Teemu Piirainen: once agents ran ten-hour sessions without asking questions, the anxiety didn't go away. It moved from answering agents fast enough to preparing enough work for the night, and mistakes started surfacing in morning review as the right execution of the wrong spec.
May 10 field note: Tian Pan on agent fleets as backlog generators unless you tier the queue — author class, supervision, risk — with traces to audit and CI never enough for business logic.
May 9, the wgmesh project found its review-and-merge bot passed any review submission, including Copilot's COMMENTED findings. Three PRs merged with 4 to 13 open findings; the fix requires an explicit APPROVED from each reviewer's latest review, re-checked just before merge.
May 6 OpenHands: agent-level critic gates broken trajectories; repo-level review skill + behavioral QA; /iterate closes the loop — 1,000+ reviews, 58% faster MTTM, ~25% coverage lift on their SDK repo.
May 6 from Felix Hellström: the first unsupervised night of his agent orchestrator ran 686 cycles in 7.5 hours, dispatched 103 tasks against 54 issues, auto-merged 11 PRs, flagged 9 for review, and killed and re-dispatched 8 stalled agents. Every failure lived in the harness, not the agents — and auto-merge on a green suite was luck, not a guardrail.
May 6 FelipeOFF Claude Code skill: Reviewer + Author subagents triage every human/bot comment, attribute CI before patching, stop on business rules; --auto is explicit, bare run only opens the PR.
Greptile mined millions of reviewed PRs and found end-to-end agent PRs rose from 0.86% to 27.6% of merges in fourteen months, with revert, churn and review-cycle numbers roughly matching humans. What changed was which bugs show up.
May 2 Developers Digest: task contract, scoped workspace, tool policy, logs, checkpoints, verification gates, cost/time budgets, final receipt — without them an overnight run is a chat session with more failure modes.
Apr 30 Florian Riquelme: Claude Code night-shift queue with a trust protocol — worktree isolation, blast-radius verify, halt-on-ambiguity, immutable evidence; the brief is a contract, not a transcript dump.
Apr 29 Developers Digest: /loop every 10m produced 12 repos and ~60 PRs — and a billing wall, a fake PR number, and a full morning of curation. Parallel overnight works; unverified claims don’t.
Apr 28 Lin et al.: ten AHE iterations lift Terminal-Bench 2 pass@1 69.7%→77.0% past Codex 71.9%; frozen harness transfers +5.1–10.1 pp cross-family; tools/middleware/memory carry the gain, not the prompt.
Apr 28 Addy Osmani: long-horizon vs long-running vs persistent agency; Ralph, Anthropic brain/hands/session, Cursor planner–worker–judge; write done-conditions and separate evaluator from generator before overnight FOMO.
Apr 22 Johnston (IMTI): PreToolUse git-commit hook + adversarial sub-agent — Verdict CLEAN pinned to diff hash; blocks review theater and post-commit fix loops.
Apr 22 gaurav0107/superhuman: multi-agent Claude/Codex harness scores its own PR on a 10-dimension rubric, iterates the weakest dimension to ≥95% over two runs, learns from merge_outcomes.jsonl — with merged PRs into transformers, airflow, oxc.
Apr 22 Besimple Broccoli OSS: assign Linear to a bot; Codex/Claude in your Cloud Run or Blaxel sandbox plan, implement, open a PR; review-feedback loops; human owns merge.
Apr 20 oa: worktree-per-task overnight queue for Claude/Codex/OpenCode — tail protocol, commit-since-start, verify cmd, AI review; fix-loop; events.jsonl + SUMMARY.md; crash-resumable daemon.
Apr 20 field report: up to seven specialist OpenCode reviewers under a coordinator; 48k MRs / 30 days, median 3m39s, $1.19 avg, break-glass 0.6%.
ppuliu skill: up to 8h on a feature branch; file-gated Codex review per commit; dual consensus to end; never push/PR — self-review only when Codex absent, explicitly flagged.
April 2026 GitHub project: DSPy-based prompt optimization that can land changes as PRs for human review.
April 2026 architecture note on Missions—workers, validators, and generate/validate loops for long-horizon work.
Apr 9 Ship with AI: 56h/week idle agent capacity; night-shift skill wraps Ralph with pre-flight, kill conditions, and morning categorization — calibrate small before you sleep on it.
Apr 9 dkdc.net: one orchestrator, a handful of workspace and review agents, CI-gated auto-merge across 12 repos, and an iMessage control channel produced 241 merged PRs by Monday. The owner's verdict: real bugs fixed, lots of suboptimal PRs, and every failure turned into a written convention.
Apr 8 Tenet: interview→spec→DAG→execute→evaluate loop — model-tier-shaped plans, grounded+ungrounded critics (code/test/e2e), doctrine that self-corrects between overnight cycles.
Apr 8 thin-supervisor: sidecar over Claude/Codex CLI — continue/verify/retry/recover/escalate; verifier + AcceptanceContract gate finish; Stop-hook closes the loop when you leave.
Apr 8 Anthropic (Martin/Cemaj/Cohen): session + harness + sandbox as swappable interfaces; pets→cattle; credentials never in the sandbox — meta-harness so overnight loops outlast any one harness recipe.
Apr 6 Orbit/Sharma: open-source overnight orchestrator — hardening + feature loops, worktree isolation, shift logs; runner enforces policy (blocked paths, verify command, anti-tunnel-vision), agent supplies intelligence.
NearForm Last Light: webhook/cron triage + PR review, Architect→Executor→Reviewer builds, permission-scoped App tokens, and evals scored on your own repos — not someone else’s bench.
Apr 4 Fornace: multi-wave overnight Claude SDK swarm in worktrees — self-review + simplify, cross-agent wave review, final architecture gate, and a librarian that compounds skills wave to wave.
Ogenrwot and Businge replayed git merges for 107,026 open or closed-unmerged agent PRs from the AIDev dataset and found 29,609 with textual conflicts and 336,380 conflict regions, with conflict rates varying sharply by agent.
Apr 3 Escape/Carossio SF field report: harness + AI-factory layers; FOMO as asset utilization — overnight produces candidates not commits; verification beats advice.
Apr 2 Thirdlayer: meta-agent edits agent.py (tools, verification, context, orchestration) against Harbor evals; keep score gains, revert regressions; >24h runs hit 96.5% SpreadsheetBench and 55.1% TerminalBench (GPT-5) from a minimal baseline.
Apr 2 Thoughtworks/Fowler: coding-agent harness as feedforward guides + feedback sensors, computational vs inferential, keep quality left — and why behaviour harnesses still demand human supervision.
Mar 31 NeoSigma: coding agent iterates the harness overnight — mine failures, cluster, propose changes, keep only what clears a self-maintained eval suite + held-out val; 0.56→0.78 on Tau3 with fixed GPT-5.4.
Mar 31 aws-samples ABCA: submit via Slack/Linear/Jira/CLI/webhook → MicroVM agent clones, tests, opens PR; blueprint mixes deterministic + agentic steps; memory from PR feedback; budgets and blast-radius limits for 24/7 runs.
Mar 30 Lee et al.: agentic outer loop over harness code with filesystem access to prior code/scores/traces — +7.7 pp vs ACE at 4× less context; TerminalBench-2 #1 Haiku / #2 Opus among reported agents.
Mar 28 Nightshift/AgentForge: Ralph loop with separate evaluator, mutation-guarded feature queue, and metrics dashboard as receipt — 29/30 features, 41 attempts, ~$0.95 in 76 minutes.
Benchmark that scores AI code-review quality against pull-request feedback rather than only against generation/pass metrics.
Mar 27 essay by Eduardo Aguilar Pelaez names the low-grade dread of an agent waiting on you, models it as a queue with the human as the server, and argues the lever is longer autonomous work intervals plus an orchestrator, not more agents.
An ASE 2026 paper ran a self-organizing agent team on four real projects for 142 wall-clock hours and 164 milestones with no human intervention after setup, for about $1,500. A dedicated verification phase rejected 18.6% of milestones on one project, unit-test gates still missed regressions the planner later caught, and reverts restored the baseline every time.
Mar 26 Roy: six-phase coverage-exhaustion loop over two production systems; 285+ iterations, ~$0.38/PR, zero oracle-detected regressions; drift control + drain/starvation gates keep the night honest.
Enterprise framing of the Agentic Software Factory—humans relocate to intent and outcomes; five transformation pillars including governance and scenario tests.
Anthropic Labs’ GAN-inspired planner/generator/evaluator harness—sprint contracts, Playwright QA, and shedding scaffolding as models improve.
Toub’s receipts: 878 CCA PRs, merge 67.9% vs 87.1% human; instructions 38%→69%; phone-flight FOMO creates review debt; fully autonomous 55.1% vs 86.2% with human commits.
Mar 23: Discovery-team field guide — CLAUDE.md plan, CHANGELOG memory, CLASS test oracle, tmux detach, Ralph loop; FOMO framed as opportunity cost of idle agents.
Sam Keen names the low-grade pressure to always have an agent working: unused usage feels like waste, an idle GPU feels like lost output. He argues it's an identity problem, not a budget one, and that the answer is deliberate goals rather than round-the-clock runs.
Mar 19 Simon Factory Lights Out: six sequential gates (Discover→Build→QA→Security→Verify→Ship), each requiring a named markdown artifact — no artifact, no progress; inner Build/QA loop capped at 10 rounds.
Mar 19 jonwiggins/optio: K8s (or laptop) control plane assigns GitHub/Linear/Jira tickets to Claude/Codex/Copilot worktrees, watches CI every 30s, auto-resumes on failures and review comments, then squash-merges — always-on swarm orchestration you own.
Mar 2026 Helmkamp/Fabro: Graphviz workflows with human gates, multi-model routing, cloud sandboxes, git checkpoints, and verification fix-loops — walk away while the line keeps running.
Mar 13 Deng/Chen et al.: on SWE-Milestone’s 98 dependency-linked milestones, frontier scores fall from >80% independently to ≤38.03% continuously—recall still grows, precision saturates, regressions snowball.
Mar 12 Vincent Tran: builder/judge separation, one browser agent per acceptance criterion, artifact-backed promotion — overnight coding without verification is morning cleanup.
Benchmark aimed at real-world utility of AI code review agents—not just comment similarity scores.
Mar 9: redis-rust and Helix — DST/TLA+/Kani pyramid replaces line-by-line review as primary correctness; production telemetry refines the harness when invariants were wrong.
March 2026 research preview: parallel find/verify/rank agents on GitHub PRs for Team/Enterprise, ~20 minutes and $15–25 per review.
March 2026 changelog: tool-calling reviews with broader repo context, running on GitHub Actions (including self-hosted caveats).
March 2026 product essay on CCR usage scale, agentic redesign, and redefining “good review” as high-signal progress rather than exhaustive nitpicking.
Mar 4 Wynne: agentic tools make soloing easier, but ‘pairing with Claude’ is a siren; as delivery speeds up, humans still co-create architecture, debt, and the constraints that check the robots.
Mar 4 AgentsMesh field note: 52 days / 600 commits building the fleet console with itself; Loops (cron) vs Tickets; tech-debt amplification; Autopilot after cognitive bandwidth broke at ~3 workstreams.
CodeRabbit’s write-up of Martian’s independent bench (~300k PRs): recall and F1 leadership claimed from whether developers act on comments.
Charles Sieg on the psychology of supervising coding agents by hand: parallel sessions finish in twenty minutes and then wait on him, an idle terminal feels like a stopped factory floor, and decision fatigue and craving arrive together. The cost lands in review quality.
Mar 2 Kelos: coding agents as K8s Tasks/TaskSpawners — GitHub/Jira/cron triggers, isolated pods, GitOps-able workflow YAML; self-development pipeline ships its own PRs.
Feb 23 Nightcrawler: bounded Claude Code episodes, HANDOFF.md cross-checked against git, immutable tasks.json, eight stop conditions including diminishing returns.
Feb 23 MSR ’26 study of 33,596 agent-authored PRs (AIDev): 71.5% merged overall, but merge odds track reviewer engagement and coordination stability more than iteration volume. Force pushes and large changes hurt; actionable review loops explain most qualitative successes.
Feb 2026 HackingDave/Nightwire: Signal-orchestrated workers; fresh-context verifier on the diff; test snapshots that fail only on NEW regressions; ≤2 auto-fix attempts — same brain never grades its own patch.
Feb 19 Stripe Part 2: 1,300+ minion-only PRs/week — blueprints interleave agent nodes with deterministic lint/push/CI; hot-ready isolated devboxes; at most two CI rounds before human review.
Feb 18 kodo: Gemini orchestrator drives Claude/Cursor/Codex workers overnight; independent architect+tester gates accept only after multi-round rejection — blackopt run: 3h, 9 verifier catches, 73 tests.
Shipwright Code Factory pattern: agents write 100% of code; risk-policy gate, SHA-tied evidence (API/CLI/DB/browser), automated remediation, and incident→harness-gap SLAs before merge.
Feb 17 Trivedy (LangChain): deepagents-cli hill-climbs via traces, PreCompletionChecklist self-verify, loop detection, and reasoning sandwich — same gpt-5.2-codex throughout.
Ryan Lopopolo’s report: ~1M lines, ~1,500 PRs, zero manually written code—environment design, legibility, and agent-to-agent review as the job.
Feb 8 Blake Crosley: fresh-context iterations + filesystem memory + stop-hook criteria shipped 3,455 LOC / 141 tests overnight — spawn budgets, non-trivial criteria, and cleanup beat one long Claude session.
Feb 7 Simon Willison on StrongDM’s dark factory: no human-written or human-reviewed code; scenarios as holdout sets; Digital Twin Universe clones of Okta/Jira/Slack for high-volume satisfaction scoring.
Feb 5 research: Wilson Lin’s multi-agent harness ran continuously a week (~1k commits/hr, 10M tool calls); planner/subplanner/worker handoffs; 100% per-commit correctness killed throughput.
February 2026 launch of Qodo 2.0 with specialized review agents and Code Review Benchmark 1.0 claims on real PRs.
A multi-language ACR benchmark that expands defect coverage ~285% via AI-assisted, expert-verified labels and stresses context retrieval choices.
Jan 24 jackzhaojin/continuous-agent: PM2 executive loop picks prioritized goals, spawns multi-vendor workers into an isolated sandbox, runs deterministic verifiers, and only sleeps when the queue is empty.
Jan 21 study of 33k agent-authored GitHub PRs: docs/CI/build merge most; performance and fix least. Among 562 labeled rejections, 38% closed with no meaningful human review, 23% were duplicates, 17% failed CI — agent misalignment was only ~2%.
January 2026 engineering essay on Bugbot versions, resolution-rate metrics, internal BugBench, and Bugbot Autofix beta.
CodeRabbit’s David Loker on why multi-bot-on-one-PR comparisons mislead, and which behavioral metrics (acceptance, noise, coverage) actually matter.
Engineering post on graders, pass@k vs pass^k, transcript review, and why SWE-bench-style outcome tests are necessary but incomplete.
Dec 28 ai-supervisor-foundry/foundry: BYOK orchestration for Cursor/Copilot/Gemini — provision sandboxes, dispatch worker nodes, hybrid local-then-LLM validation, DragonflyDB resume across multi-day runs.
Dec 25 Open Engine Zeroshot: authored executor–verifier graphs, independent reviewers with no shared session, SQLite ledger, bounded repair; delivery only after checks pass.
December 2025 post on deterministic verifiers + LLM-as-judge gates that block bad agent PRs from opening.
December 2025 launch: governance for AI-generated code plus a hybrid deterministic+LLM PR reviewer.
Alignment blog on training/deploying an agentic code reviewer—precision over recall, repo-wide tools, and verification cheaper than generation.
November 2025 engineering write-up comparing v2 vs v3 on upvote ratio, action rate, and cache economics after reviewing over a billion lines.
How to keep long-running coding agents coherent: feature lists as JSON, git+progress artifacts, clean handoffs, and end-to-end testing as a human would.
November 2025 docs notice: no new CodeGuru Reviewer repo associations; AWS points users to successor capabilities.
A difference-in-differences study of open-source projects that adopted Cursor, against 1,380 matched projects that did not, finds a large but transient velocity jump and a persistent rise in static analysis warnings and code complexity, which in turn drags later velocity down. Sonar cited it this week to argue verification belongs inside the agent loop and CI, not only at the pull request.
Part 1 of Spotify’s background coding agent: prompt-defined fleet shifts, pluggable agent CLI, 60–90% migration time savings, and new validation needs.
September 2025 funding + product post: Benchmark-led Series A alongside v3 agentic review, MCP server, and rule-file autodetection.
CodeRabbit frames AI code review as the quality gate for vibe coding, announces Series B, and ships a CLI that plugs into Claude Code, Codex, Cursor, and Gemini agents.
September 2025: GPT-5-Codex launch with trained code-review behavior and GitHub auto-review / @codex review flows.
July 2025: one-click LLM fix suggestions for Sonar-detected issues in UI/IDE, included on eligible plans.
July 2025 Google Cloud blog with Delivery Hero notes on Gemini-as-reviewer in the PR workflow.
ICLR 2026 oral: Genetic-Pareto prompt optimization that diagnoses trajectories in natural language—strong results including code-optimization search.
July 2025 launch: Bugbot GA with BUGBOT.md rules, one-click handoff to Cursor/background agents, and beta stats on PRs reviewed.
May 2025 Google I/O update: free individual + GitHub code-review agent generally available, powered by Gemini 2.5.
May 2025 product note: CodeRabbit ships IDE reviews as a lightweight pre-PR pass, with fuller PR reviews still recommended for production.
May 2025 AWS News Blog: Q Developer GitHub app for feature agents, automatic PR code review, and Java transforms.
April 2025 changelog: Copilot code review GA for paid subscribers, with automatic or on-demand PR reviews.
Graphite positions Diamond/Agent as accessible AI PR review paired with stacking, merge queue, and inbox—keeping the post-write toolchain as fast as generation.
March 2025: Accel-led Series B and Diamond as a standalone AI PR reviewer (later rebranded into Graphite Agent).
February 2025 integration note extending CodeRabbit beyond GitHub/GitLab/Azure DevOps into Bitbucket Cloud (beta).
October 2024 product blog celebrating Snyk Code auto-fix powered by DeepCode AI.
October 2024 SnykLaunch: DeepCode AI Fix merged into IDE workflows with verified remediation emphasis.
October 2024 rewrite pitch: Greptile indexes a codegraph and reviews PRs with whole-repo context on GitHub and GitLab.
September 2024 rename from Codium/CodiumAI to Qodo, with Qodo Merge (ex PR-Agent) as the review/automation product.
Same-day-as-rebrand funding note: $40M Series A to $50M total, with roadmap items for Cover, Merge learning from PRs, and Gen agents.
Broad 2024 survey of LLM agents across SE tasks—useful map of roles, tools, and feedback loops including review/testing.
August 2024 product post: @codereview agent in Bito’s IDE extensions for pre-PR reviews, with optional Git context.
OpenAI × Princeton subset of 500 solvable SWE-bench tasks—still the common public yardstick, with known contamination caveats later.
OpenAI’s LLM Critics Help Catch LLM Bugs—specialized critique models that find errors in LLM-generated code more reliably.
Stanford’s TextGrad treats LLM feedback as textual gradients over a computation graph—including measurable gains on hard coding problems.
June 2024 release: Review tab in the IDE, PR summaries/guides on GitHub, and multilingual review config.
Custom agent-computer interface for editing, navigating, and testing repos—large SWE-bench gains over shell-only baselines.
March 2024 changelog: Sourcery code review generally available on GitHub, replacing legacy bot PRs with inline review comments.
March 2024 PR: Semgrep Assistant GA for false-positive triage, autofix context, and custom rule workflows.
Early Greptile thesis: codebases are graphs, not file bags—reading at scale needs structure-preserving index, not naive embeddings.
Multi-agent code generation with iterative testing and optimization—explicit separation of writing code and writing/running tests.
ICLR 2024 oral introducing 2,294 issue→patch tasks graded by fail-to-pass tests—the default yardstick for agentic software engineering.
A 2023 framework that stores verbal reflections from failed trials and reuses them—strong gains on coding benchmarks without weight updates.
EMNLP 2022: grow a memory of misunderstandings plus user clarifications, then edit prompts for similar future queries—no retraining.
Early foundation model for automating code review activities—quality estimation and comment generation from diffs and review data.