Research Brief: Metrics for an Agent-Run Software Factory (2026)

#

Scope

#

This brief maps the 2026 state of the art for measuring an agent-run software factory, meaning infrastructure where LLM agents carry work through the software lifecycle and a human supervises by exception. The question: what do practitioners and researchers say to measure, and why, when agents do the producing?

In scope: delivery frameworks (DORA and its agent-era extensions), factory and fleet metrics, autonomy and intervention measures, unit economics of agent runs, quality and rework guardrails, the leading indicators of agent effectiveness, capability benchmarks, and agent observability standards. Out of scope: model architecture, prompt technique, and code-generation methods.

Time bounds: roughly October 2025 through October 2026, with a few older anchors (METR 2025). “State of the art” here means two things together: what shipped in production tooling and what independent telemetry shows, since the most-cited factory metric set is still vendor-defined.

Search Strategy

#

Note: the search index mixes vendor guides with independent telemetry. Where a number comes from a vendor with a product in the category (Augment, Warp, Port, Factory, Span, LinearB), the entry is labeled as such.

State of the Art

#

Main Approaches

#

Approach	Key idea	Representative work	Strengths	Limitations
DORA delivery metrics (pipeline)	Measure throughput and stability of the deploy pipeline	DORA 2025, DORA ROI 2026, five metrics guide	Established, comparable, free from existing CI/CD and incident data	Assumes human authorship; blind to generation volume; flat while agent output rises
Factory / fleet metrics	Measure the agent fleet and human reviewers as one production unit	Augment, “Software Factory Metrics”, Warp factory metrics, Port.io catalog	System-level; makes review capacity visible; gives cadence and owners	No dominant framework as of mid-2026; mostly vendor-defined; inconsistent definitions
Autonomy / intervention metrics	Measure how much work runs without a human	Harness Engineering KPIs (agent autonomy), Warp (automation percent), Factory Analytics (autonomy ratio), Span (turn yield)	Directly measures the factory’s purpose; moves week to week	No standard for what counts as an intervention; easy to raise by shrinking scope
Unit economics	Price output per accepted unit, not per token	Cognition (equivalent human hours), Warp (cost per PR), Hiddedesmet (AI credits per merged PR)	Ties spend to delivered work; the metric a CFO reads	Attribution is noisy; token efficiency is only meaningful at system level, not per person
Quality / rework guardrails	Hold stability while autonomy climbs	DORA stability, Faros tokenmaxxing data, SlopCodeBench, silent-failure telemetry	Catches the failure mode that volume hides	Churn and rework lag by 30 to 90 days; silent failures are invisible to agent self-report
Leading indicators / inputs	Steer the inputs that predict outcomes	Span (prompt clarity, environment readiness, quality stewardship), Harness Engineering (context density)	Actionable in the current sprint; the levers a team can change	Correlational; vendor-run studies; small samples
Capability benchmarks	Rank models and harnesses on fixed tasks	Terminal-Bench, SWE-bench, SWE-bench Pro, DeepSWE	Repeatable, comparable, good for shortlisting	Scaffold confound (2607.22585), contamination (SWE-bench Verified retired), saturation, weak predictor of production reliability
Observability standards	Standardize agent telemetry	OpenTelemetry GenAI semantic conventions	Cross-vendor spans, metrics, and evaluation attributes; emitted by Copilot, Codex, Claude Code	Development status; client-side; no factory-level semantics yet
Productivity research (anchor)	Ground productivity claims in measurement	METR, Anthropic Claude Code study, NBER w35275	Independent of vendors; large samples	Snapshots of a moving target; different settings

Current Best Results

#

What the numbers show, grouped by claim.

Generation rises, delivery does not follow proportionally.

• Faros AI (vendor telemetry), across 22,000 developers on 4,000 teams: task completion up 34%, epics per developer up 66%, but bugs per developer up 54%, median review time up 5x, PRs merged with no review up 31%, and code churn up 861% in high-adoption environments.
• A second Faros study across 1,255 teams found 98% more merged pull requests alongside 91% longer review times, with no organizational-level correlation between AI adoption and outcomes (Augment, citing Faros).
• NBER w35275, over 100,000 GitHub developers: commits rise 40% for autocomplete, 140% for interactive agents, and 180% for autonomous agents, attenuating to 50% for projects and 30% for releases.

Verification is the binding constraint.

• LinearB (vendor telemetry), 8.1 million pull requests across 4,800 teams: agentic pull requests wait 5.3 times longer than unassisted ones before pickup, and AI pull requests 4.6 times longer for a first review.
• Duma et al., EASE 2026: most AI-generated pull requests receive no review at all, and the reviews they do receive are dominated by AI agents, not humans.
• DORA names the mechanism the verification tax: time saved writing is re-spent auditing, and 30% of developers report little or no trust in AI-generated code.

Autonomy is measurable and currently modest.

• Anthropic’s study of about 400,000 Claude Code sessions (October 2025 to April 2026): people make most planning decisions and the agent makes most execution decisions. Verified success is about 30% for software occupations and 26% elsewhere, rising to 34% and 29% for sessions that produce code. Sessions rated expert reach verified success more than twice as often as novice ones.
• Harness Engineering defines agent autonomy as the percentage of PRs merged without human code intervention, places high autonomy in the 70% to 90% band, and reports 35% current against a 60% near-term target.

Benchmarks do not predict production.

• Terminal-Bench 4.0, released 2026-08-28, tops out near 58% on the public board (GPT-6 Astra in Codex at 58.2%, Claude Fable 5.1 in Claude Code at 57.9%).
• SWE-bench Verified is retired at the frontier; OpenAI stopped evaluating on it on 2026-02-23 after a test audit, and the leaderboard has no new entries since 2026-02-26.
• The Scaffold Effect paper finds harness choice induces up to a 40x difference in tokens per solved task, with pass-rate differences of 0 to 8 points, so a leaderboard row is a harness plus a model, not a model.
• Reported lab-to-production gap: about 37 points; production success across 4.5 million runs reported at 56.6%, with 45% to 75% of failures reported as success.

The metric-gaming failure is documented.

• DORA on tokenmaxxing: raw token count is an input, easy to game, and a vanity metric. The piece cites Salesforce Agentic Work Units, Shopify replacing a leaderboard with a usage dashboard plus circuit breakers, and Starburst measuring DORA plus cost per accepted change and rework.
• Meta’s internal token leaderboard and Amazon’s Kirorank were both withdrawn after engineers gamed them by running agents on idle work.

Consensus and Disagreement

#

Where the field agrees:

• AI raises activity, not delivery, on its own. DORA 2025 finds AI amplifies the existing system, with higher adoption associated with both throughput and instability. The NBER and Faros data show the same gap.
• The constraint moved from generation to verification. Augment, Warp, Port, and DORA all locate the bottleneck at review capacity.
• Effort and input metrics are not targets. Tokens, lines, commits, and PR counts are consensus vanity metrics. DORA’s Goodhart framing, Faros’ tokenmaxxing critique, and IBM’s “valuemaxxing” all agree.
• Measure the system, never the individual. DORA, DX, and Augment all warn against per-person use; DX treats agents as extensions of the team that directs them.
• You need a pre-AI baseline and PR tags by producer type before any attribution means anything (Augment, DX, Hiddedesmet).
• Cost per accepted or completed unit beats cost per token. Warp uses cost per PR; Hiddedesmet uses AI credits per merged PR; DORA cites cost per accepted change.

Where the field disagrees or has no standard:

• No dominant fleet framework exists. Augment states outright that no fleet-level framework has become dominant as of mid-2026, citing a Harness survey of 700 developers and leaders.
• What counts as an intervention. Warp counts human touchpoints per PR, Factory counts tool calls per user message, Span counts merged lines per human turn, Harness Engineering counts PRs merged without human code intervention. These are different units.
• Whether autonomy or outcome leads. Port.io puts outcome-verified “agent success rate” as the gold standard; Harness Engineering puts agent autonomy first.
• Whether LLM judges are reliable. Berkeley and Stanford work cited by Prefactor found LLM judges reach a maximum AUROC of 0.65 on false-success detection, and that deterministic state checks beat them.
• How to count DORA lead time for agent-initiated PRs. Hiddedesmet notes there is no consensus and recommends keeping agent PRs separate.
• Model versus harness attribution. The Scaffold Effect paper and the reliability monograph both treat this as open.

Key Sources

#

Source	Type	Year	Authority signal	Credibility	Why it matters
DORA, State of AI-assisted Software Development 2025	Report	2025	Long-running Google Cloud research program (State of DevOps since 2014)	Strong	The 90% adoption, 80% perceived productivity, 30% distrust figures; AI as amplifier; throughput up with instability
DORA, ROI of AI-assisted Software Development	Report	2026	Same DORA program; ROI report leans advocacy	Strong	The J-curve and verification tax; a worked ROI model; the instability cost
DORA, “Balancing AI tensions”	Insight	2026	Same DORA program	Strong	Names the verification tax and the review-capacity shift
DORA, “Finding balance in the era of tokenmaxxing”	Insight	2026	Same DORA program	Strong	The clearest statement that input metrics are gameable, with company counterexamples
Augment Code, “Software Factory Metrics”	Vendor guide	2026	Vendor with a product in the category	Moderate	The five-category taxonomy, the instrumentation order, and the DORA-versus-factory comparison
Warp, “Closing the loop with self-improving cloud software factories”	Vendor blog	2026	Vendor with a product in the category	Moderate	The core factory metric set: PR throughput, cost per PR, automation percent, savings over human work
Port.io, “How to measure agentic engineering”	Vendor guide	2026	Vendor with a product in the category	Moderate	A 25-metric catalog including outcome-verified agent success rate and automation coverage by phase
Cognition, “Estimating the Productivity of an Autonomous AI Software Engineer”	Vendor research	2026	Vendor (Devin) measuring its own product	Moderate	Equivalent human hours as the unit, and evidence that lines changed is a weak proxy
Span, “AI Coding Agent Effectiveness: Leading Indicators”	Vendor research	2026	Vendor with a product in the category; small, vendor-run study	Moderate	Prompt clarity, environment readiness, quality stewardship, with effect sizes
Harness Engineering KPIs	Open repo	2026	21 stars, 12 forks; small org repo	Weak	A concrete autonomy target, the harness-coverage and context-density inputs, and a composite unwatched-safety outcome KPI
Faros, “Tokenmaxxing”	Vendor research	2026	Vendor with a product in the category; telemetry methodology not published	Moderate	Telemetry that pairs rising throughput with rising bugs, review time, and churn
NBER w35275	Paper	2026	NBER working paper (Demirer, Musolff, Yang)	Strong	Commit gains attenuate across the production hierarchy
Scaffold Effect in Coding Agents	Paper	2026	arXiv preprint; no citations yet	Moderate	Harness choice drives a 40x token spread; leaderboard rows are model-plus-harness
Engineering Reliable Coding Agents	Monograph	2026	arXiv preprint; single author	Moderate	Framing agents as systems; evaluation and operation as a dependency chain
Duma et al., AI PR review	Paper	2026	arXiv preprint (EASE 2026)	Moderate	Most AI PRs go unreviewed; AI agents do much of the review that happens
Anthropic, Claude Code expertise	Vendor research	2026	Vendor (model maker) studying its own product; large n	Moderate	Autonomy and success rates from about 400,000 live sessions
METR, early-2025 AI and developer productivity	Paper	2025	Independent nonprofit; 16 developers, not a large sample	Strong (small n)	19% slower while believing 20% faster; the perception gap
OpenTelemetry GenAI semantic conventions	Standard	2026	open-telemetry org repo; 407 stars, 116 forks	Strong	The emerging standard for agent telemetry, spans, token metrics, and evaluation attributes
Bun, rewriting Bun in Rust	Postmortem	2026	Company postmortem; self-reported and promotional	Moderate	The positive case: 535,496 lines in 11 days, zero tests removed, adversarial review, a present supervisor

Inline Citations Not Promoted to Key Sources

#

Sources cited in the body that do not carry a Key Sources row, listed with a rating so the gap is visible.

Source	Type	Authority signal	Credibility	Issue
DORA, “Software delivery metrics”	Docs	Same DORA program	Strong	Left off the Key Sources table only to avoid a fourth DORA row
SWE-bench Verified	Leaderboard	Maintained benchmark; 500 human-reviewed instances	Strong	Cited for its retirement, not for a score
Terminal-Bench	Leaderboard	Maintained by Stanford, the Laude Institute, and Harbor	Moderate (model-plus-harness)	Scores shift with harness and version
LinearB, “AI in software development”	Vendor research	Vendor of engineering-metrics tooling; two large PR datasets	Moderate (vendor)	Promotes the category it sells into
Factory, “Factory Analytics”	Vendor blog	Vendor with a product in the category	Moderate (vendor)	The autonomy ratio is vendor-defined
DX, “Measuring AI code assistants and agents”	Vendor research	Vendor (DX platform), millions of benchmark samples	Moderate (vendor)	Cited for the system-not-individual stance
IBM, “Tokenmaxxing is dead, long live valuemaxxing”	Vendor blog	Vendor (IBM Bob)	Moderate (vendor)	Coins a term on a page selling a platform
Harness, State of Engineering Excellence	Vendor survey	Vendor (Harness); 700 self-reported respondents	Weak (self-reported vendor survey)	Reached second-hand through Augment
Hiddedesmet, “AI coding agents need KPIs”	Personal blog	Named author, no sales pitch	Weak	A practitioner proposal, not evidence
Sonnetcode, “37% lab-to-production gap”	Services blog	No named author; ends in a sales pitch	Weak	States an 88.6% SWE-bench Verified score as current in July 2026 while the brief says the benchmark was retired in February 2026
Prefactor, “Silent wins, visible fails”	Vendor blog	Vendor (Prefactor)	Weak	Cites its own posts; the Berkeley/Stanford LLM-judge finding links back to Prefactor, and the 56.6% figure links to a personal blog

Also cited inline without a link, so they cannot be rated: SlopCodeBench, SWE-bench Pro, DeepSWE, the Meta token leaderboard and Amazon Kirorank withdrawals, and the OpenAI SWE-bench retirement date.

Open Questions / Gaps

#

1. No standard autonomy metric. Every vendor defines intervention differently, so cross-organization comparison is impossible today.
2. Silent failure is unmeasured. Agent self-report succeeds while the environment shows failure in 45% to 75% of cases, and the field has no settled independent-state check.
3. Model versus harness attribution. Benchmarks and production results both mix the two, and the fix (a controlled harness variable) is only recently proposed.
4. Long-horizon reliability measurement. Per-step success compounds, and proposed tools (reliability decay curves) are not yet standard.
5. Silent quality erosion. SlopCodeBench and churn data suggest maintainability degrades across long agent runs while unit tests stay green.
6. Economic attribution at scale. Cost per accepted change is agreed in principle, but there is no common definition of “accepted.”
7. Metric governance. The field agrees not to use these metrics on individuals, but no framework encodes that as a constraint.
8. Comprehension debt. Who can maintain the agent-written code, and how is ownership measured, remain open.

Recommendation for the Article

#

Core insight: the field has converged on the claim the draft already makes. Generation is cheap, verification is the binding constraint, and the only factory metrics worth a target are the share of work that runs without a human and the share of that work that survives, priced per accepted change and guarded by stability. The draft’s autonomy-plus-yield framing matches the strongest 2026 evidence; the new material is the proof behind it and the two refinements below.

Angle: keep the draft as the measurement layer of the software-factory literature, and use the 2026 telemetry as its evidence. Three upgrades are worth making.

1. Ground the “output rises, outcomes do not” claim in telemetry. The draft asserts it; Faros (98% more PRs, 91% longer reviews, churn +861%), NBER (commits +180% but releases +30%), and DORA (throughput up with instability) prove it. Add one or two of these numbers.
2. Add independent verification as a named guardrail. The 2026 evidence says agent self-report is not a success signal (45% to 75% false-success). This extends the draft’s yield definition: yield must be measured by querying the environment, not by reading the agent’s summary.
3. Refine two claims. DORA now carries five metrics and added deployment rework rate itself, so rework is not only a factory addition. And the autonomy definition should name the unit of intervention explicitly, since the field’s disagreement is exactly there.

Sources to lead with: DORA 2025 and the ROI report; Augment “Software Factory Metrics”; Warp’s factory metrics; Faros tokenmaxxing; Cognition; Anthropic Claude Code study; OpenTelemetry GenAI. The Bun postmortem remains the one positive case where high autonomy shipped, and it shipped because verification was dense.