AI coding agent success rate on private code: 38.8% at best

What is a realistic AI coding agent success rate on your own company's code? A new benchmark offers the first direct answer we have seen, and it is sobering. On tasks cut from private, licensed production codebases, the best agent-and-model combination solved 38.8% of attempts. The average across eight combinations was about 27%. Public leaderboards, built on open-source code the models may have seen in training, suggest far higher numbers.

What Real-SWE measured

Real-SWE comes from Specific Labs, an evaluation start-up whose co-founder described it as a Y Combinator Fall 2025 company. Instead of scraping issues from public GitHub repositories, each task comes with a private production codebase licensed from a real business. The New Stack reports that these include a consumer product with more than 200,000 users and a fintech platform that has processed more than 100,000 bank statements.

Because neither the code nor the solutions are public, they are much less likely to have appeared in a model's training data. Specific Labs cannot guarantee that, but the design removes the most common objection to public benchmarks. The tasks are also broad: solutions touch a median of 11 files, nearly double the six-file median of other recent benchmarks.

The scale is small — ten tasks, eight models, eight attempts per task, 640 runs in total — and each model ran in its maker's own coding tool, so the scores measure the full setup rather than the model alone.

AI coding agent success rate by model

• Fable 5.1 on Claude Code: 38.8%
• GPT-6 Astra on Codex CLI: 33.8%
• Gemini 3.8 Flash on Gemini CLI: 31.2%
• GLM 5.3: 28.8%; Grok 4.6 and Muse Spark 1.3: 23.8% each; Kimi K3: 18.8%; GPT-5.6 Sol: 16.2%

Those eight figures average 26.9%, a number an independent analysis by Pebblous confirmed from the per-task results: 172 successful runs out of 640. Six of the ten tasks had success rates below 15%. A tax-jurisdiction bug was fixed 3.1% of the time. No model solved an analytics stream reducer in 64 attempts. Some agents went eight for eight on one task and zero for eight on the next.

Why AI coding agents fail on real projects

The failure analysis is the most useful part for anyone planning delivery. According to The New Stack, Fable 5.1's failures were mostly missed requirements (36.7%) and integration errors (34.7%). GPT-6 Astra's split evenly between integration errors and unverified assumptions, at 34% each. Integration errors appeared in nearly half of Gemini 3.8 Flash's failed runs.

None of these is a failure to write code. They are failures to understand the system: not noticing a requirement, not wiring a change into the existing flow correctly, or assuming how something works instead of checking. Pebblous notes that Real-SWE's task instructions deliberately withhold where in the codebase a change belongs — much as a real ticket rarely tells a new engineer which file to open.

That matches what we see on client projects. Agents are strong at well-bounded work with clear acceptance criteria and good tests. They struggle with cross-cutting changes in unfamiliar code, where the hard part is knowing what the change touches. It is also why we argue for scoring an agent's trajectory, not just its answer: the wrong assumption usually shows up in what the agent read, or failed to read, long before the final diff.

Will AI coding agents speed up my project?

Probably, on the right tasks. Real-SWE does not say agents are useless; it says an unsupervised agent handed a vague ticket on unfamiliar production code will fail most of the time. Several practical conclusions follow for clients planning AI-assisted delivery:

1. Plan from private-code numbers, not leaderboards. If a vendor's estimate assumes agents complete most tasks unaided, ask what codebase that assumption was measured on.
2. Specify more, not less. Missed requirements are the leading failure. Explicit acceptance criteria — including what must never happen — make agent output checkable. We have written before about why verification becomes the bottleneck.
3. Invest in tests first on legacy code. Integration errors are caught by integration tests. Where those do not exist, characterization tests are the cheapest safety net to add before agents touch the code.
4. Measure consistency, not a single run. The same agent solved some tasks every time and others never. Running tasks several times, as we recommend in AI agent consistency testing, reveals which kinds of work you can delegate.
5. Keep review with people who know the system. The failures here are exactly the ones a reviewer familiar with the codebase catches quickly and an agent does not.

How to read one benchmark

Real-SWE is new, single-source and small. Ten tasks cannot characterise all enterprise software, and the task designers chose deliberately hard problems. It does not prove that public benchmarks are inflated by contamination. The ranking between the top three is close enough that a different task set could reorder it, and changing the tool around a model can shift its score.

What it does establish is a floor for expectations: on unfamiliar private code, with minimal guidance, even the best current agent fails more often than it succeeds. That is a planning input, not a verdict on the technology. Our AI engineering services team uses agents daily on client work, and the gains come from pairing them with specifications, tests and engineers who know where the change belongs.