AI code review automation: how Duolingo auto-approves 10%

Duolingo now lets an AI system approve about one in ten pull requests with no human reviewer. The interesting part is not the model. It is everything around it: a risk classifier with hard exclusions, a training programme that came first, and the willingness to measure reverts rather than assume safety. For anyone weighing AI code review automation, it is the most concrete published account of what it takes.

The numbers

In an InfoQ presentation, Duolingo software engineer Sarah Deitke described the results:

• Use of AI tools across engineering rose from roughly 80% to roughly 100%.
• Median time to merge a pull request fell from about 18 hours to about 12.
• Auto-approval reached about 10% of all pull requests within six months.
• Across roughly 200 auto-approved pull requests, there were around one to two reverts a day, with no significant increase in the defect rate.
• Between 20% and 30% of developers still request a manual review even when their change qualifies for auto-approval.

That last figure is worth dwelling on. A quarter of engineers choosing a human reviewer when they do not have to is not resistance. It is a healthy signal that the system is optional, and that people use judgement about when a second pair of eyes matters.

How the AI code review automation decides risk

The system classifies each pull request as low, medium or high risk, or as undetermined. It draws on the pull request title, the verification steps the author describes, the diff itself and an evaluation by a language model. Only low-risk changes that also meet additional criteria are approved automatically.

The categories are concrete:

• Low risk, eligible for auto-approval: Markdown changes, small straightforward code modifications and pre-approved configuration updates.
• Medium risk, requiring a human: API version updates, changes to foundational components such as authentication and logging, and large feature additions.
• High risk, never auto-approved: user permission changes, large infrastructure modifications and changes to core app features.

Notice what the classifier is really doing. It is not judging whether code is correct — that remains the job of tests. It is judging whether the consequence of the code being wrong is small enough to catch after merge. That is a far easier question, and it is the right one.

The guardrails that make it safe

The exclusions matter more than the model. Duolingo describes restricting the feature to repositories with a small number of code owners, allowing teams to include or exclude directories, disabling auto-approval for AWS resource changes entirely, excluding repositories subject to SOX or ISO regulation, blocking new engineers during their onboarding period, and offering optional daily summaries of what was approved.

Each of those encodes a decision humans made about where automated approval is unacceptable regardless of how confident the model is. Regulated code, infrastructure and the work of people still learning the codebase are exactly where a confident wrong answer is most expensive. None of those boundaries depends on the model behaving well.

Literacy before access

The part of the talk most teams will skip is the part Deitke emphasised most. Before automating review, Duolingo invested in AI literacy: lab-style workshops on MCP servers, editor rules, batch requests and evaluations, which 95% of the engineering organisation said taught them something new; observability dashboards tracking tool usage, cost and model families; short office-hour slots; shared channels and regular meetups; and early access through vendor partnerships.

Her framing is the line to keep: before adding agents and automation, ask whether you have built AI literacy in the organisation, not just AI access. The reason is practical. Engineers who understand how the system decides are the ones who notice when it decides badly and report it — and those reports became a dataset that improves the risk-scoring prompt over time.

What to take from it

This is the most useful answer yet to a problem we have written about before: coding agents generate change faster than teams can review it, so the bottleneck moves to review. Duolingo's response is not to review faster. It is to stop spending human review on changes where review adds little, and to protect the categories where it adds a lot.

If you are considering something similar, the order matters.

1. Start with the exclusion list, not the model. Decide which repositories, directories and change types are never auto-approved, and make that list the product.
2. Define risk by consequence. Ask how bad and how reversible a mistake would be, not how clever the change looks.
3. Measure reverts from day one. Duolingo's case rests on showing defect rates did not rise. Without that baseline you cannot make the claim.
4. Keep manual review a free choice. An opt-out that a quarter of engineers use is a feature.
5. Treat misclassifications as evaluation data. Every wrong call is a test case — the same discipline as scoring agents on what they did, not only what they returned.

Most teams will not reach ten per cent in six months, and should not try to. But the structure transfers to a company of any size: tests prove correctness, a classifier decides consequence, and humans spend their attention where consequence is high. That is how we would approach review automation on any AI engineering engagement, and why it starts as a quality engineering question rather than a tooling one.