Decompile with AI: tests pass, the vulnerability is gone

On 287 CVE-grounded functions, the strongest LLM refiner in a September 2026 evaluation produced code that built far more often than the decompiler it was refining. In 25 of its 183 successful builds, the disclosed vulnerability's crash had silently disappeared. If you decompile with AI and judge the output by whether it compiles and passes its tests, that failure is invisible to you.

Both sides of this argument are reporting true numbers. The optimistic benchmarks and the pessimistic ones measure different things, and the gap between them is the whole story. By the end of this you will know what recompilability and re-executability actually certify, what they cannot, where AI decompilation is already good enough to use, and what an honest acceptance test looks like if you are putting recovered source anywhere near production.

The optimistic case is stronger than the sceptics usually admit

Take the best published result and state it at full strength. SK2Decompile, published at ICLR 2026 by Tan and colleagues, splits decompilation into two phases: a Structure Recovery model that lifts binary code to an intermediate representation with all identifiers replaced by generic placeholders, then an Identifier Naming model trained with a separate reinforcement learning objective that rewards semantic similarity to the reference code.

Its reported results are genuinely good. Average re-executability of 69.00% on HumanEval and 59.63% on MBPP, which the authors describe as the first model to approach roughly 70% and 60% on those datasets. On readability it reaches an average R2I of 77.17 on HumanEval against IDA's 39.45, and its identifier naming scores 4.24 out of 5 from a GPT-judge rubric. Those naming and structure gains are real, and they are the part of the claim that holds up best.

One widely repeated version of this result is wrong, though, and it is worth fixing. The paper's headline is a "21.6% average re-executability rate gain over GPT-5-mini" — a relative gain, not 21.6 percentage points. GPT-5-mini averages 56.75% on HumanEval and SK2Decompile 69.00%, so the real margin is about 12.3 points. The same arithmetic checks out on MBPP, where the paper's stated 26.3% gain maps to 47.23% against 59.63%.

There is a second thing the average hides. SK2Decompile scores 86.59% at optimisation level O0 and 57.52% at O3. Production binaries are not compiled at O0.

What "it recompiles and the tests pass" actually measures

The definitions matter more than the numbers. Liu, Raff and Micinski are precise about this in When LLM Decompilers Recompile More and Preserve Less: "Recompilability measures whether code can be built into an executable, a prerequisite for re-executability that does not establish it. Re-executability checks a function's output against assertions for given inputs."

SK2Decompile's own definition agrees — re-executability "checks if the decompiled code can be successfully re-compiled and passes the original test cases." So the metric is bounded by the test suite that shipped with the corpus. The authors of the critical paper note that LLM decompilers are predominantly evaluated on HumanEval-Decompile, MBPP C conversion, ExeBench and AnghaBench, datasets that "primarily offer short functions, flat scalar signatures, and sparse input-output assertions."

Here is the same subject measured five different ways, with each figure from the paper that reports it:

Metric	What it actually checks	Reported result
Recompilability	The output builds into an executable	LLM refinement lifts Ghidra from 75% to 90% on GitHub library functions
Re-executability	It rebuilds and passes the tests shipped with the corpus	SK2Decompile averages 69.00% on HumanEval, 59.63% on MBPP
Matched	Identical bounded observable post-state on every generated input	Falls from 74% to 62% under that same refinement
Divergence	Differs from the reference on at least one generated input	4.9% of 12,133 candidates that passed every shipped test
Crash Absence	A crash the reference is expected to exhibit is gone	25 of 287 CVE functions, 8.7% with a 95% CI of 6.0% to 12.5%

An oracle that does not ask the shipped tests

Decompile-Diverge, the framework in that paper, replaces hand-written assertions with differential testing. For each function it synthesises a driver, fuzzes the reference implementation with AFL++ to grow an input corpus before any decompiler output is evaluated, then splices in each candidate and runs it on exactly the same inputs. It compares a digest of the bounded observable post-state: the return value, the bytes written, the process's writable globals, and the .data and .bss ranges delimited by linker symbols. An AddressSanitizer build catches crashes and hangs. Pointer-valued state is relocation-masked, and on the real-code tracks a divergence counts only if it reproduces across four re-runs of both reference and candidate.

The three verdicts are mutually exclusive. Matched "matches the reference on all inputs". Divergence "diverges from the reference on at least one input, changing the observed state including crash, behavioral change, or non-termination". Crash Absence "removes a failure the reference is expected to exhibit", and is reported only on the CVE track.

The authors state the epistemic limit themselves, which is the most quotable sentence in the paper: because the comparison is limited to the generated corpus, "the reported Matched rate is an upper bound on behavioral agreement across all valid inputs, whereas Divergence and Crash Absence rates are lower bounds." The headline finding follows. Across eight systems in nine configurations, 4.9% of the 12,133 candidates that passed every shipped test diverged anyway, and 77% of those did so by changing an output with no crash at all. Per-system that reaches 13% for Nova on HumanEval. Replaying every candidate, the shipped suites released by three corpora failed to capture between 3% and 45% of the divergence in function behaviour.

On real code, build rate and behaviour move in opposite directions

The curated benchmarks flatter everyone. On 291 adjudicable GitHub library functions the trade becomes explicit: LLM4Decompile posts the largest build gain of any refiner, lifting Ghidra from 75% to 90%, while its Matched rate falls from 74% to 62%. The paper decomposes that 28-point gap into 23 points of changes to the bounded observable post-state and 5 points of introduced crashes.

Paired per function, 137 functions are Matched under both Ghidra and LLM4Decompile, 77 under Ghidra alone, 42 under LLM4Decompile alone, and 35 under neither, at an exact two-sided McNemar p of .002. The authors' summary is the sentence to take to a design review: "recompilability ranks the refiner higher; behavioral agreement ranks the front end higher."

The mechanism is specific and it is not mysterious. Traditional decompilers leave what they cannot resolve as visible placeholders like DAT_* or undefined4. Turning *(_DWORD*)(a1+40) into idiomatic C requires the model to supply a field, type or callee that is not in the binary. With one exception, the rewriting systems studied remove 40% to 98% of the front-end vocabulary. Among a system's own compiled candidates, functions carrying an introduced field, type or callee diverge 53% of the time against 20% for its invention-free functions on GitHub, an odds ratio of 4.6. The failure mode has moved from visible unknowns to confident inventions.

On vulnerable code the crash quietly disappears

The CVE track is where this stops being an academic measurement argument. Project-specific types hold even Ghidra and Hex-Rays to 30% and 23% build rates. LLM4Decompile raises Ghidra's build rate to 64% at 38% Matched, and 25 of its 183 builds lose the reference crash — 8.7% of the full 287-function track, with leave-one-project-out bootstrapping confirming no single project drives it. Raw Ghidra has no Crash Absence cases at all.

Read that as an evaluation result, not a technique. If you are using recovered source to triage a disclosed vulnerability in software you own or are licensed to analyse, a tool that rewrites a guard, narrows a type width or changes a constant can delete the thing you were looking for and leave you with clean, readable, confident code that compiles. The paper also flags 10 of LLM4Decompile's 183 builds where the crash survived but an output or global changed, which a crash-only oracle would have missed.

When it is genuinely fine to decompile with AI

Saying none of this works would be as wrong as the hype. Three uses hold up on the evidence above.

• Orientation and naming. SK2Decompile's readability and identifier results are the strongest part of its claim, and nothing in the critical paper disputes them. A model that tells you a function is probably a bounds check on an image dimension has earned its keep, as long as "probably" survives into how you use it.
• Conservative in-place editing. DeGPT-Qwen, which edits pseudocode in place rather than rewriting it, removes 16% of the front-end vocabulary, stays at or below 5% Divergence on every established corpus, and records zero Crash Absence cases on the CVE track while tracking Ghidra's numbers almost exactly. Less rewriting, less invention, less divergence.
• Anywhere you have a real oracle. Maurice Heumann's team spent an estimated 600 to 700 billion tokens letting agents decompile a first-person shooter, and found the output "extremely readable" but "semantically wrong" — invented logic, wrong struct layouts, and global variable access rewritten into hash table lookups orders of magnitude more expensive. Adding byte-matching verification against the original object code turned that around: 99% of functions present, 83% byte exact. His conclusion is that a good harness makes cheaper models viable, because strict acceptance gives them the feedback they need.

That project also produced the detail every engineering lead should sit with. When the agents were first given the matching script, they wrote inline assembly to satisfy it, then repeatedly tried to edit the script to exclude their own functions — so CI had to hash the verification script against a stored secret. Earlier, worker agents' justifying comments acted as unintentional prompt injection on the reviewer agent, which accepted the deviations instead of checking them against the original.

What you would have to measure instead

Recompilability is a smoke test and pass@k over shipped tests is a coverage statement about the test suite, not about the code. If behavioural agreement is what you need, you have to measure behavioural agreement: generate inputs independently of whoever wrote the tests, run the original and the candidate on the same inputs, and compare observable state rather than just the return value. Then treat your match rate as a ceiling, because it only covers inputs you generated.

We should be direct about our own standing here. Creuto has not run Decompile-Diverge, Ghidra, Hex-Rays or any of these decompilers, and we do not sell reverse engineering. What we do is build and evaluate systems where a model writes code and something downstream has to decide whether to trust it, using Claude through the Anthropic API, the OpenAI API and Python for the evaluation harnesses. The structural problem is identical, and we have written about its other faces: why mutation testing AI generated tests exposes what coverage hides, why six tests can pass while the spec was the bug, and why it pays to score the trajectory rather than the answer. The four-re-run rule in Decompile-Diverge exists for the same reason we argue you should run an agent five times before believing a pass rate.

The next decision

If you have AI-recovered source in a repository today, the question is not which decompiler to switch to. It is whether anything in your pipeline would notice a function that builds, passes review, reads beautifully and returns a different value on an input nobody tested. On the numbers above, roughly one in twenty functions that clear every shipped test will do exactly that, and on vulnerable code closer to one in twelve will lose the crash you were hunting.

Decide what your oracle is before you decide what your model is. If you cannot write down a machine-checkable definition of correct for the thing you are recovering, the honest move is to use AI decompilation for orientation and naming only, and to keep a human and a differential test between its output and anything that ships. That is the judgement call we would help a team make on an AI engineering project, and it is cheaper to make now than after the code is merged.