A detection benchmark asks one question a customer cannot ask for themselves: across a large corpus where the ground truth is known, what does this tool actually catch, and what does it flag that it should not? The OWASP Benchmark is the public standard for that question. It is 2,740 compilable Java servlet test cases, half genuinely vulnerable and half deliberately safe, with an answer key. Tools have published scores on it for a decade.
SetupWhat we ran, exactly
Acutis verifies declared contracts; it is not a scanner you point at a directory. So on a detection benchmark it runs the way our engine study (Study 1) ran: a disclosed oracle-contract generator plays the role the AI assistant plays in production. It reads only the Java source of each case, never the answer key, and emits the sources, sinks, and transforms declaration the engine then formally checks. The generator's entire catalog, the exact contract it produced for every case, and every per-case verdict are published in acutis-eval; the answer key is consulted only at scoring time, and the run is deterministic: two full passes produce byte-identical results.
Of the 2,740 cases, 1,572 fall in the six taint-flow categories this run measures: command injection, SQL injection, XSS, path traversal, LDAP injection, and XPath injection. The remaining categories are a different species of finding. Weak cryptography, for one, is not a dataflow bug at all; it is a forbidden-choice bug, an algorithm a policy disallows, and Acutis addresses that class through a separate policy lane in which a deployment declares the attributes it forbids and the engine enforces them. That lane is configured per deployment, not exercised in this taint-flow benchmark, so those categories sit outside this scorecard by design, counted and named rather than quietly dropped.
Within the six categories measured, coverage is total: every one of the 1,572 cases receives a verdict. No unknowns, no skips. And because each case's contract declares dangerous sinks only for that case's own vulnerability class, a block can only register as a detection of the class the case is labeled for; there is no credit for flagging the wrong thing.
RecallEvery vulnerable case, blocked
| Category | Cases | True-positive rate | False-positive rate |
|---|---|---|---|
| Command injection | 251 | 1.00 | 0.89 |
| SQL injection | 504 | 1.00 | 0.88 |
| XSS | 455 | 1.00 | 0.59 |
| Path traversal | 268 | 1.00 | 0.93 |
| LDAP injection | 59 | 1.00 | 0.84 |
| XPath injection | 35 | 1.00 | 0.95 |
| Overall | 1,572 | 1.00 | 0.81 |
All 819 vulnerable cases block, in every category. For a tool whose entire design premise is that the failure mode must be over-blocking, never a silent miss, this is the number that has to be perfect, and it is.
And yes, the fourth column is high. Most benchmark write-ups would crop it out. It is the most informative part of the result, so we are going to spend the rest of the post on it.
PrecisionThe false-positive floor, decomposed
Of the 753 deliberately safe cases, Acutis over-blocks 611. But an aggregate FPR is only alarming while it is amorphous. Every single one of those 611 decomposes into one of four structural classes, with nothing left over:
| Count | Class | What it is |
|---|---|---|
| 308 | Opaque dead branches | The benchmark guards the safe path with compile-time constant arithmetic, like (7*42)-num > 200 ? safe : tainted. The tainted branch is dead, but no conservative taint analyzer evaluates constant arithmetic to prove it. A documented artifact of this benchmark. |
| 278 | Collection round-trips | A tainted value and a safe value are stored in the same map or list, and the safe one is read back. Acutis tracks taint per variable, deliberately without a per-element heap model, so the collection stays tainted. |
| 13 | Helper passthrough | The raw request object is passed into a local helper whose safe return is itself guarded by an opaque dead branch. The same floor, one call deeper. |
| 12 | Nested sinks, cross-file sanitizers | A declared sink's return value is treated as untrusted and feeds an outer sink, or the sanitizer lives in a separate file outside the local verification boundary. |
On the 154 safe cases that fall outside the two structural floor classes, the false-positive rate is 7.8%. The floors are not bugs to be fixed; they are the price of a verifier that is conservative, local, and rule-list-free by design. Constant-folding dead branches or modeling every collection element would buy points on this benchmark and nothing in production.
An aggregate FPR is a mood. A decomposed FPR is a specification of exactly what the tool refuses to assume.
The controlThe best free detector lands on the same floor
Here is the part that makes the floor claim testable rather than convenient. The benchmark repo ships official per-case results for well-known tools. FindBugs with the FindSecBugs plugin is the strongest free static analyzer in the benchmark's own published scorecards, and the comparison is not symmetric in its favor being here: FindSecBugs ran fully automatically on the compiled code, while Acutis's row required the disclosed contract generator described above. On the identical 1,572 cases, scored the same way, and cross-checked against the benchmark's own per-case pass and fail column with zero disagreements:
| Tool | TPR | FPR | cmdi FPR | ldapi FPR | xpathi FPR | xss FPR |
|---|---|---|---|---|---|---|
| Acutis (v7) | 1.00 | 0.81 | 0.89 | 0.84 | 0.95 | 0.59 |
| FindSecBugs (official) | 0.99 | 0.82 | 0.89 | 0.84 | 0.95 | 0.63 |
Category by category, the false-positive rates are nearly identical: 0.89 and 0.89 on command injection, 0.84 and 0.84 on LDAP, 0.95 and 0.95 on XPath. Two tools with completely different architectures, one a mature pattern-based scanner and one a contract verifier with no rule list at all, hit the same wall in the same places. That is what a benchmark property looks like. The difference between the rows is the first column: FindSecBugs misses vulnerable cases; Acutis does not.
The benchmark's anonymized commercial SAST scorecards (its v1.1 release, 13,399 in-scope cases) span true-positive rates from 0.40 to 0.95 at false-positive rates from 0.19 to 0.75. Lower false-positive rates are available in that set, but always by trading away recall; no published tool reaches 1.00. All of these numbers are computed from the per-case result files the benchmark itself distributes, with the comparison script published in acutis-eval so you can rerun the arithmetic yourself.
The distinctionLanding on the same floor is not being the same tool
It would be easy to read that table as a tie with a free tool, so let us be precise about what it does and does not say. A detection benchmark measures one thing: given a fixed blob of code, what does the tool flag. It is the single axis on which a scanner and a verifier are supposed to resemble each other, and on that axis, an old free scanner and a new verifier land near each other because the axis rewards recall and punishes the same synthetic safe-case tricks for everyone. What the benchmark cannot see is everything that separates the two.
FindSecBugs is an alarm on a developer's desk. Acutis is a gate inside the AI's loop. FindSecBugs runs after the fact, over a codebase, and hands a human a list. Acutis verifies a contract the AI declares about code it just wrote, and blocks in the loop, before a person ever sees the diff. The benchmark has no way to measure that difference, because it is a static pile of files with no author still in the room. The workflow result that difference produces, generated-vulnerability rate falling to zero inside the loop, lives in a different study entirely.
And that 0.82 the free scanner posts here, it posts on your real code too. A false-positive rate in the eighties is exactly why pattern scanners get switched off in week three. Acutis's 0.81 is not that: it is a benchmark artifact, made almost entirely of two constructions, dead branches guarded by constant arithmetic and values fished back out of shared collections, that this corpus is built from and real code is not. On a suite of ordinary benign tasks the terminal false-block rate was zero. So identical false-positive rates on this benchmark do not imply identical behavior in a codebase; one tool genuinely floods and the other does not. The number that looks like parity is measuring the one setting in which the two are least distinguishable.
Add the rest and the gap is not subtle: a name-blind engine that works on frameworks it has never seen, versus a rule list that covers what its authors enumerated; a machine-readable proof bundle an auditor can file, versus a severity label. None of that is on this scorecard, because none of it is what a detection benchmark measures. The honest reading of the table is not "as good as a free tool." It is "even on the one turf where a free tool competes, we do not miss, and that turf is the smallest part of what we are."
The uncomfortable partWhere a skeptic should push
The contracts came from an oracle generator, not an AI assistant. This is an engine-arm result, like our F1 study: it isolates the verifier from the quality of the declarer. In production the AI writes the declaration and iterates when Acutis blocks; our dogfooding studies measure that loop separately. The generator is published precisely so you can check it does not peek at the answers.
The benchmark is synthetic and from 2016. It measures dataflow reasoning over servlet-era Java, not modern framework code. Its value is the answer key and a decade of comparable published scores, not realism.
Perfect recall on a benchmark is not perfect recall in the wild. FindSecBugs also scores near-perfect recall here. What the benchmark shows is that the engine's dataflow reasoning has no holes this corpus can find, which brings us to the last point.
The benchmark audited us before we scored it. Getting to this result surfaced four genuine soundness bugs in our own parsers, cases where a switch statement, a mutated collection, or a transform on a tainted value could quietly drop taint. Each was fixed with paired regression tests, and we verified case by case that not one true positive in the final score depends on anything other than those fixes plus honest contract corrections. A benchmark that finds bugs in the tool being benchmarked, and gets credited for it in the write-up, is a benchmark being used honestly.
Because the reviewer who matters will compute it anyway. The benchmark's safe cases are built so that clearing them requires assumptions a sound verifier refuses to make. We would rather show you exactly which assumptions those are, and show an independent tool hitting the identical wall, than publish the one column that flatters us.
The generator catalog, the scoring script, the per-case contracts, the per-case verdicts, and the official-tool comparison script are public at github.com/Robbatron/acutis-eval. The engine itself is proprietary, so what the repo makes checkable is the instrument and the arithmetic: that the contracts never peek at the answers, that the scoring is strict, and that every number here follows from the published files. The claim is narrow and checkable: given honest declarations, the engine missed nothing this corpus knows how to hide, at millisecond-scale latency, with every refusal accounted for.