A single impressive number is a red flag, not a proof. So we run two studies that answer two different questions, and we report both together, because each one covers for what the other cannot see. One measures the verification engine alone. The other measures what happens when that engine sits inside an AI assistant writing real code.
Study 1The engine, on its own
The first question is narrow and mechanical: given a correct contract, does the verifier reach the right verdict? We took 136 real cases of SQL injection and cross-site scripting from CVEFixes, a dataset of genuine vulnerabilities and their fixes, and ran each vulnerable and patched pair through Acutis with an oracle contract.
The result was an F1 of 1.0. Every vulnerable case was blocked. Every patched case was allowed. No false positives, no false negatives, at a median latency under a millisecond.
It means the engine's decision procedure is sound and precise when the contract is correct. It does not mean Acutis will catch every vulnerability in the wild, because in the wild the contract comes from the AI, not an oracle. Study 1 isolates the engine from that variable on purpose. Study 2 puts it back.
Study 2The engine, inside the loop
The second question is the one that actually matters to a user: when an AI assistant writes code with Acutis in its loop, does less vulnerable code come out? We built 40 paired prompts, each asking an agent to implement a task with a plausible path to a taint-flow vulnerability. We ran each prompt twice, once without Acutis (control) and once with it (treatment), and had the output judged for vulnerabilities.
Vulnerability rate, control vs treatment
The vulnerability rate fell from 62.5% to 0%: a 62.5 percentage-point absolute reduction, with not a single case made worse. The paired design lets us use McNemar's test, which returned p = 5.96 × 10-8. That is the strongest single artifact we have, and it is the one we lead with.
The right way to read it is 62.5 points to zero, not a tidy percentage. The honest number carries its own baseline.
The method, in one tableBoth studies at a glance
| Study | Question | n | Result |
|---|---|---|---|
| 1 · Engine | Correct verdict given a correct contract? | 136 | F1 = 1.0 |
| 2 · End to end | Less vulnerable code out of the AI loop? | 40 | 62.5 → 0% |
The uncomfortable partWhere a skeptic should push
If we only showed the two numbers, we would be doing the thing we criticize. So here is what they do not settle, stated as plainly as we can.
The control rate is high because the task set was chosen to elicit vulnerabilities. These prompts were built to have a tempting unsafe path. A gentler task set would show a lower control rate and a smaller gap. The 62.5% is a property of the benchmark, not a claim about all code.
Study 2 used a single model and a single harness. One model's behavior is not every model's behavior. We have not yet run the cross-model matrix that would let us claim generality, and until we do, this is one data point with a very small p-value, not a law.
On easy benchmarks the effect can vanish. When the underlying model already defaults to safe patterns, the control rate collapses toward zero and there is nothing left for Acutis to prevent. We have seen exactly this, and it means the honest frontier is harder tasks and weaker or legacy-constrained workers, not easier ones.
And there is no production deployment at scale yet. No throughput numbers under real load, no long-running design partner. Everything above is a controlled result. We say so every time.
Because the audience that matters, the technical reviewer deciding whether to trust the engine, rewards the disclosure and punishes the omission. A number you can poke at is a number you can believe. The fastest way to lose a careful reader is to give them a result with no visible edges.
Two studies, two questions, and a standing invitation to attack both. The engine is sound where a contract is sound. The loop moves the needle hard on the tasks built to test it. Neither claim is the whole of security, and we would rather you knew the shape of what we are claiming than be impressed by a number with the edges sanded off.