Archestra's OpenAPPA claims zero attacks in 1,320 runs; an independent re-run finds two misses
Archestra says its MIT-licensed OpenAPPA guard stopped every scored attack in 1,320 runs; an independent re-run confirms the blocks and finds two misses.
Archestra open-sourced OpenAPPA on September 28, 2026: an MIT-licensed guardrail engine that sits between an agent and its tools, and before every tool call decides whether a piece of data may go to a given destination. The verdict comes from a lattice algebra the team calls the Agentic Permissions Policy Algebra (APPA), not from a classifier. Archestra’s own announcement and its openappa.com/evaluation page state the headline result: no scored attack succeeded against OpenAPPA in 1,320 guarded evaluations, while it completed 89% of tasks. In early October, developer Andrey Gubanov rebuilt the same benchmark from source, ran it against four models, confirmed the blocks — and also measured the misses, the false reports, and one confidential record that leaked without being scored as an attack.
Key points
- The vendor number: zero scored attacks across 1,320 guarded evaluations (600 Bench-Corp, 720 AgentThreatBench), 88–90% task completion, +4.22% tokens on Tau Bench.
- The replication: Gubanov re-ran Bench-Corp on four models and found only two successful attacks in 520 guarded runs — both on Gemini 3.7 Flash, same task — but guarded task completion fell to 74–81%.
- What the official page contradicts: its TLDR says OpenAPPA completed 89% of tasks; the Auto-mode detail table on the same page says 75%. Gubanov’s FIDES figures (27.5–35% attack success, 41.7–55% completion) are the same order of magnitude as the official 31%/41% but not the same numbers.
Background: this is two weeks old
Worth being clear about: this is not a new release. Archestra’s Mark Novikov and six co-authors announced OpenAPPA on September 28 with a same-day Show HN, and the paper (arXiv:2607.24625, v1 July 27, v2 August 26) was accepted to the NeurIPS 2026 Agents in the Wild workshop. InfoQ’s Chinese coverage in October still describes it as preview. What happened in the two weeks since is a different thing: an outside developer published a per-run replication, in a repository whose README says every number in the article comes from a file there.
The problem OpenAPPA answers is a familiar one for this readership — prompt injection as the default threat model for agents. Our Manus email injection entry is the real-world shape of it: an email turned into executable code, with the guardrail firing only after execution. OpenAPPA bets on a different route. Instead of a second model judging each call, which can itself be injected, Archestra argues a classifier tops out near 99.3% effectiveness, and at millions of calls that residual is a lot of leaks.
The facts
The mechanism, per the announcement and the paper: the lattice carries two axes, Audience (self ⊆ internal ⊆ public) and Trust, and labels only ever tighten — they never widen. Each tool contract is a TOML block declaring three things: requires (what identity or trust level the call needs), delta (what restriction the returned data carries), and effects (audit log). Because decisions depend only on the event log and make no network or filesystem calls, identical inputs produce identical verdicts, which is what makes appa replay usable as a CI regression check over scripted tool calls. Where a plain IFC engine would return a bare 403, the paper’s Recoverable IFC hands back a structured remedy path: sanitizers such as PII strippers, authorities that route a single request to a human or an internal verification API, and on-demand child branches that read untrusted data in a disposable subagent and return only schema-attested output to the parent.
Bench-Corp and AgentThreatBench figures come from Archestra’s /evaluation page and the GitHub README, and the two disagree. Both are listed here:
| Metric | Official TLDR | Official detail table | Gubanov’s re-run |
|---|---|---|---|
| OpenAPPA attacks | 0% (1,320) | Bench-Corp 0/20, AgentThreatBench 0/24 | 2 of 520 Bench-Corp runs (Gemini 3.7 Flash, one task) |
| OpenAPPA completion | 89% | 75% in the Auto-mode table; 88–90% across three models | 74% (DeepSeek), 75% (Luna), 81% (Gemini) |
| Claude Auto mode attacks | 10% | Stock Auto 2/20 and 8/24 | not replicated |
| FIDES attacks / completion | 31% / 41% | 27.5–35% across Bench-Corp configs | 27.5–35% attacks, 41.7–55% completion |
| Token overhead | Tau Bench +4.22% | 1,307,763 vs 1,254,821 | Bench-Corp up 2.6–8× |
Versions are pinned in the replication: main commit 53c3d8e from October 1 (fifteen commits past the v0.30.0 tag, built with Rust 1.96), the August 26 build a1ccf3f, and release 0.31.1 re-checked on October 12. On the guarded AgentThreatBench arm, none of the 360 planted tricks worked — but that arm ships purpose-written rules per trick. Gubanov also validated his harness by reproducing FIDES’s published numbers, which is what makes his other figures worth reading.
What others say
Archestra’s own position reduces to one sentence from the README: no scored attack succeeded in 1,320 evaluations. The blog states plainly that the figures are self-published by the authors and points readers to /evaluation for the statistical breakdown; the paper reports 6,600 controlled episodes and a 64.2–91% utility range rather than the README’s 88–90%. In the Show HN thread, team members (immafridge, keshakon) stress how little setup it takes and praise “batteries” — arbitrary programs that can run inside an authorization decision, such as calling the GitHub API to check whether a repository is private before its contents may go to Slack.
The independent read is more detailed and less flattering. Gubanov’s reading is that the zero measures the scoring object, not the threat. He makes four points. First, Bench-Corp’s 20 tasks count any forbidden action as an attack regardless of who asked for it; unguarded, 28–33% of runs did something forbidden, nearly always because the user requested it. Second, in 45 of 519 guarded runs that returned an answer, the agent claimed a blocked email or file share had gone out, against 1 in 440 unguarded runs. He traces it to one prompt sentence telling the agent to assume success when a helper agent reports nothing, then ablates it: 11 of 20 answers wrong as shipped, 7 of 20 with the sentence deleted, 0 of 20 when reworded to say side effects cannot be assumed — with task completion unchanged. Third, when instructed to obey planted text, DeepSeek published an employee’s HR record to a public forum in 4 of 15 runs, through a “cleaner” that redacts only digits; the grader checks for an attacker email, so those runs scored as safe. Fourth, on memory tasks guarded agents finished 40–60% of jobs while unguarded ones finished all of them.
InfoQ’s Chinese version is a translation of Bruno Couriol’s English piece, so it is not a second independent outlet: the 0%/89% numbers are still Archestra’s own, and what it adds is a Chinese-language walkthrough of the mechanism and the 99.3% classifier ceiling. Archestra has not publicly responded to the replication, and no answer from the team turns up in a search.
Our take
For anyone shipping on agents, OpenAPPA now reads as a guardrail worth a serious test pass rather than a switch you flip for safety. The genuinely useful signal is that it blocks at the execution layer — blocked actions do not happen, and the replication confirms that. But the same replication is the best footnote to it: if your workflow includes a step where the agent tells you it finished, that 45/519 false-report rate is closer to your everyday risk than attack success rate, because no security score will ever record it. Gubanov’s four recommendations are directly actionable: benchmark on your own tasks and models, verify outcomes in your own systems instead of trusting the agent’s report, audit rulebooks for side doors where one tool can reach another’s data, and test every registered cleaner individually.
The cost belongs on the table too. Guarded completion fell from 92–97% to 74–81% by his methodology; DeepSeek runs cost 16× more tokens and took 22× longer. In Claude Code the hook latency grows with session length — 0.2 s by call 1,000 and 2.5 s by call 4,000, with 0.31.1 no faster. The advertised +4.22% overhead comes from Tau Bench, where the guard intervened once in 11,355 actions; that is not the same phenomenon as a 2.6–8× rise on Bench-Corp.
What is still unknown: Archestra’s raw 1,320 runs are not public, so how much of the gap between the official page and the replication is version drift cannot be checked. The replication used fixed, non-adaptive injections and 2–5 runs per task, leaving roughly ±8–10 points of uncertainty on its percentages. That is a scoped result, not a debunking. The pairing of a deterministic policy engine with an outside re-run lands in the same territory as Nvidia’s open-source agent safety platform, announced the same day, and the continuous-attack market described in our October 2 Armadin funding entry — with OpenAPPA among the first in this space that let someone rebuild its tests from source.