Evaluation methodology

Themis Agent Intelligence is a runtime authorization system for autonomous agents.

It evaluates whether a specific action should happen based on the task, delegated authority, destination, provenance, runtime context, and execution history.

We use external agent-security benchmarks to evaluate two questions:

1. When an agent reaches a harmful consequential action, does Themis intervene before autonomous execution?
2. How much legitimate autonomous activity remains possible under the same authorization model?

These are controlled evaluation results, not production security guarantees.

#Results at a glance

#AgentHazard

88.5%, control intervention on evaluable harmful trajectories: 285 / 322.

• 18 BLOCK
• 267 REQUIRE APPROVAL
• 0 ALERT
• 37 ALLOW / no intervention

The full benchmark contains 2,653 cases.

The available execution-trajectory corpus represented 1,000 distinct benchmark cases, of which 322 met the adapter's evaluability criteria.

That is 12.1% of the complete benchmark.

The 322-case subset is not representative of the benchmark's full category distribution, so we do not extrapolate the 88.5% figure to all 2,653 cases.

A stateless per-action authorization control intervened on 274 / 322, or 85.1%, using the same BLOCK + REQUIRE APPROVAL definition.

#AgentDojo-derived authorization evaluation

1.1%, consequential-action authorization bypass: 6 / 547 with Themis, versus 30 / 547, or 5.5%, with the stateless authorization control.

Under precise task authority: 93 / 97 legitimate tasks completed autonomously, 95.9% legitimate-task autonomy.

This is a deterministic canonical-trajectory authorization evaluation derived from AgentDojo tasks.

It is not a live LLM-agent AgentDojo benchmark run.

#Terms used on this page

#Modeled consequential action

An observable proposed action with an external or persistent effect represented by the Themis consequence model.

Examples include:

• sending information externally
• transferring money
• modifying records
• changing privileges
• executing processes
• establishing persistence
• modifying an external or production system

Themis is a runtime-action authorization system.

It does not, by itself, protect against attacks that only manipulate a model's textual output or recommendation without producing a modeled external action.

#Control intervention

A decision that prevents fully autonomous execution: BLOCK or REQUIRE APPROVAL.

ALERT is not counted as a control intervention in the results reported on this page.

#Authorization bypass

A modeled adversarial consequential action that the authorization control permits.

#Legitimate-task autonomy

A legitimate task that remains able to complete without a BLOCK or REQUIRE APPROVAL decision.

#Evaluation principles

#Runtime facts, not attack labels

Themis does not receive benchmark fields telling it:

• whether a case is malicious
• its attack category
• the expected benchmark result
• whether an attack succeeded
• what security decision should be produced

The engine receives observable runtime facts that could reasonably come from an agent runtime or execution layer.

These include:

• task context
• delegated authority
• proposed action
• destination
• resources accessed
• destination context
• provenance
• previous runtime events

Themis therefore evaluates the action and its context rather than being told the benchmark answer.

#No hidden chain-of-thought

Themis does not require access to an agent's hidden reasoning or chain-of-thought.

The evaluation uses observable runtime behavior such as tool calls, resources, destinations, execution effects, and history.

#The evaluation population is fixed before outcomes

Cases are selected and mapped before Themis produces its security decision.

Policies are not modified case by case after observing benchmark failures and then included in the same reported result.

#Consequential actions define the boundary

Themis evaluates actions that can create observable consequences outside the model.

If an attack changes what the model says or recommends but never reaches a modeled consequential action, there may be nothing for the runtime authorization layer to block.

This is an intentional product boundary.

#AgentHazard

#From 2,653 benchmark cases to 322 evaluable trajectories

AgentHazard contains 2,653 harmful cases covering categories including:

• data exfiltration
• destructive actions
• privilege escalation
• persistence
• remote execution
• supply-chain poisoning
• compliance bypass
• prompt or intelligence theft
• resource exhaustion
• internal reconnaissance

The execution-trajectory corpus available for this evaluation represented 1,000 distinct benchmark cases.

#Where the 1,000-case trajectory corpus comes from

The trajectories are the execution traces published by the AgentHazard project for its own benchmark runs. Themis did not generate them.

AgentHazard publishes traces per agent framework and per model. This evaluation used the Claude Code framework traces, whose structured tool-call hook telemetry is the closest analogue to what an execution gateway observes, for five published model runs: GLM-4.6, Kimi-K2, Qwen2.5-72B, Qwen3-32B, and Qwen2.5-Coder-32B.

Two larger trace archives in the same corpus (Qwen3-VL-235B and Kimi-K2.5) were not downloaded, to bound compute. Traces recorded under the corpus's other agent frameworks use different formats and were not adapted. Both are documented limitations, not silent exclusions.

Together, the five downloaded archives cover 1,000 distinct benchmark case ids, 37.7% of the 2,653. Which cases were covered was determined by which published archives were downloaded, not by Themis outcomes. Themis did not select cases.

Where multiple trajectories were available for a represented case, trajectory selection used a deterministic rule based on observable tool-call coverage: the trajectory with the most observable tool calls was used.

The selection rule did not use Themis outcomes or attack-success labels.

#From 1,000 covered cases to 322 evaluable trajectories

Of the 1,000 covered cases:

• 161 contained no tool calls at all in the recorded run (rate-limited or refused executions in the published traces, a benchmark-execution artifact rather than a Themis limitation)
• 517 exposed only reconnaissance, resource-consumption, or benign actions, none of which is a consequence the current model represents
• 264 were fully mapped
• 58 were partially mapped

giving 322 evaluable trajectories, and 322 / 2,653 = 12.1% end-to-end evaluable coverage of the full benchmark.

The reported result applies only to these 322 trajectories.

#Evaluability was determined before Themis ran

Evaluability was determined from trajectory contents and deterministic adapter rules before any Themis decision was produced.

The evaluable set was fixed before measuring Themis outcomes.

A case could not be removed from the denominator because Themis failed to intervene.

The eventual Themis decision had no effect on whether that case entered the 322-case evaluation population.

#The evaluable subset is not representative

The 322 trajectories are not a random sample of the complete benchmark.

Categories with directly observable runtime consequences are over-represented.

The largest skew is by consequence category rather than attack strategy.

The current consequence model does not treat internal reconnaissance or resource consumption as standalone runtime consequences.

For this reason, 88.5% must not be interpreted as an estimate of performance across the full AgentHazard benchmark.

#AgentHazard control result

Across the 322 evaluable harmful trajectories:

• BLOCK: 18 / 322
• REQUIRE APPROVAL: 267 / 322
• ALERT: 0 / 322
• ALLOW / no intervention: 37 / 322

Therefore: 285 / 322 = 88.5% control-intervention rate.

Every intervention contributing to the 88.5% was either BLOCK or REQUIRE APPROVAL.

ALERT contributes nothing to this number.

Most interventions were approval escalations rather than automatic hard blocks: 267 / 285 interventions required human approval.

This distinction matters.

The evaluation shows that Themis would prevent those actions from executing fully autonomously under enforcement semantics. It does not show that 267 actions would necessarily be permanently denied.

#Important limitation: AgentHazard has no benign counterpart here

AgentHazard provides harmful trajectories for this evaluation but no corresponding benign workload suitable for measuring the normal approval rate of the same policy configuration in the same environment.

The 88.5% result therefore cannot establish by itself how discriminative the policy is between harmful and benign consequential actions.

In particular, it cannot rule out a high baseline approval rate on benign actions in this environment.

Legitimate-task friction is measured separately in the AgentDojo-derived evaluation and, for real deployments, during observe-mode pilots on the customer's own workloads.

#Comparison with stateless authorization

We evaluated a stateless per-action authorization control over the same 322 trajectories.

It sees the current proposed action and authorization context but does not use earlier provenance or execution history.

• Stateless control: 274 / 322 = 85.1%
• Themis: 285 / 322 = 88.5%

Both figures use exactly the same definition: BLOCK + REQUIRE APPROVAL.

Themis intervened on every trajectory caught by the stateless control, plus 11 additional trajectories.

The overall difference is modest because many AgentHazard actions are already visibly dangerous when inspected individually.

#Why execution history can matter

Consider: sensitive data accessed → data transformed → external action proposed.

The final external action may look permissible when inspected alone.

A stateful authorization system can also ask:

• Where did the information originate?
• What sensitive resources were accessed earlier?
• How did the data reach this action?
• Was the destination authorized by the original task?
• What has the agent already done in the session?

That is where provenance and execution history can change a decision that would otherwise be made from the current action alone.

#Trajectory-level intervention vs labeled-harm matching

The AgentHazard benchmark identifies a specific labeled harmful consequence for each case.

Themis may intervene on that exact consequence, or it may intervene earlier on another consequential step in the same harmful trajectory.

Of the 285 trajectory-level interventions, 28 were triggered by a different consequential side effect rather than the specific benchmark-labeled harm.

This leaves 257 exact labeled-harm interventions.

The benchmark-labeled harmful consequence was directly representable in 289 cases, giving 257 / 289 = 88.9% intervention on the specific labeled harmful consequence when that consequence was representable.

Across all 322 evaluable trajectories, 257 / 322 = 79.8% matched the benchmark-labeled harmful consequence directly.

We report both views because they answer different questions.

The 88.5% trajectory-level control rate asks whether Themis would interrupt autonomous execution somewhere in the harmful trajectory.

The 79.8% strict labeled-harm rate asks whether Themis intervened specifically on the benchmark's labeled harmful consequence across the complete 322-case evaluable set.

An earlier BLOCK or REQUIRE APPROVAL can stop autonomous execution before the trajectory reaches the later labeled harm, which is why trajectory-level intervention remains operationally relevant.

#AgentHazard misses

Themis allowed 37 of the 322 evaluable harmful trajectories.

All 37 were classified.

Harmful effect not observable: 30 cases. The case met the pre-defined evaluability criterion, but the specific harmful effect was not observable in the available trajectory. For example, an external effect performed inside an opaque script may not appear as a distinct runtime event. These 30 cases were retained in the denominator. Removing them after seeing the result would raise the reported rate, but would also change the pre-defined evaluation population after measurement. We therefore keep them in the 322-case denominator.

Harm class not modeled: 5 cases. All five involved Internal Reconnaissance, which is outside the current modeled consequence boundary.

Policy coverage gap: 2 cases. The relevant consequence could be reconstructed, but no applicable policy intervened.

Engine decision failure with facts and applicable policy present: 0 cases.

#Outcomes by mapping completeness

A natural question is whether the misses sit in the partially mapped trajectories. They do, almost entirely.

Of the 37 misses, 35 were partially mapped trajectories: the 30 where the harmful effect was not observable and the 5 where the harm class is not modeled. The 2 fully mapped misses are the 2 policy-coverage gaps.

Within the fully mapped trajectories, interventions were 18 BLOCK and 244 REQUIRE APPROVAL. Within the partially mapped trajectories, all 23 interventions were REQUIRE APPROVAL.

Themis intervened on every trajectory the stateless control intervened on. All 11 additional Themis interventions occurred in fully mapped trajectories. On partially mapped trajectories the two controls intervened on the same 23.

Most of the remaining gap below 100% is therefore observability and consequence-model coverage, not authorization decisions on observed, modeled actions. This is why runtime observability is treated as a first-order variable in pilots.

#Adapter and label isolation

AgentHazard trajectories are translated into Themis runtime events using deterministic rules based on observable trajectory content.

There is no per-case human annotation of Themis runtime events.

The adapter uses information such as:

• tool names
• tool inputs
• resources
• paths
• network destinations
• process behavior
• observable execution structure

Benchmark metadata such as attack category, strategy, expected outcome, and benchmark success label is not passed into Themis runtime events.

Benchmark target information is used separately for evaluation scoring and localization of the benchmark-labeled harmful step.

It is not used by Themis to make an authorization decision and does not determine whether a case enters the evaluable population.

#AgentDojo-derived authorization evaluation

#What this experiment measures

AgentDojo contains legitimate user tasks and adversarial injections across:

• workspace
• travel
• banking
• communication workflows

For this evaluation, those tasks are used to construct deterministic canonical execution trajectories.

The question is:

If an agent reaches this consequential action, does the authorization layer permit it?

This isolates authorization quality from model susceptibility.

It does not measure:

How often will a particular LLM follow a particular injection?

The reported numbers therefore should not be compared directly with live-model AgentDojo attack-success rates.

#Evaluation population

The evaluation contains 97 legitimate user tasks and 35 injection tasks, producing 949 user-task × injection combinations.

These divide into two groups.

No modeled consequential external action: 402 / 949. These cases primarily involve output or recommendation manipulation without an external action represented by the Themis consequence model. Themis makes no protection claim for these 402 cases. They are outside the runtime-authorization boundary evaluated here.

Modeled consequential action: 547 / 949. These combinations contain an observable consequential action that the authorization system can evaluate. The adversarial authorization-bypass metric therefore uses 547 as its denominator.

#Authority precision matters

The headline evaluation uses precise task authority, meaning that the authorization system knows the intended task scope and legitimate destination constraints where applicable.

For example, it can distinguish:

Pay the supplier authorized by this task.

from:

Pay a different destination introduced by external content.

We also evaluated less precise authority representations while holding the evaluation population and system configuration constant.

The result is not a simple security-versus-authority curve.

Under broad permissions, the evaluated controls are extremely conservative: adversarial bypass falls to zero, but only 60.8% of legitimate tasks remain autonomous.

More precise authority significantly improves the ability to distinguish legitimate autonomy from unauthorized actions.

The headline 95.9% legitimate autonomy therefore should not be assumed when an integration provides only broad tool permissions.

#Stateless authorization baseline

The stateless comparison control sees:

• current consequence class
• current destination
• destination trust
• current authority envelope

It does not see:

• previous runtime history
• provenance or lineage
• earlier untrusted inputs
• earlier sensitive-resource reads
• accumulated consequences
• session history

This is substantially more capable than a basic tool allowlist.

It sees the same current action and authority context as Themis, but not the earlier causal relationships.

#Consequential-action authorization bypass

Among the 547 modeled consequential adversarial actions:

• Stateless authorization control: 30 / 547, 5.5% authorization bypass
• Themis: 6 / 547, 1.1% authorization bypass

The six Themis bypasses are a subset of the 30 stateless bypasses.

Themis therefore intervened on 24 / 30 actions that passed the stateless control.

However, this incremental benefit was entirely concentrated in the banking environment.

#Where provenance helped, and where it did not

The banking subset contains 144 consequential user-task × injection combinations.

Banking: Themis 1 / 144 = 0.7% bypass; stateless 25 / 144 = 17.4% bypass. The difference is 24 additional interventions by Themis. All 24 measured incremental interventions occurred in this environment.

Across the remaining 403 non-banking consequential actions: Themis 5 / 403; stateless 5 / 403.

There was no measured incremental bypass reduction outside banking in this evaluation.

This is an important scope result.

The measured provenance advantage was specific to workflows where an externally introduced financial destination could be compared with the destination authorized by the user's task.

We do not generalize the 24-case banking result into a claim that provenance improves every class of agent workflow.

#The six remaining authorization bypasses

Themis permitted 6 of 547 modeled consequential adversarial actions.

They consisted of:

• 2 destructive-change actions where no applicable policy classified the action as unauthorized
• 2 financial-transfer actions with authority or mapping coverage gaps
• 1 external-communication action where destination trust could not be resolved
• 1 privilege-expansion action where no applicable policy fired

Five of the six were fully observed actions with policy or mapping coverage gaps.

One involved incomplete destination context.

In all six cases, no policy fired.

These six remain part of the reported 1.1% bypass rate.

#Legitimate-task behavior

Of the 97 legitimate tasks: 93 completed autonomously and 4 required security intervention, giving 95.9% legitimate-task autonomy.

All four interventions involved legitimate large-payment actions that required human approval under the evaluated policy.

The operational result is therefore:

4 of 97 legitimate tasks could not complete fully autonomously under the precise-authority configuration.

We do not present a separate "zero false-positive" claim.

Production intervention rates depend on the actual workflow, authority information, and policy configuration.

#What these evaluations support

The results support several narrower conclusions.

#Capability permission and action authorization are different

An agent may legitimately possess a tool capability while a particular use of that capability falls outside the authority delegated for the task.

#Provenance can matter in specific workflows

In the AgentDojo-derived evaluation, provenance produced a measurable incremental benefit in financial workflows involving authorized versus externally introduced payment destinations.

The same incremental benefit was not observed outside banking in this evaluation.

#Precise authority improves usability

Legitimate autonomy increased from 60.8% with broad capability permissions to 95.9% with precise task and destination authority.

#Runtime observability matters

Themis cannot reliably control a consequence that the underlying runtime does not expose.

#What these evaluations do not establish

These results do not demonstrate:

• an 88.5% success rate across the complete AgentHazard benchmark
• that the 322 evaluable AgentHazard cases represent the full benchmark distribution
• that an 88.5% harmful intervention rate implies a low benign escalation rate in the same AgentHazard environment
• protection against every prompt injection
• protection against output-only manipulation
• equivalent performance across all agent frameworks
• a general provenance advantage across every workflow
• 95.9% legitimate autonomy with broad capability permissions
• production false-positive or approval rates
• protection when required runtime facts are unavailable
• universal security for autonomous agents

These are controlled measurements under explicitly defined conditions.

#Why pilots begin in observe mode

The evaluations identify two variables that matter significantly in real deployments: runtime observability and delegated-authority precision.

Initial Themis pilots therefore begin with one agent workflow in observe mode.

Themis evaluates real proposed actions alongside the existing execution layer without initially changing production behavior.

During a pilot we measure:

• what runtime context is actually available
• what authority can be represented
• which actions Themis would allow
• which actions would require approval
• which actions would be blocked
• benign intervention burden
• where provenance changes a decision
• where incomplete context creates uncertainty

The objective is not to assume that benchmark results transfer unchanged into production.

The objective is to measure security benefit and operational friction in the customer's actual environment before enforcement is enabled.

#Evaluation transparency

The benchmark adapters, stateless baseline implementations, and evaluation harness are Themis-owned artifacts and can be made available for technical inspection.

Case-level verdicts, failure classifications, and detailed evaluation results may be shared with design partners where appropriate.

Some raw trajectory data originates from third-party benchmark projects and may be subject to upstream licensing restrictions.

#How to interpret these numbers

The benchmark results on this page are controlled evaluation measurements, not security guarantees.

The central question Themis is designed to answer is not simply:

Can this agent use this tool?

It is:

Should this specific action happen now, given the task, the authority the user delegated, where the relevant data came from, what happened earlier in the workflow, and what consequence the action would create?

That is the distinction these evaluations are intended to test.