Guard models vs LLM judges: the cost and latency trade-off

An LLM judge is a general-purpose language model given the action, the context and a rubric, and asked for a decision. It needs no training, can take any policy written in plain language, and can explain its reasoning.

A guard model is a small classifier trained on labelled examples of the specific question. MCP Guard is one: a DeBERTa-v3-base model (184M parameters) fine-tuned on about 31,000 labelled agent actions, which answers nine questions about a tool call (safe, violation type, severity, destructive, exfiltration, injected, approval policy, blast radius, argument grounding) in one forward pass. It cannot follow a new rubric you write tomorrow; it answers the questions it was trained on, quickly.

The trade-off between them is mostly about where the checks run and how many there are.

Figures for a judge depend heavily on the model and the prompt length; the ones below are an illustrative reference point, not a benchmark.

On a CPU (8 threads) the same full check takes around 120 ms, which is why the model can also be run close to where agents run.

Latency matters more for agents than it first appears. A coding agent that runs 40 shell commands per task, each gated by a one-second judge, spends 40 seconds waiting on its guard. In code mode, where generated code can fire many tool calls in a loop, per-call judge latency either slows the program down badly or pushes teams to skip the check.

A small guard is not a replacement for a judge in every case. A judge tends to be the better tool when:

• The domain is new. A guard generalises from its training data; a judge can apply a written policy to situations nobody labelled.
• The input is long. Reviewing a whole document, a long diff or a multi-page contract fits a large context window better than a compact classifier.
• You need an explanation someone will read. A rationale for an auditor or an end user is what language models are good at.
• The policy changes often. Editing a prompt is faster than collecting labels.

Judges have their own failure modes. Zheng et al. (2023), studying LLMs as judges of other models' answers, documented position bias, verbosity bias and self-enhancement bias, and a judge reading untrusted content can itself be a target for prompt injection. Neither approach is a guarantee.

The two approaches combine naturally. Run the fast guard on every tool call, and escalate only what it is unsure about:

1. The guard checks every action.
2. allow runs immediately; block stops immediately with the reason sent back to the agent.
3. ask goes to a slower reviewer: an LLM judge with your full policy, a human, or the judge first and a human if the judge also hesitates.

result = guard.check(action=call, intent=task, trigger=trigger)
if result["verdict"] == "allow":
    run(call)
elif result["verdict"] == "block":
    refuse(call, result["reasons"])
else:  # "ask": escalate only the uncertain band
    decision = llm_judge(call, policy=POLICY) or ask_human(call, result["reasons"])

The cost of the cascade is dominated by how many calls land in the ask band, which is something you control with thresholds (see calibrated scores and thresholds). The judge only sees the calls where its flexibility is worth paying for.

On accuracy: on R-Judge, a public benchmark of agent safety trajectories (held-out test half, 304 trajectories), MCP Guard reached an AUROC of 0.855, against 0.824 for saroku-guard, an open 184M guard model. The difference is not yet statistically significant, so read it as "on par with or better than" that baseline, not as a ranking. We have not published a head-to-head against specific LLM judges; if you run one on your own traffic, the cascade above is a good way to compare them where it matters.