MCP Guard vs an LLM-as-a-judge

An LLM-as-a-judge setup sends the proposed tool call, the user's request and some context to a general-purpose model with a prompt such as "Should this action run? Answer allow, ask or block and explain why." It needs no training data and adapts to any policy you can write down in words. See the glossary entry.

MCP Guard is a small classifier (DeBERTa-v3-base, 184M parameters) fine-tuned on about 31,000 labelled agent actions. It reads the same kind of input and, in one forward pass, returns a verdict (allow, ask or block) and scores for nine fixed questions: whether the call is safe, the type of violation, severity, whether it is destructive, whether it exfiltrates data, whether it was injected, the approval policy, the blast radius and whether the arguments are grounded in the user's request.

The trade-off is the usual one between a general tool and a specialised one.

Agents make many tool calls, and in code mode a single turn can fire dozens. Whatever checks them runs on the hot path.

• An LLM judge costs roughly $2–5 per 1,000 calls and around one second per call. These are illustrative figures; they depend on the model, the prompt length and the provider.
• MCP Guard costs $0.20 per 1,000 checks. We measured about 15 ms per full nine-question check on an RTX 4090 (p95 about 23 ms, batch 1). Over the internet, add the network round trip, typically around 100 ms from Europe.

A second of latency per call is often acceptable for a handful of high-stakes actions and unacceptable for every file read in a coding agent. That difference, more than accuracy, is usually what decides the architecture.

LLM judges have documented biases. Zheng et al. (2023), in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", describe position bias, verbosity bias and self-enhancement bias when models grade answers. Tool-call judging is a different task, but the lesson carries over: a judge's output can shift with prompt wording, ordering and model version, and a free-text "I think this is fine" is hard to turn into a threshold.

A classifier gives the same scores for the same input, and its scores can be calibrated so that a 0.3 means something close to "about 30% of calls like this are unsafe". That is what makes thresholds and approval policies tunable. Calibration still needs checking on your own traffic; see calibrated scores and thresholds.

One more difference: an LLM judge reads the same untrusted text as the agent, so a well-crafted injection can try to talk the judge round too. A classifier does not follow instructions, which makes that attack harder, though not impossible.

You do not have to choose. A practical setup:

1. Deterministic rules first: hard denies for known-bad patterns, hard allows for known-safe read-only calls.
2. MCP Guard on everything else. Allow runs immediately; block is refused.
3. Ask goes to a slower reviewer: an LLM judge with a detailed policy prompt, or a human, depending on the stakes.

With the default verdict rule (block if P(unsafe) ≥ 0.8 or the policy head says reject; ask if P(unsafe) ≥ 0.3, the policy head says require_human, or args_grounded < 0.5), only the uncertain middle is escalated. How large that middle is depends on your traffic and on the thresholds you choose, so measure it on your own calls rather than assuming a number.

• Low volume, high stakes. If your agent makes a few dozen consequential calls a day, the cost and latency of a judge hardly matter, and its flexibility is worth having.
• Policies that change weekly or are long and specific. A judge reads a new policy immediately; the guard's nine questions are fixed at training time (you can pass constraints and context, but the questions do not change).
• You need a written rationale for each decision, for example for a reviewer queue.

Neither option guarantees safety. Both are layers on top of least privilege, sandboxing and deterministic rules.