MCP Guard vs saroku-guard

saroku-guard is an open guard model, distributed through the saroku package on PyPI. It is a 184M-parameter DeBERTa model that scores a proposed agent action as safe or unsafe. It is small enough to run on a laptop CPU or a modest GPU, and it costs nothing to use.

MCP Guard's model is also DeBERTa-v3-base with 184M parameters. We fine-tuned it on about 31,000 labelled agent actions: public agent-safety data plus our own labelled benign and adversarial tool calls. Instead of one output, it has nine heads that answer, in the same forward pass:

• safe: probability the action is safe to run now
• violation: none, policy violation, scope violation, injection, goal drift or corrigibility
• severity, destructive, exfiltration, injected
• approval_policy: auto-approve, require a human, or reject
• blast_radius: read-only, local or reversible write, or production-mutating / external side effect
• args_grounded: whether the arguments are supported by what the user asked

R-Judge (Yuan et al., 2024) is a public benchmark of agent interaction records labelled for safety risk. We split it in half by trajectory, used one half only for model selection, and report the other half, 304 trajectories that were never used for any choice. Neither model was trained on R-Judge.

A paired bootstrap on those 304 trajectories gives an interval for the difference that includes zero. In plain terms: MCP Guard is on par with or better than saroku-guard on this benchmark, and 304 trajectories are too few to say more. We are building a larger hand-labelled test set to settle it, and we would rather you test both on your own traffic than rely on our numbers.

AUROC measures ranking, not calibration. Both models need thresholds fitted to your own traffic; see calibrated scores and thresholds.

• It runs locally. No network round trip, no account, and no data leaves your machine. For air-gapped environments, or when your policy forbids sending tool arguments to any third party, that settles it.
• It is free. You pay only for your own compute.
• It is slightly faster for its one question. In our measurement on the same RTX 4090, about 13 ms p50 at batch 1, against about 15 ms for MCP Guard's full nine-question check. Over the internet, MCP Guard also carries the network hop.
• It is open. You can inspect it, fine-tune it and pin a version indefinitely.

• Nine answers instead of one. A safe/unsafe score tells you whether to worry; the other heads tell you what to do. blast_radius and approval_policy map directly onto approval rules, args_grounded catches hallucinated arguments, and injected flags actions driven by a tool result rather than the user.
• A verdict out of the box. The API applies a default rule (block if P(unsafe) ≥ 0.8 or the policy head says reject; ask if P(unsafe) ≥ 0.3, it says require_human, or args_grounded < 0.5) and returns the raw scores so you can apply your own.
• Nothing to operate. No GPU to provision, no model to update. It is also an MCP server, so MCP clients can call it as a tool.
• Training data aimed at tool calls. Besides public agent-safety data, it includes our own labelled benign tool calls and adversarial ones with tampered arguments, so it sees ordinary work as well as attacks.

Choose saroku-guard if you must keep all data local, want zero cost, or only need a single safety score to gate a small set of tools.

Choose MCP Guard if you want the breakdown (blast radius, approval policy, grounded arguments, injection) to drive different handling per call, and you are comfortable with a hosted API that processes payloads in memory in Switzerland without storing them.

Either way, a guard model reduces risk; it does not guarantee safety. Keep least privilege and deterministic rules for known-bad patterns.