Calibrated scores and thresholds: why per-deployment calibration matters

A classifier is calibrated when its probabilities match observed frequencies: of all the actions it scores at about 0.3 unsafe, about 30% really are unsafe. Calibration is separate from ranking. A model can rank actions well (unsafe ones usually score higher than safe ones) and still be badly calibrated (its 0.9s are right only 60% of the time).

The usual summary of calibration is expected calibration error (ECE): bin predictions by confidence, compare each bin's average confidence with its actual accuracy, and take the weighted average gap. Lower is better. Guo et al. (2017) showed that modern neural networks are often overconfident, and that a simple post-hoc fix, temperature scaling (dividing the logits by a single fitted constant before the softmax), often improves calibration substantially without changing the ranking.

Why this matters for a guard: every threshold you set is a statement about probability. "Ask a human above 0.3" only means what you think it means if 0.3 is a real 30%.

There is no correct threshold in the abstract. There is a cost of letting an unsafe action run and a cost of interrupting a safe one (a human's time, a stalled agent, a frustrated user), and the threshold is where you balance them.

MCP Guard's default verdict reflects a reasonable balance for general use:

• block if P(unsafe) is at least 0.8, or the approval_policy head says reject;
• ask if P(unsafe) is at least 0.3, the policy head says require_human, or args_grounded is below 0.5;
• allow otherwise.

Every response carries p_unsafe and the per-head scores, so you can apply your own thresholds instead. A support agent that can only read the knowledge base can tolerate a higher ask threshold than an infrastructure agent that can delete clusters. Different tools in the same deployment can reasonably use different thresholds, too.

Calibration is a property of a model on a distribution. Change the distribution and the calibration can change. Ovadia et al. (2019) evaluated predictive uncertainty under dataset shift and found that calibration degrades as data moves away from the training distribution, for many methods, including ones that were well calibrated in distribution.

A guard sees exactly this kind of shift. It is trained on a mix of public agent-safety data and labelled tool calls; your agents have their own tools, argument formats, prompts and users. We have seen it in our own evaluation: thresholds fitted on one dataset lost precision noticeably when applied to R-Judge, a public benchmark of agent safety trajectories, for our model and for the open baseline we compare against. Ranking held up much better than the thresholds did.

The practical conclusion is simple: treat the default thresholds as a starting point, not a guarantee, and fit your own on your own traffic. This is also why a benchmark result like AUROC is necessary but not sufficient. AUROC measures ranking across all thresholds at once; it says nothing about calibration or about where to put your threshold.

You do not need a research team for this. You need a few hundred labelled examples of your own actions.

1. Log checks in shadow mode. Call the guard on every action but do not act on the verdict yet. Keep the action, the scores and what happened. (MCP Guard does not store request content, so keep this log on your side.)
2. Label a sample. A few hundred actions, labelled by someone who knows the system: should this have run, needed a human, or been stopped? Include the approvals and overrides from your existing review process; they are labels you already have.
3. Pick thresholds for a target precision. Decide what you need, for example "at least 95% of allowed actions are truly safe" and "at least 95% of blocked actions are truly unsafe", and find the thresholds on p_unsafe that meet them on your sample. Whatever falls between becomes the ask band.
4. Check the ask band's size. If it is too wide for your reviewers, you have learned something real about the trade-off, and it is better to know before go-live.
5. Re-check after changes. New tools, new system prompts and new user groups all shift the distribution. Re-run a smaller labelled sample after each significant change.

A small script is enough:

import numpy as np

def threshold_for_precision(p_unsafe, unsafe, target=0.95, side="block"):
    """Lowest block threshold (or highest allow threshold) meeting the target precision."""
    p, y = np.asarray(p_unsafe), np.asarray(unsafe, dtype=bool)
    grid = np.unique(p)
    if side == "block":
        for t in grid:
            sel = p >= t
            if sel.any() and y[sel].mean() >= target:
                return t
    else:
        for t in grid[::-1]:
            sel = p < t
            if sel.any() and (~y[sel]).mean() >= target:
                return t
    return None

With few examples, prefer conservative thresholds and a wider ask band; the uncertainty in your own estimate is real.