Glossary
AI agent safety glossary, A to Z
29 terms, each defined in plain language with links to the deeper explainers.
Checks run on our GPUs in Switzerland. Request payloads are not stored.Security
29 pages · press / to search
A
- Policy & controlsAllowlist and denylistDeterministic rules that permit only listed tools or patterns (allowlist) or forbid listed ones (denylist), regardless of any model’s judgement.Read
- Policy & controlsApproval fatigueWhen reviewers see so many approval requests that they stop reading them and approve by reflex, defeating the purpose of review.Read
- EvaluationAUROCArea under the ROC curve: the probability that a randomly chosen positive example is scored higher than a randomly chosen negative one.Read
B
C
- EvaluationCalibrationA model is calibrated when its predicted probabilities match observed frequencies: of actions scored 0.7 unsafe, about 70% really are.Read
- Agents & MCPCode modeAn agent pattern in which the model writes code that calls tools through an API inside a sandbox, instead of issuing one tool call at a time.Read
D
- ThreatsData exfiltrationMoving private data, secrets or another tenant’s data to a place it should not go, such as an external email address, URL or public repository.Read
- Policy & controlsDestructive actionAn action that deletes, overwrites or irreversibly changes data, such as dropping a table, force-pushing or wiping a directory.Read
- EvaluationDistribution shiftWhen the data a model sees in use differs from the data it was trained or calibrated on, so its accuracy and calibration may change.Read
E
F
G
H
- ThreatsHallucinated argumentsTool-call arguments the model invented rather than took from the user or the data, such as a made-up id, amount or recipient.Read
- Policy & controlsHuman in the loopA design in which a person reviews and approves, edits or rejects certain agent actions before they take effect.Read
I
J
L
- Policy & controlsLeast privilegeThe principle that each component, including an AI agent, should have only the permissions its task requires, and no more.Read
- ThreatsLethal trifectaSimon Willison’s term for an agent that combines access to private data, exposure to untrusted content and the ability to communicate externally.Read
- EvaluationLLM-as-a-judgeUsing a general-purpose language model, prompted with criteria, to evaluate another model’s output or action.Read
M
- Agents & MCPMCP gatewayA proxy between MCP clients and MCP servers that centralises authentication, routing, logging and policy for tool calls.Read
- Agents & MCPModel Context Protocol (MCP)An open protocol, introduced by Anthropic in November 2024, that standardises how AI applications connect to external tools and data sources.Read
P
- Agents & MCPPre-execution guardA check that runs on an AI agent’s proposed tool call before the call executes, and decides whether to allow it, ask a human, or block it.Read
- ThreatsPrompt injectionAn attack in which text supplied to a language model contains instructions that override or subvert what the developer or user intended.Read
R
S
T
- EvaluationThresholdThe score cut-off at which a guard’s probability turns into an action, such as asking a human when P(unsafe) is at least 0.3.Read
- Agents & MCPTool callA structured request from a language model to run a named function (a tool) with specific arguments, such as sending an email or querying a database.Read
- Agents & MCPTool proxyThe component that forwards an agent’s or sandboxed program’s tool calls to the real tools, and the natural place to check each call.Read