Glossary
LLM-as-a-judge
Using a general-purpose language model, prompted with criteria, to evaluate another model’s output or action.
Checks run on our GPUs in Switzerland. Request payloads are not stored.Security
What LLM-as-a-judge means
LLM-as-a-judge means asking a capable language model to assess something, such as an answer’s quality or whether a proposed action is safe, by prompting it with the item and the criteria. It is flexible: change the prompt and you change the policy, with no training.
Its costs are latency, price and consistency. A judge call adds a full model round trip to every action, often around a second, and costs roughly $2–5 per 1,000 calls depending on model and prompt size. Research on LLM judges (for example Zheng et al., 2023) has documented biases such as sensitivity to position and verbosity, and outputs can vary between runs. The judge is also a language model reading untrusted content, so it can itself be prompt-injected.
A small fine-tuned guard such as MCP Guard trades flexibility for speed, cost and fixed scores. Many teams use both: the guard on every call, and a judge or a human for the uncertain ones.
Related terms
How LLM-as-a-judge connects to the rest of the vocabulary.