Glossary
R-Judge
A public benchmark of agent interaction records labelled for safety risk, used to test whether a model can judge when an agent is doing harm.
Checks run on our GPUs in Switzerland. Request payloads are not stored.Security
What R-Judge means
R-Judge (Yuan et al., 2024, arXiv 2401.10019) is a benchmark for evaluating whether models can recognise safety risks in the interaction records of LLM agents. Each record is a multi-turn agent trajectory, including tool calls and results, labelled as safe or unsafe and spanning several application categories and risk types.
MCP Guard evaluates on R-Judge by checking each agent step before it runs and counting a trajectory as flagged if any step is flagged. On the held-out test half (304 trajectories, never used for any model or epoch choice), it measured AUROC 0.855, against 0.824 for saroku-guard. With a test set this size, the difference is not statistically significant.
R-Judge is one benchmark with its own mix of scenarios, including some content-safety conversations alongside tool calls, so results on it are an indication, not a guarantee of performance on your agents.
Related terms
How R-Judge connects to the rest of the vocabulary.