Glossary
AUROC
Area under the ROC curve: the probability that a randomly chosen positive example is scored higher than a randomly chosen negative one.
Checks run on our GPUs in Switzerland. Request payloads are not stored.Security
What AUROC means
AUROC (area under the receiver operating characteristic curve) summarises how well a score separates two classes across all possible thresholds. It equals the probability that a randomly chosen positive (say, an unsafe action) receives a higher score than a randomly chosen negative. 0.5 is chance; 1.0 is perfect separation.
Because it is threshold-free, AUROC is convenient for comparing models. It does not tell you how well calibrated the scores are, nor how a model performs at the specific threshold you will use, and small differences on small test sets are often not statistically significant.
On the held-out test half of R-Judge (304 trajectories), MCP Guard measured AUROC 0.855 versus 0.824 for saroku-guard. That difference is not yet statistically significant, so the fair reading is “on par with or better than”, not a clear win.
Related terms
How AUROC connects to the rest of the vocabulary.