AI guardrail scores can hinge on which definition of 'unsafe' a model was trained on

Guardrail classifiers are the filters that sit in front of AI chatbots and flag requests that should not be answered. They are usually ranked by benchmark scores, as if 'unsafe' were a fixed property of a piece of text. A new study argues that it is not. In a preprint posted to arXiv, which has not yet been peer reviewed, the authors report that on some widely used safety benchmarks a guardrail's score depends far more on which definition of 'unsafe' it was trained on than on how well it detects harmful content.

The study draws on what the authors call an accident of an industrial deployment. The same set of roughly 53,000 prompts had been labelled twice, under two working definitions. One asked whether a prompt is a request for harmful content. The other asked whether a human moderator would block it. Both label sets were used to train ten guardrail variants, or 'arms', at two model sizes. The arms shared the same architecture, codebase and optimiser family (the optimiser is the algorithm that adjusts a model during training). Because so much was held fixed, differences between arms could be traced largely to the labels.

The authors first checked for contamination, meaning benchmark test items that had leaked into training data. They report that 88% of JailbreakBench-benign and 82% of OR-Bench-hard were already present in the guardrail's training pools. Both are over-refusal benchmarks, which test whether a filter wrongly blocks harmless requests. They are exactly the kind of evidence a guardrail cites to show that it does not over-block. According to the study, all later results use decontaminated data.

Benchmarks then differed sharply in how much the label definition mattered. With architecture and optimiser fixed, scores across the ten arms varied by 0.99 on HarmBench-copyright but by only 0.043 on WildGuard-benign, a 23-fold spread. HarmBench-copyright scores ran from 0 to 100, and over-refusal on OR-Bench-hard from 3.4% to 56.4%. The authors report that changing just 0.03% of a model's parameters could move a score across its whole range. The four arms whose final stage of training carried one particular policy sat at one extreme on both benchmarks, while their underlying detection capability was unchanged. The authors give a p-value of 0.005 for this ranking, or between 0.04 and 0.14 under their least favourable analysis.

The effect also carries through distillation, in which a smaller 'student' model learns from a larger 'teacher'. The teacher's soft labels are its graded probabilities rather than yes-or-no verdicts. According to the study, they passed on the teacher's policy even when every hard label the student saw reflected the other definition. The authors argue that this makes the teacher behind any released checkpoint an unrecorded confounding factor. Finally, one validation split graded all twenty arms. On it, accuracy tracked agreement with the annotators' policy, with a rank correlation of 0.74, more closely than it tracked detection capability, at 0.34.

The authors recommend that guardrail evaluations report three things as first-class variables. The first is contamination. The second is a 'seed floor', a baseline for how much scores move between otherwise identical training runs. The third is the definition of 'unsafe' a benchmark was graded against. The evidence comes from a single industrial dataset and from models sharing one architecture and codebase. How widely the pattern holds for other guardrails and benchmarks remains to be tested.