They measure bias using "the Bias Benchmark for QA (BBQ), a dataset of question sets...that highlight attested social biases against people belonging to protected classes along nine social dimensions relevant for U.S. English-speaking contexts. Our task evaluates model responses at two levels: (i) given an under-informative context, we test how strongly responses reflect social biases, and (ii) given an adequately informative context, we test whether the model's biases override a correct answer choice" [1].

Aside: I'm curious how distillation affects such scores. If I distill an unbiased model, how are my distillations distributed?

[1] https://arxiv.org/abs/2110.08193 Table 1 is quite hilarious