A. Silbersdorff

Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges

Typed decision models (TDMs) return a probability for each allowed answer instead of generating text, which makes them candidates for cheaply replacing LLM judges that compare two answers. Whether they share the biases of LLM judges is untested. We run the CALM benchmark's bias tests on eight TDMs and seven LLM judges...