Qiaozi Gao

Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces

Reasoning Jury, a system that replaces the single judge with a jury of LLMs and a moderated consensus mechanism, is introduced to improve the fidelity of judgments for identifying reasoning defects and can be leveraged to understand failure modes of reasoning LLMs on benchmarks, which allows much deeper understanding of a model's performance.