Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks
BenCheck is proposed, a package for benchmark validity analysis that encapsulates the main checks performed in a case study on HellaSwag and can be used to audit commonsense reasoning benchmarks, including PIQA, Global PIQA, and Winogrande.
Pavel Chizhov, Anton Changalidis, Vishnu Prasad Vijaya Kumar et al.
· 9 citations
· ⚡2