LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model
A model-based evaluation framework that combines multidimensional item response theory model with question contexts to predict LLM performance on unseen questions is studied, suggesting that context-aware psychometric modeling is a promising direction for efficient and interpretable LLM evaluation and highlighting cross-scenario generalization as a central open challenge.