Lin-Zhuo Li

Knowing Your Uncertainty - On the application of LLM in social sciences

A unified framework for evaluating LLM uncertainty based on Hill numbers, a family of diversity measures, is introduced, which provides a common and intuitive scale for interpreting variation in LLM outputs while accommodating different notions of semantic similarity and different sensitivities to output distributions.