Dirk Hovy

Investigating Multi-Cultural Value Selection Biases in LLMs

Large language models (LLMs) serve users whose cultural identities are multi-layered, yet cultural evaluations commonly reduce identity to one label. In this work, we study LLMs under simultaneous identity cues, with a country persona in the system prompt and a second identity signaled in the user turn, either by namin...

ACID: On the Perception of Online Classism

The Role of Implicit and Explicit Demographic Signals in Large Language Model-based Student Assessment

Large Language Models are now common in student assessment, but we know little about how student demographics affect their use. Sometimes, considering student demographics may be necessary -- for example, to improve readability for users with lower educational levels. However, it also risks being a cause of discriminat...

WildSEEK: Evaluating Language Models for Information-Seeking

This work introduces WildSEEK, a manually annotated dataset of 3k information-seeking queries from real user interactions, and an evaluation framework for LLM-generated responses, and finds that over a third of information-seeking queries are high-risk and more often analytical.