This work evaluates six large language models (LLMs) on their ability to estimate American social norms across 555 everyday scenarios and compares these to estimates from 320 humans, indicating that combining a human collective with an LLM substantially outperforms either source alone.
Abstract
As AI assistants and social robots enter human environments, their ability to navigate context-dependent social norms is essential to avoid harm and ensure successful collaboration. We evaluate six large language models (LLMs) on their ability to estimate American social norms across 555 everyday scenarios (measured in prior work) and compare these to estimates from 320 humans. LLMs achieve remarkably high accuracy, clearly outperforming the average human. However, the errors LLMs make are systematic; they are similar across runs of the same LLM and even across different LLMs. As a consequence of this homogeneity, aggregating estimates of LLMs produces little improvement. Individual humans make much worse estimates, often defaulting to extreme right-or-wrong judgments even when asked to estimate population averages, but their errors are idiosyncratic and, consequently, aggregating their estimates yields dramatic improvement as these independent errors cancel under averaging. Because humans and LLMs make different errors, combining a human collective with an LLM substantially outperforms either source alone.
A key aspect of culture is a society's norms about everyday behavior. How accurately do large language models (LLMs) represent cultural differences in such norms? To answer this question we used the recent Global Study of Everyday Norms (GSEN), which collected ratings of 150 scenarios in 90 societies, as the human benc...
Kimmo Eriksson, I. Vartanova, P. Strimling· 0 citations
Understanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and multi-agent environments. Social intelligence comprises two aspects: cognitive intelligence, the ability to infer others'mental states, and behavioral intelligence, the abi...
Alieh Zamani, Soroush Esfahanian, Erfan Shahabi et al.· 0 citations
MindEvolve is introduced, an autonomous workflow designed to predict behavior in social interactions by generating interpretable symbolic models of cognition, and provides a roadmap for advancing LLM-based cognitive modeling toward human-expert-level theory construction.
Ying-Ying Ye, You-Le Fang, Xiao-Xue Gao et al.· bioRxiv· 0 citations
Findings support debate as a viable path toward scalable human oversight, but cast doubt on its benefits for weaker AI supervisors, raising concerns about fully automated training and control pipelines in high-stakes settings.
María Victoria Carro, Denise Alejandra Mester, Facundo Nieto et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.