Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation
Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task designs, three independent runs, and 3,000 tweets labeled for offensive language and hate speech. Repeating the same model and task design pr...
Thomas Reiter, Christoph Kern, F. Miasnikov et al.
· 0 citations