Benchmark of linguistic variation in LLM‑generated texts

Abstract

This study investigates the register variation in texts written by humans and comparable texts produced by large language models (LLMs). Multidimensional analysis (MDA) is applied to a sample of human-written texts and AI-generated counterpart texts to find the dimensions of variation in which LLMs differ most significantly and most systematically from humans. We introduce the LLM-generated corpus AI-Brown, which is comparable to BE-21: a Brown family corpus representing contemporary British English. Since all languages except English are underrepresented in the training data of frontier LLMs, similar analysis is replicated on Czech language data using the AI-Koditex corpus and a Czech multidimensional model Cvrček et al. (2021) . Sixteen frontier models were examined in various settings and prompts, with emphasis placed on the difference between base models and instruction-tuned models. We offer a benchmark through which models can be compared with each other and ranked in interpretable dimensions.