Evaluating the Effectiveness of Open‐Source LLMs for Automated Analysis of Multilingual Consultation Feedback: A Swiss Case Study
TL;DR
Assessment of an open‐source and proprietary large language models for analyzing stakeholder inputs from three Swiss pre‐parliamentary consultations highlights both the potential and limits of algorithmic support for participatory governance in linguistically and technically diverse environments.
Abstract
Public consultations are a cornerstone of democratic policymaking, yet the sheer volume and heterogeneity of submissions often overwhelm traditional analysis. Emerging large language models (LLMs) offer new opportunities for making sense of such data, but their performance in contexts with high linguistic and technical diversity remains underexplored. In this research note, we present a case‐based demonstration rather than a generalizable evaluation, assessing an open‐source LLM (Gemma3:27b) and a proprietary LLM (GPT‐5‐mini) for analyzing stakeholder inputs from three Swiss pre‐parliamentary consultations. The coexistence of multiple languages and varying technical complexity in submissions creates uniquely challenging datasets, providing a rigorous test for automated tools. We benchmark the model against official administrative reports, assessing accuracy and completeness. Results show the open‐source model performs excellently in straightforward consultations, achieving a mean accuracy of 90%, but encounters difficulties in retrieving the substantive content (overall completeness: 70%). Comparatively, the proprietary model obtains better results than the open‐source one in these two settings. Overall, LLMs can streamline analysis and enhance interpretive capacity, complementing (but not replacing) human expertise. The study highlights both the potential and limits of algorithmic support for participatory governance in linguistically and technically diverse environments.