Background Large language models (LLMs) are increasingly used to generate health information, yet their reliability as evaluators remains unclear. This study investigated the feasibility of an LLM-as-a-judge methodology in the context of infection prevention and antimicrobial resistance (AMR), comparing automated ratings with human expert benchmarks. Methods We performed a secondary analysis of an expert-annotated dataset of health messages. Three leading LLMs (ChatGPT, Claude, Gemini) independently evaluated the same messages using an adapted DISCERN tool across five domains: information reliability, quality, AMR impact, persuasiveness, and overall score. We utilized descriptive statistics, intra-rater reliability tests, and mixed-effects ordinal regression to analyze divergence between automated and human assessments, adhering to CHART reporting guidelines. Results Analysis of 404 evaluations revealed a systematic upward divergence: all LLMs consistently assigned higher scores than human experts. This optimism bias persisted after adjusting for domain-specific differences and clustering effects. The gap was particularly pronounced in domains of persuasiveness and AMR impact, while information quality showed more heterogeneous results. Intra-rater reliability assessments demonstrated that LLMs maintained stable scoring patterns under identical prompting conditions. Conclusions LLMs exhibit a consistent leniency bias, systematically overestimating the quality of AMR-related health communication compared to human evaluators. These results do not support the use of LLMs for autonomous evaluation in high-stakes public health contexts. Rather, LLM-based judging is best suited as a scalable screening tool within supervised human-in-the-loop workflows, where expert oversight serves as a necessary safeguard for evidence-based accuracy.
Marcello di Pumpo, Leonardo Villani, M. R. Gualano et al.· Frontiers in Public Health· 1 citation
Background: Education is a cornerstone of antimicrobial stewardship, yet evidence on its impact across the entire hospital workforce remains limited. This study aimed to assess knowledge, attitudes, and practices regarding antibiotic use and AMR among workers employed at an Italian tertiary-care hospital, and to explore the association between questionnaire performance and participation in antibiotic-related training activities. Methods: A repeated cross-sectional survey was conducted in 2024 and 2025 at a large Italian tertiary-care hospital (Padua, Northern Italy). A structured questionnaire adapted from an ECDC-validated instrument was distributed through the institutional email system to all hospital employees, including healthcare professionals and non-healthcare workers. The questionnaire investigated demographic and professional characteristics, participation in antibiotic-related training activities, and knowledge, attitudes and practices regarding antibiotic use and AMR. Questionnaire performance was assessed using both a binary scoring system and a partial credit scoring system. Multivariable linear regression analysis was performed to identify independent predictors of performance. Results: A total of 3193 participants completed the survey (1928 and 1265 in 2024 and in 2025, respectively). The mean (SD) total score was 5.75 ± 2.25 out of 15 points, with the lowest performance observed in the practice domain. No significant differences were found between the two survey years (2024 and 2025). Participants who attended antibiotic-related training performed significantly better than untrained participants (6.23 vs. 5.01; p < 0.001), with prior training emerging as the strongest independent predictor of questionnaire performance (β = 4.8; p < 0.001). Younger age was associated with better performance, while longer seniority (>30 years) was associated with lower scores. Among healthcare professionals, modestly higher scores were observed in 2025 compared with 2024, particularly in the practice domain. Conclusions: Persistent opportunities for improvement remain in AMR-related knowledge, attitudes and particularly practices across the hospital workforce. Participation in antibiotic-related training activities was associated with improved questionnaire performance, supporting the role of structured and continuous educational interventions within hospital AMS programs. Repeated institutional educational initiatives may contribute to strengthening stewardship-related awareness and practices over time.
D. Mengato, Elisabetta Conte, Giacomo Berti et al.· Antibiotics· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
