Human Perceptions Meet AI Analysis: A Usability Evaluation of Climate Conference Websites
Abstract
Introduction: Digital platforms play a central role in mediating major international events, serving as gateways for information, mobilization, and public engagement. Evaluating usability in heterogeneous, cross-cultural contexts such as the Conference of the Parties (COP) poses significant challenges. The emergence of Generative Artificial Intelligence (GAI) offers new opportunities for automated usability assessment, but its alignment with real user experience remains underexplored. Objective: This study critically compares GAI-generated usability evaluations against human perceptions of climate conference websites. GAIs assessed both COP29 and COP30 platforms, while human evaluations focused exclusively on COP30 during the event in Belém, Brazil. The objective is to examine convergences and divergences between automated and human-centered assessments. Methods: Four GAIs (ChatGPT, Gemini, DeepSeek, and Copilot) performed heuristic evaluations based on ISO 9241-210 criteria. GAI outputs were analyzed using Directed Categorical Content Analysis. These findings were complemented by an exploratory evaluation involving ten human participants (n=10) with diverse profiles, whose perceptions were collected via an online questionnaire during COP30. Final analysis involved triangulation of GAI findings with human feedback. Results: Results revealed a notable divergence in overall assessment. Humans reported high satisfaction and usability for critical tasks, contrasting with GAIs, which penalized COP30 for structural deficiencies, particularly low Controllability. However, convergence occurred in Individualization Adequacy, the lowest-scoring criterion for both GAIs and humans, confirming systemic failures in adaptability and clarity. Conclusion: GAIs and users converge on structural issues but diverge on subjective and contextual evaluations. This highlights the value of hybrid models: GAIs provide systematic heuristic data, while human feedback grounds practical and sociocultural interpretations. Future work should include longitudinal evaluations, automated accessibility checkers, and hybrid workflows for robust assessment frameworks.