An Exploratory Study on Prompting Strategies for LLM-Assisted Heuristic Evaluation of Web User Interfaces

Abstract

Usability is a key non-functional requirement in interactive software systems, and heuristic evaluation can support the verification and validation of user interfaces by identifying violations of usability principles. However, this process still depends on human expertise, is subject to evaluator variability, and may be difficult to scale. Although recent advances in Large Language Models (LLMs) create opportunities for AI-assisted heuristic evaluation, there is limited empirical evidence on the reliability of LLM-generated diagnoses. This paper presents an empirical study of GPT-5.4-assisted heuristic evaluation using Nielsen’s heuristics (H), with diagnoses evaluated against an expert-derived ground truth.We evaluate zero-shot, fewshot, and chain-of-thought prompting on 12 web user interfaces, comparing model diagnoses with the reference evaluation using precision, recall, F1-score, and Cohen’s kappa. Few-shot prompting achieved the best precision-recall balance (F1=0.51, 𝜅=0.193), while chain-of-thought achieved the highest recall but produced more false positives. Heuristic-level results revealed distinct error patterns, with high recovery for H8 and H10 and frequent unsupported predictions for H4, H5, and H6, although these patterns are affected by uneven ground-truth prevalence. These findings suggest that prompting strategy is an important design consideration in LLM-assisted heuristic evaluation, while human validation remains necessary.