AccessDetector: Android accessibility audit dataset (720 screens x 3 VLMs, 9-tag taxonomy, human-validated ground truth)

Abstract

Dataset and analytics produced by the AccessDetector pipeline: 835 captured Android screens across 138 apps with valid captures (from 140 explored), audited by 3 vision-language models (GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Pro) under a 9-tag accessibility taxonomy. The analytic universe of the paper is the 720-screen subset that remains after near-duplicate removal; on it the three models report 11,573 candidate violations, of which 3,453 unique cases were sampled per (issue type, model) cell, de-duplicated across models and validated by human reviewers. The capture pipeline patches uiautomator2's on-device u2.jar to expose the AccessibilityNodeInfo attributes (accessibility-heading, role-description, label-for / labeled-by, etc.) that the standard UIAutomator dump omits, so every taxonomy tag can be verified statically. Each violation carries a non-null som_index (anchor in the screenshot's Set-of-Marks overlay) and an intended_scope field (element, screen). Files: dataset.tar.gz (per-app layout XML, screenshots, SoM overlays and mappings, per-model report JSONs, manifest); manual_review_verdicts.tar.gz (the human-validated ground truth: for each of the 9 issue types, the pack manifest that maps every reviewed case to its screen and element, the two independent reviewer verdicts, and the consolidated verdict per case); results_csv.tar.gz (analysis tables, including manual_review_results.csv with one row per reviewed case and sampling model, and precision_by_bucket.csv with precision and Wilson 95% intervals for the 27 cells); results.xlsx (analytics workbook: overview, distributions, severity, WCAG, inter-model agreement, sampling plan). Limitations: 4 screens (0.5%) lack a Claude report due to Anthropic's 5 MB base64 image limit; OpenAI and Gemini cover all 835. Recall is not measured. Source code, the rich-dump patch (small source + pre-built JAR), the analysis scripts and the full methodology audit live at https://github.com/Benyo27/AccessDetector.