Aug 2026· Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2· pp. 5638-5649· 1 citation· 14 references
TL;DR
This work proposes ComUICoder, a component-based UI code generation framework that emphasizes semantic-aware segmentation, code reuse, and fine-grained refinement and significantly improves overall generation quality and code reusability on complex multipage websites.
Abstract
Multimodal Large Language Models (MLLMs) have demonstrated strong performance on the UI-to-code task, which aims to generate UI code from design mock-ups. However, when applied to long and complex websites, they often struggle with fragmented segmentation, redundant code generation for repetitive components, and frequent UI inconsistencies. To systematically investigate and address these challenges, we introduce ComUIBench, a new multi-page complex webpage benchmark with component annotations, designed to evaluate MLLMs' ability to generate reusable UI code in realistic website scenarios. Building upon this benchmark, we propose ComUICoder, a component-based UI code generation framework that emphasizes semantic-aware segmentation, code reuse, and fine-grained refinement. Specifically, ComUICoder incorporates (1) Hybrid Semantic-aware Block Segmentation for accurate UI semantic coherent block detection, (2) Visual-aware Graph-based Block Merge to consolidate structurally similar components within and across webpages for reusable implementation, and (3) Priority-based Element-wise Feedback to refine generated code and reduce element-level inconsistencies. Extensive experiments demonstrate that ComUICoder significantly improves overall generation quality and code reusability on complex multipage websites. Our datasets and code are publicly available at https://github.com/WebPAI/ComUICoder.
Two language-model-based strategies are proposed for semantic code document segmentation, including a line-by-line approach that classifies each line of code separately before grouping the results into functional units, and a range-based approach that aims to directly determine groups of code lines from the input.
Abdelhalim Hafedh Dahou, A. Scherp, Sebastian Kurten et al.· Proceedings of the 2026 ACM...· 0 citations
User Interface (UI) programming is challenging due to the complex abstraction gap between code and graphical software representations. To bridge this gap, UI programming tools often rely on screen retrieval and clustering, which require accurate similarity measures based on overlapping features. However, computing feature-oriented similarity is difficult because screens with similar functionality often exhibit design variations. To address this, we propose FRAME (ReinForced UseR InterfAce Screen EMbedding with Graphical Structural ComprEhension), a multi-modal, neuro-symbolic embedding technique. FRAME constructs symbolic, graph-based representations of UI components to encode salient relationships and capture feature patterns across different screens. It leverages large vision-language models for visual and lexical encoding, alongside a novel UI-specific computational geometry algorithm that enables weighted embedding propagation. Across three benchmarks, FRAME outperforms strong baselines by up to 13% MRR in search and 7.6 percentage points in clustering accuracy. A comprehensive ablation study further confirms the benefit of each component, demonstrating FRAME's potential for enhancing automated UI design and testing tools.
LLM-based repository-level code generation aims to generate code using the context available in a software repository, requiring LLMs to reason over complex code dependencies. Due to limited context windows and insufficient repository-specific understanding, LLMs typically rely on retrieval-augmented generation (RAG) to incorporate relevant code. Early RAG approaches primarily employ similarity-based retrieval, which often fails to retrieve code snippets that the target function depends on. Recent work introduces graph-based retrieval to model such dependencies, but typically relies on manually designed rules and static global graphs, leading to limited flexibility and high construction and maintenance costs. In contrast, human developers collect helpful context by implicitly constructing a partial dependency graph and iteratively inspecting along it. Inspired by this behavior, we propose DyRetriever, an efficient context retrieval method via partial dependency graphs. DyRetriever uses an LLM to first select a set of entry-point functions and then perform multi-hop reasoning along the code dependency graph. During multi-hop reasoning, it uses the LLM's semantic understanding to validate whether a function can help generate the target function, eliminating manually designed rules and enabling flexibility across scenarios. Instead of statically constructing a global dependency graph, DyRetriever builds a partial graph on demand and discards it after use, reducing construction and maintenance costs. We integrate DyRetriever with a similarity-based code retriever to build DyCoder and evaluate it on CoderEval and DevEval. Experimental results show that DyCoder achieves relative Pass@1 improvements of 25.63% and 59.73% on CoderEval and DevEval, respectively, compared with existing RAG-based methods, while being 7.4x faster than baselines based on static dependency graph construction.
Zhongxin Liu, Zhonghao Jiang, Zhifan Ye et al.· 1 citation
UI2App is the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone, without any textual or behavioral guidance, and designs an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference.
Grace Man Chen, Litao Guo, Yifan Wu et al.· arXiv.org· 0 citations
It is shown that current MLLMs can reproduce visual appearance but remain limited in generating the data semantics and interactive logic required by coordinated multi-view interfaces, and Iterative refinement improves code executability but does not substantially reduce the gap in data binding and interaction generation.
Yue Zhao, Hongxu Liu, Feiyu Wang et al.· arXiv.org· 0 citations
Chat2Map is presented, an agent framework for automated web map generation from natural-language instructions that integrates skill and tool grounding, runtime schema construction for uploaded GeoJSON and JSON files, and an iterative generate-execute- diagnose-repair loop based on runtime and visual feedback.
Hong-Ping Zhang, Peilong Ma, Cong Wang et al.· The International Archives o...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
