Preprint

StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design

Yin Wang Haotian Hu J. Han Wentao Qiu Zhenhua Ge Liujian Tang Fanyi Wang

Jul 2026 · 1 citation · 74 references

Computer Science

TL;DR

StepX-Edge is presented, a 0.9B-parameter on-device UI vision-language model that resolves this tension through three-layer co-design of architecture, training, and deployment, and achieves the strongest overall UI understanding among<=1B models.

Abstract

Deploying a vision-language model with full UI understanding on end devices has long been trapped between accuracy and efficiency: on one side is the accuracy bar for OCR, screen understanding, visual question answering, and element grounding; on the other is the strict compute, memory, and power budget of mobile chips. Existing work either trades one for the other, or stops at simulation without real-device validation. We present StepX-Edge, a 0.9B-parameter on-device UI vision-language model that resolves this tension through three-layer co-design of architecture, training, and deployment. Architecturally, UI-aware Layered Visual Encoding (ULVE) and a Progressive Dimensionality Projection (PDP) connector target the extreme aspect ratios and fine-grained perception of screens, while standard full attention throughout ensures native compatibility with mainstream mobile NPU operators. For training, the five-stage StepX-Curriculum framework is designed around our observation of mutual-promotion effects among UI subtasks, so that all four capabilities grow synergistically under a tight parameter budget rather than interfering. For deployment, a module-wise differentiated two-stage PTQ-to-QAT quantization scheme keeps the post-quantization accuracy loss within 1%. StepX-Edge achieves the strongest overall UI understanding among<=1B models, surpassing all 2B-2.3B baselines on ScreenQA (88.76 F1) and Chinese OCRBench v2 (57.25), and matching 1.3B-2.3B general VLMs on RefCOCO (92.0%) and OCRBench v1 (831) with far fewer parameters. After W4A16+KV8 quantization, the model runs stably on Snapdragon 8 Gen5 devices with ~0.84 s TTFT, 98 tok/s decode, and 1.4 GB peak memory. We will open-source the training data, the full training recipe, and the quantization deployment pipeline.

arXiv Semantic Scholar

View source

Similar papers

Jul 2026

UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models

Extensive experiments demonstrate that the on-device latency-informed design combined with the tailored training strategy establishes a new state-of-the-art for efficient LVLM encoding, significantly outperforming existing encoder-centric baselines while operating on-device at nearly 1.7xthe speed.

Ioannis Maniadis Metaxas, Adrian Bulat, Alberto Baldrati et al. · arXiv.org · 0 citations

Preprint Jul 2026

Device-First Feedback: Toward Mobile-Native LLM-Driven Neural Architecture Search

The two studies show that closed-loop GPU fine-tuning does not guarantee monotonic mobile gains, especially on harder classification tasks, and that multi-dataset, on-device measurement is needed to stress-test deployment objectives.

Saif U Din, Muhammad Hussain, R. Timofte et al. · 0 citations

Preprint Aug 2026

Opt.Gear Technical Report

OptGear-1M is the first generative language model to achieve 20 TPS with W4A32 quantization on the ARM Cortex-M7 CPU of the STM32H747I-DISCO, and is the most data-efficient of the existing foundation models.

Juneyoung Park, Youngwoo Kwon · 0 citations

µ VLM: A Vision Language Model for µ NPUs

Zijie Chen, Guiyun Fan, Zhaoxing Yang et al. · 0 citations

Review Jul 2026

3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects

3D-DefectBench is introduced, a benchmark and framework for systematic analysis of VLM-based 3D defect detection pipelines, and finds that automated judges should be evaluated as complete pipelines and calibrated across human reference regimes, rather than benchmarked only as standalone models.

Zhenyu Zhao, Nanshan Jia, Jihyeon Je et al. · arXiv.org · 0 citations

Preprint Jul 2026

Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment

The emergence of vision language models with fewer than 3 billion parameters has accelerated the implementation of on-device multimodal intelligence. However, a detailed understanding of component-wise quantization remains a bottleneck for optimal deployment. This paper presents a systematic evaluation framework for empirically validating five hypotheses across six quantization configurations on the Jetson Orin NX and AGX. By separating the vision encoder, projector, and large language model backbone yields the following results: (1) Quantization sensitivity is governed by the structural paradigm (MoE vs. dense) rather than scale alone, with MoE backbones mitigating INT4 noise where dense backbones degrade; (2) SigLIP encoders incur disproportionate INT8 latency on Jetson Ampere--a deployment-specific encoder-kernel-hardware interaction, not a SigLIP flaw; (3) Although INT4 quantization of LLMs greatly reduces VRAM consumption, it also causes slower token generation due to dequantization overhead; (4) Composite quantization errors are largely additive, except along the modality-alignment path, which is architecture-dependent; (5) The intelligence-per-joule profile varies significantly across platforms owing to memory bandwidth constraints.

Hyeju Shin, Chorwon Kim, Ryangsoo Kim et al. · 0 citations