Skip to content

S HARED C ONTEXT , S EPARATE E NGINES : S TREAMING V ISION –L ANGUAGE I NFERENCE ON E DGE S O C S

· 0 citations · 69 references

TL;DR

This work coordinates heterogeneous accelerators to execute stateful VLM inference concurrently and beats the strongest single-engine baselines on Apple M4 Max, Rockchip RK3588, and Qualcomm QCS6490, even when they reuse context.

View source

Similar papers

Preprint Oct 2026

Shared Context, Separate Engines: Stateful Streaming Inference for Multimodal LLMs on Edge SoCs

Edge SoCs combine CPUs, GPUs, and NPUs, yet runtimes for multimodal large language models (MLLMs) keep generation on one engine and leave others idle. Streaming workloads allow ingestion and answering to overlap, but backend-local KV state binds queries to the engine that built their history. Stage performance and co-r...

Yi-Long Li, Cheng-Po Yan, Yi-Jing Zeng et al. · 0 citations
Preprint Oct 2026

EdgePace: Deadline- and Freshness-Aware LLM Scheduling on Unified-Memory Platforms

Large language models (LLMs) increasingly run on edge devices alongside tasks that continuously process sensor data. In many applications, these foreground tasks prepare observations as inputs to the LLM, whose outputs guide subsequent processing or actions. The observations must be processed by foreground deadlines, a...

Teng Mei, Cheng-Xuan Pei, Wei-Jun Wang et al. · 0 citations
Book Open access Sep 2026

MCSched: Memory-Controller-Aware Scheduling for Embodied LLM Workloads on NVIDIA Jetson

Preliminary but compelling evidence that MC-aware scheduling is a practical operating system/runtime direction for deployable embodied AI on unified memory edge devices is provided.

Teng Mei, Cheng-Xuan Pei, Marco Canini et al. · 2 citations
Preprint Oct 2026

DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM Serving

Large language model (LLM) serving relies on batching to improve inference throughput, but the key-value (KV) cache grows with sequence length and limits GPU-resident concurrency. CPU offloading can expand KV capacity using host memory, yet existing systems remain largely GPU-gated: CPU-resident requests make progress...

Yuxuan Lin, Ke-Qiang Wu, Kingsum Chow · 0 citations
Preprint Oct 2026

FlashRT-PAI: Fast Kernels Are Not Fast Pipelines for Physical AI Inference

Vision-language-action (VLA), video-action (VA), and world action models (WAMs) form the backbone of modern physical AI systems, yet their real-time single-batch inference remains a defining bottleneck. Latency is dominated not by individual kernel computation, but by system-level overhead: thousands of short kernel la...

Liang Su, David Chen, Jason Li et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.