Edge SoCs combine CPUs, GPUs, and NPUs, yet runtimes for multimodal large language models (MLLMs) keep generation on one engine and leave others idle. Streaming workloads allow ingestion and answering to overlap, but backend-local KV state binds queries to the engine that built their history. Stage performance and co-r...
Yi-Long Li, Cheng-Po Yan, Yi-Jing Zeng et al.· 0 citations
This work coordinates heterogeneous accelerators to execute stateful VLM inference concurrently and beats the strongest single-engine baselines on Apple M4 Max, Rockchip RK3588, and Qualcomm QCS6490, even when they reuse context.
Yi-Long Li, Cheng-Po Yan, Yi-Jing Zeng et al.· 0 citations
HCCL, a collective communication library co-designed with Meta's MTIA 300 accelerator, the first Meta chip to integrate backend networking directly on chip package, and describes collective designs that improve compute-communication pipelining for latency-sensitive workloads are presented.
W. Bland, Tiago Antunes, Lars Paul Huse et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.