Skip to content
Book Open access

ML Inference Acceleration via Scalable In-network Computing

Nov 2026 · Proceedings of the 35th ACM International Conference on Information and Knowledge Management · 0 citations · 11 references

Abstract

Modern inference engines have significantly advanced the scalability of large models through innovations in serving techniques, proving instrumental in driving the widespread adoption of AI. However their true potential is constrained by traditional server architecture where reliance on host CPUs and operating systems introduces unnecessary overhead and degrades performance. We identify a significant opportunity to eliminate these bottlenecks by offloading data management directly to the network layer by proposing a unified architecture based on our previous research and ongoing work using programmable Data processing units (DPUs)/SmartNICs for in-network computing. Our previous work, NICTokenizer is a framework that offloads the initial tokenization phase to programmable network dataplane eliminating the host OS Kernel overhead lowering the overall tokenization latency by 61.3% and increasing the token throughput by 407.62% compared to conventional server based implementations and was previously published as a poster at SIGCOMM' 25. Alongside we introduce CacheFlow a disaggregated prefix cache pool which utilizes the DPU memory and GPUDirectRDMA for efficient sharing, storage and retrieval of KV cache across a cluster. CacheFlow reduces the average Time to first token (TTFT) by 82% imporves overall throughput by 39% and decreases memory storage requirements by 44% compared to standard inference engines like vLLM. As a natural progression our future research explores offloading semantic routing to a DPU enabling the network plane to dynamically select the appropriate model experts for each incoming sequence.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.