Jun 2026· Practice and Experience in Advanced Research Computing· 0 citations· 16 references
Computer Science
TL;DR
STREAM (Smart Tiered Routing Engine for AI Models) addresses the gap between local, HPC, and cloud inference with a three-tier routing architecture combining local, HPC, and cloud inference with a local LLM-based complexity judge.
Abstract
Researchers and practitioners working with large language models face a fragmented landscape: local models are free and private but hardware limits the model size and context windows a researcher can use; institutional HPC centers offer powerful GPU resources at no marginal cost and keep data within institutional boundaries, but operate behind firewalls and are designed for batch jobs rather than interactive use; commercial cloud APIs provide frontier-model quality on demand but impose significant cost and data retention policies unsuitable for sensitive research data. No existing system unifies all three. STREAM (Smart Tiered Routing Engine for AI Models) addresses this gap with four contributions: (1) a three-tier routing architecture combining local, HPC, and cloud inference with a local LLM-based complexity judge; (2) a dual-channel HPC streaming architecture that separates the Globus Compute control plane (authentication and job dispatch) from a WebSocket relay data plane (token delivery), enabling sub-second TTFT (0.54 s median, 21.1 × over batch mode’s 11.40 s) through institutional firewalls without VPN or firewall rule changes, with end-to-end AES-256-GCM encryption ensuring the relay operator cannot read token payloads; (3) tier-aware context summarization that prevents long conversations from forcing simple queries onto expensive tiers; and (4) an HPC-as-API proxy mode that exposes HPC inference as an OpenAI-compatible endpoint callable from any standard client with no HPC expertise, a deployment pattern made practical only by the sub-second TTFT of contribution (2). Llama 3.2 3B achieves 85.1% free-tier retention on a 1,200-query benchmark spanning ten domains. Measured TTFT: 0.26 s local, 0.54 s HPC (relay), 1.68 s cloud.
OpenTela is presented, a user-space orchestration overlay that turns existing fragmented HPC clusters into a unified, cross-institutional serving platform and provides a replicable blueprint for other sovereign AI initiatives to harness their own federated GPU infrastructure.
Xiaozhe Yao, Youhe Jiang, Ilia Badanin et al.· 0 citations
An architecture that decouples complex policy enforcement from high-speed packet forwarding to support VPC semantics on back-end NICs and enable front-end/back-end integration is proposed, suggesting that commodity hardware can support both high-throughput AI training and flexible VPC features.
Yinhe Wang, Xing Li, Enge Song et al.· Asia-Pacific Workshop on Net...· 0 citations
Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires balancing quality, latency, model footprint, and energy. This paper presents a controlled measurement study of self-hosted LLM inference across edge and near-edge deployment nodes: an NVIDIA Jetson AGX Orin and a near-edge server with CPU-only and GPU-enabled inference modes. We evaluate multiple open-weight LLMs and quantization variants using a fixed question-answering workload, and compare them against GPT-4o as a cloud-hosted accuracy and latency reference. Our benchmarking pipeline reports accuracy, model footprint, per-token decoding latency, prefill latency, and overall execution energy. The results show that GPU-enabled server execution provides the lowest compute-side latency, while Jetson Orin shows lower measured energy, consistent with its lower platform power under our setup. CPU-only execution is consistently dominated in latency for our workload and shows higher measured energy. We also show that parameter count and downloaded weight-file size alone do not reliably predict observed accuracy or latency. Finally, using Pareto-frontier analysis, we study how deployment decisions may change under possible streamed-token delivery overheads, highlighting that compute-side inference metrics alone can lead to suboptimal placement for latency-sensitive interactive web services.
Maysam Khatib, Moysis Symeonides, Demetris Trihinas et al.· 0 citations
CELLServe formalizes SLO-constrained joint resource provisioning as an optimization problem with a dedicated algorithm, and introduces an opportunistic instance merging strategy for decode phase functions to reclaim fragmented resources.
Zejian Wang, Nan Lin, Zinuo Cai et al.· ACM Transactions on Architec...· 0 citations
Edge AI is evolving from isolated inference toward long-running services that coordinate model pipelines, data streams, state, and accelerators near users and physical environments. Cloud-native and edge-native platforms offer useful foundations, but their primary control objects–containers, nodes, links, and enrolled sites–generally do not expose the model, data, state, quality, and participation semantics required by these services. This paper presents eAI+, a vision for an edgeAI-native service platform built around three first-class control objects: AI service graphs, dynamic edge resource fabrics, and participant contracts. eAI+ aims to preserve service quality under latency, privacy, reliability, cost, and participation constraints through three coordinated mechanisms. Runtime would select safe execution adaptations based on current workload, environment, and contract signals. Deployment would map service-graph components and prepared fallbacks to heterogeneous resources. PolyLink is the participant-contract module for plug-and-play resource onboarding; it would register contributors, verified resource offers, capabilities, and participation terms. Once a resource is onboarded, it would become available to Deployment for placing eligible service-graph components under the registered contract, while PolyLink would maintain metering, reputation, rewards, and exit events. Migration would transfer only continuity-critical state or control when mobility, overload, policy changes, or contributor lifecycle events invalidate the current placement. This framing treats edge AI as a coordinated service-platform problem across models, data, state, resources, and contracts.
Jiannong Cao, Zhiyuan Hu, Mingjin Zhang et al.· International Symposium on S...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
