Hardware-Adaptive Inference Orchestration: Zero-Overhead Local LLM Agents on Resource-Constrained Edge Devices

Abstract

Autonomous AI agents typically rely on multi-turn ReAct loops that demand repeated system-prompt evaluation, persistent state tracking, and frequent tool selection. On constrained edge hardware—especially CPU-only devices with approximately 8 GB of RAM—this style of orchestration creates two compounding failure modes: memory fragmentation that eventually triggers out-of-memory (OOM) crashes, and promptevaluation latency that grows with the size of the active conversation. This paper presents FastAgent, a lightweight, headless inference orchestrator built around Longest Common Prefix (LCP) slot caching, 8-bit key-value (KV) cache quantization, grammar-constrained decoding, and an Adaptive Compute Controller. The design goal is not merely to run a smaller model locally, but to reproduce the stability characteristics of cloud-hosted agent APIs on hardware that would otherwise be too limited for sustained autonomous loops. On a baseline CPU environment, the framework reduced prompt-evaluation latency from roughly 3,500 ms to about 73 ms in steady state while eliminating OOM failures during long-running tool loops, showing that cloud-like responsiveness is achievable on local devices when runtime and memory management are co-designed.