Adaptive Hybrid Quantization Framework for Real-Time Speculative Decoding in Edge-Cloud Systems
High-throughput inference for Large Language Models (LLMs) remains bottlenecked by memory bandwidth constraints and compute latency, particularly in hybrid edge-cloud topologies. This paper proposes an Adaptive Hybrid Quantization (AHQ) framework integrated with real-time speculative decoding. By dynamically allocating...