Quantization Trade-offs for LLM Inference on Apple Silicon: A Practical Speed, Memory, and Accuracy Benchmark
Deploying large language models (LLMs) on consumer hardware typically requires quantization to reduce memory footprint and increase inference speed, but the practical trade-offs of this compression are not well characterized on Apple Silicon, where most existing benchmarks focus on NVIDIA GPU deployments. We present a...