LLM Inference Performance Optimization in Limited-Resource Environments

Abstract

With the widespread adoption of LLM-based chatbots, cloud-dependent solutions have come to dominate the market. However, open-source pre-trained LLMs are enabling implementation of local solutions. Achieving competitive performance locally requires the ability to run high-parameter models. Here, the primary bottleneck is GPU VRAM capacity, which limits model parameter size. Furthermore, the efficiency of inference optimizations such as kv caching depends directly on the amount of available VRAM remaining after the model is loaded. In such a scenario with hardware constraints, we conduct user tests by applying kv cache quantization. As a result, we identify distinct performance trends in critical metrics such as Time-to-First-Token (TTFT) and Total Generation Time. Additionally, we evaluate model accuracy results using the LLM-as-a-judge paradigm.