Wan-Rong Gao

Stage-Aware Optimization of Llama 3 Model Inference for Language Services on Many-Core CPUs

LLM-based language services (e.g., summarization, translation, and question answering) require low-latency and cost-effective inference, yet GPU resources may be costly, unavailable, or difficult to integrate in many deployments. Many-core CPUs provide a flexible alternative, but achieving efficient inference requires...