Performance Evaluation of LLM Inference Engines

LMTracer: Fine-Grained and Real-Time Performance Profiling for Production LLM Systems

LMTracer is presented, a fine-grained and real-time performance profiling framework for production LLM services that embed profiling logic into the execution through graph-embedded probing and streaming buffered profiling data to CPUs on demand to keep the execution of user kernels uninterrupted.

A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware

A controlled measurement study of self-hosted LLM inference across edge and near-edge deployment nodes: an NVIDIA Jetson AGX Orin and a near-edge server with CPU-only and GPU-enabled inference modes, highlighting that compute-side inference metrics alone can lead to suboptimal placement for latency-sensitive interactiv...

Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction in GPU-Accelerated LLM Inference

Results show that MTP improved inference through amortization: greater token progress reduced repeated GPU execution sufficiently to outweigh the additional speculative-execution cost.

O P S ERVE : Opportunistic LLM Inference over Fragmented GPU Capacity in HPC Systems

Master Computer Science

A split-phase heterogeneous deployment strategy is proposed, and key optimization paths, including operator ecosystem completion and deep operator fusion, are identified.

LLM Inference in a Flash!

This work designs an end-to-end integer-only quantization approach to eliminate expensive floating-point computations on Flash compute-in-memory devices, and designs a dictionary-based KV cache compression strategy based on sparse dictionary coding that represents each KV vector as a linear combination of static dictio...