LLM Inference in a Flash!
This work designs an end-to-end integer-only quantization approach to eliminate expensive floating-point computations on Flash compute-in-memory devices, and designs a dictionary-based KV cache compression strategy based on sparse dictionary coding that represents each KV vector as a linear combination of static dictio...
Sebastian Zhao, Minseo Kim, C. Hooper et al.
· 1 citation