HHR: Hierarchical Hash Retrieval for Efficient LLM Generation
Efficient long-context inference is essential for large language models (LLMs), yet it poses a severe computational bottleneck. Hash-based retrieval offers an efficient alternative by encoding queries and keys into binary codes and using Hamming distance for key selection. However, this leads to a critical mismatch bet...
Lian Liu, Tian-Tian Zheng, You Huang et al.
· 0 citations