We have 2 of 13 papers
We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.
Not the right person? Other researchers publish under this name.
This work proposes MoEZip, a routing-aware KV cache compression for sparse MoE language models, and proposes a novel metric based on the Fisher information matrix of routing distributions that combines attention weights for routing-aware scoring of contexts.
To support long contexts efficiently, Sparse Gated Attention (SGA), which combines sparse attention with gated attention, and adopt Gated Norm (GN) to stabilize large-scale training is introduced, which keeps 4-bit NVFP4 serving within one point of FP8 accuracy.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
