On-policy distillation (OPD) improves language-model post-training by letting the student learn from its own sampled trajectories under teacher supervision, aligning the training process with inference-time behavior. However, the rollout policy in OPD is usually treated as fixed: each round samples directly from the cu...
Si-Rui Chen, Yun-Zhe Qi, Hao-Bo Xu et al.· 0 citations
STEPQuant is proposed, a spatial-temporal post-training quantization framework for Delta-rule recurrent states that allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error.
Bing-Chen Yao, Hao-Bo Xu, Hao-Kun Lin et al.· 0 citations
First, it is demonstrated that quantization is significantly more effective in preserving trustworthiness compared to pruning, and more importantly, it is demonstrated that compressing a reliable large model via quantization can produce SLMs with superior trustworthiness and adaptability compared to using small models...
Hao-Kun Lin, Kai-Jie Zhu, Hao-Bo Xu et al.· 2 citations
PILL (Probing-based InfiLling with preset-Length-free decoding), an efficient infilling method for DLMs that requires no preset initial length and adds far fewer extra forward passes than baselines, substantially reducing inference time is proposed.
Hao-Bo Xu, Si-Rui Chen, Yuan-Chen Bei et al.· 6 citations
This work proposes method that unifies textual reasoning and graph message passing within a masked diffusion language model, a language model with bidirectional attention and generative decoding that outperforms graph neural networks, graph transformers, and LLM-based baselines on all three TAG benchmarks across two ta...