Two test-time monitoring strategies are proposed: reasoning expression monitoring and hidden states monitoring, that reduce token usage by 62.7-93.6%, substantially improving efficiency and reliability while largely preserving accuracy.
Qingjie Zhang, Yujia Fu, Yang Wang et al.· 2 citations· ⚡1
Quantile-Guided Density Estimation (QGDE), which approximates this distribution with multiple quantile trends and uses local density weighting to produce token-level estimates and suggests that released tokenizer vocabularies provide a useful signal for fine-grained corpus estimation beyond coarse composition inference.
Qingjie Zhang, Xingzhang Ren, Zi-Xuan Chen et al.· 0 citations
This work proposes Sampled-BPE, a lightweight token-level auditing pipeline that sample a small subset and train BPE tokenizer to surface polluted tokens, and releases a hierarchical Chinese web token dataset with 660k+ token records, organized as trees to support review and tracing of pollution.
Qingjie Zhang, Ziqi Tang, Jie Zhang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.