NS-Filtered Fine-Tuning: 86.8% Compute Reduction with Equal or Better Benchmark Performance

Abstract

Modern language model training applies a uniform gradient update to every token, regardless of its information content. We introduce NS-weighted training, which filters training examples by their Shannon information. On a 50MB sample of the C4 web corpus, unigram NS-data filtering matches a 1000-step standard training run at step 132, reducing the number of gradient steps required by 86.8%. We further show that analytical initialization from a small teacher model, followed by only 4% of standard fine-tune steps, beats full standard training - a pipeline that reduces total training compute to ~30% of baseline. Our results suggest that the Shannon entropy of data, not raw token count, is the correct lower bound for gradient descent cost. For more information, please visit: Our website - ofmagnitude.com Our Github - https://github.com/OrdersOfMagnitudeLLC/type1-toolkit