The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Era
Although performance of language-based models is improved by scaling, whether the gap to a structure-aware architecture can eventually be eliminated remains untested.