Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning
This work characterize task-space complexity through covering numbers under a prescribed metric, thereby quantifying the low-dimensional cross-task structure without requiring an explicit parametric representation and explicitly construct a Transformer that exploits their low-dimensional structure to perform ICL.