VC Dimension and Expressivity of Real-Valued Transformers
This work studies softmax-attention, multi-layer transformers operating on real values, with very few additional assumptions, and proves limitations on how many bits of a real number a transformer can access.
Gavin Dooley, Andy Yang, Y. Zhu et al.
· 0 citations