Tianchen Zhao

Convolution for Large Language Models

These results support depthwise convolution as a lightweight complement to self-attention for modeling short-range token interactions and suggest that the convolution makes repeated token IDs more sensitive to their immediate context.