LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers
We introduce methods to quantify how Large Language Models (LLMs) encode and store contextual information, revealing that tokens often seen as minor (e.g., determiners, punctuation) carry surprisingly high context. Notably, removing these tokens -- especially stopwords, articles, and commas -- consistently degrades per...
Anton Razzhigaev, Matvey Mikhalchuk, Temur Rahmatullaev et al.
· North American Chapter of th... · 18 citations
· ⚡2