Retrofitting language models to operate over bytes
Abstract Recent advances in artificial intelligence (AI) have largely been driven by large language models, deep neural networks that operate over discrete units called tokens. To represent text, most large language models use words or word fragments as the tokens, known as subword tokenization 1 . Subword tokenization...
Benjamin Minixhofer, Tyler Murray, Tomasz Limisiewicz et al.
· Nature · 1 citation