32 Researchers Publish Comprehensive Survey on Tokenization for Modern NLP

A new comprehensive survey on tokenization has been published, compiled by 32 researchers over the past eight months. The work addresses a widely understudied area of language modeling that significantly impacts all of NLP.

Scope and Content

The survey provides an exhaustive overview of the field, covering:

• Core aspects: Algorithms, evaluations, multilinguality, encodings, and theory.
• Alternatives: Methods to replace traditional tokenizers, such as latent or visual tokenization.
• Adjacent topics: Constrained generation, token healing, and tokenizer security concerns.