Tokenization
Definition
Tokenization (tokenisation in British spelling) is the process of splitting text into units called tokens, which a language model converts into numerical identifiers before processing. Tokens are often whole words, parts of words or punctuation.
Why it matters
Models read, limit and price text in tokens rather than words. Tokenization affects how much information fits within a Context Window, how rare names and terms are represented, and how efficiently different languages are processed.
How it works
Subword vocabularies
- Most modern models use subword methods, such as byte-pair encoding or similar algorithms, built from frequent character sequences in training data.
- Common words may be single tokens, while rare names can be split into several pieces.
Mapping to identifiers
- Each token maps to an identifier in a fixed vocabulary, which is then converted into a vector representation.
Practical consequences
- Context limits and usage costs are usually expressed in tokens.
- Different tokenizers split the same text differently, so token counts vary between models.
How Netsleek uses the term
Netsleek uses Tokenization to explain practical constraints, such as why long pages may be truncated within context limits, and why distinctive brand names may be split into multiple tokens. It treats this as a technical constraint, not a ranking factor.
Comparisons
- Tokenization vs Word count: Word count measures words. Token count measures model units and is usually higher than the word count.
- Tokenization vs Semantic Chunking: Tokenization splits text into model units. Semantic chunking splits content into meaningful passages for retrieval.
How Tokenization Connects Across AI Systems
Tokenization does not operate in isolation. It connects to other concepts across the AI search and visibility process:
- Feeds models: Produces the input processed by Transformer models.
- Constrains context: Determines how much text fits in a Context Window.
- Operates before attention: Tokens are the elements Attention Mechanisms act upon.
Related glossary concepts
Common misinterpretations
- A token is not the same as a word.
- Tokenization is not a visibility or ranking signal.
- Being split into many tokens does not mean a brand name cannot be recognised.
Summary
Tokenization converts text into the units language models process. It shapes context limits, cost and how rare terms are represented, but it does not decide relevance or selection.