What Is Tokenisation: A Plain-English Definition
Tokenisation is the process by which text is converted into a sequence of tokens before being processed by a language model. A token is the basic unit of text that an LLM works with. It is not a character, and it is not necessarily a word. It is a sub-word unit determined by the tokeniser algorithm used when the model was trained. Common tokeniser algorithms include Byte Pair Encoding (BPE), used by GPT models, and SentencePiece, used by some other models. These algorithms were designed to find a balance between very granular character-level tokenisation (which produces very long sequences) and coarse word-level tokenisation (which struggles with morphological variation and rare words). The result is a vocabulary of typically 50,000-100,000 tokens that covers common English words, word fragments, punctuation marks, code syntax elements, and special characters. Some intuitions for English text: common short words like 'the', 'is', 'and' are usually single tokens. Longer common words like 'important' might be a single token or split into two. Rare or technical words are more likely to be split into multiple tokens. A word like 'tokenisation' might be represented as 'token', 'isation', which is two tokens. Numbers and special characters often consume more tokens relative to their character count than plain prose. Code is tokenised differently from prose. Programming language keywords are usually well-represented in model vocabularies. Variable names and string literals can vary widely. JSON with many curly braces, brackets, and commas can use significantly more tokens per character than plain text. Practically, for English prose, a common rule of thumb is that one token is approximately three to four characters, or roughly 0.75 words. A 1,000-word document is approximately 1,300-1,400 tokens. A 10-page PDF with no images might be 4,000-6,000 tokens depending on content density.
How Tokenisation Works
Tokenisation in modern LLMs uses a byte pair encoding approach. The training process starts with individual characters and iteratively merges the most frequent adjacent character pairs into single tokens until the desired vocabulary size is reached. The result is a fixed vocabulary that the model uses throughout its operation. When you send a prompt to an LLM, the tokeniser converts your text string into a sequence of token IDs from this vocabulary. These integer IDs, not the original text characters, are what the model actually processes. The model's attention mechanisms compute relationships between token IDs. At the output stage, the model produces probability distributions over the vocabulary, selects the next token ID, and the tokeniser converts it back to text. Different models use different tokenisers. OpenAI's models use the tiktoken tokeniser. Anthropic's Claude models use a similar BPE approach but with a different vocabulary. This means the same text produces different token counts on different models, which affects cost comparisons. You can explore tokenisation using OpenAI's Tokenizer tool, which shows you exactly how your text is split into tokens. This is genuinely useful for understanding why your prompts are costing more than expected or why certain phrases seem to behave oddly in model outputs. For a concrete UK example, consider a startup building a contract analysis tool. They need to process 20-page commercial contracts regularly. They test a sample contract and find it is 8,500 tokens using the tiktoken tokeniser for GPT-4o. Their analysis prompt adds another 1,200 tokens of system instructions. The total input is 9,700 tokens per contract. With expected output of 800 tokens for the analysis summary, each contract costs approximately USD 0.032 in inference at current GPT-4o pricing. At 200 contracts per month, that is USD 6.40 per month in inference for that feature. The tokenisation analysis tells them the cost model works. Without this analysis, they might have underestimated costs significantly by assuming 'token roughly equals word'.
Why Tokenisation Matters for AI Product Development
Tokenisation matters for product development in four specific ways: cost estimation, context window management, prompt design, and multilingual behaviour. Cost estimation accuracy depends on token counting. Every pricing model for LLM inference is per token. If you estimate costs per word or per character rather than per token, your estimates will be wrong. For most English text, the discrepancy is modest. For code, JSON, or non-English languages, it can be significant enough to materially affect your product economics. Context window management requires understanding token limits. Every model has a maximum context window measured in tokens. GPT-4o has a 128,000 token context window. Claude 3.5 Sonnet has a 200,000 token context window. When your input plus expected output exceeds this limit, the request fails. For products processing long documents, you must either truncate inputs, split documents into chunks (and process each chunk separately), or use a model with a sufficiently large context window. Chunking strategy depends on understanding how many tokens each chunk will consume. Prompt design affects token efficiency. Verbose system prompts with extensive instructions cost more per request than concise ones. If your system prompt is 2,000 tokens and you are making 100,000 API calls per month, that is 200 million tokens of system prompt cost alone. Prompt compression, which rewrites prompts to convey the same information in fewer tokens, can meaningfully reduce costs at scale. Multilingual behaviour differs because non-English languages are often tokenised less efficiently. Languages with complex morphology, characters outside the standard Latin set, or scripts like Devanagari, Chinese, or Arabic can consume significantly more tokens per word than English. A 1,000-word document in Japanese might require three to four times as many tokens as the same document in English. This has direct cost and context window implications for products serving non-English markets.
Common Use Cases in Production AI Products
Token counting tools are embedded in almost every production LLM integration. Before sending a request to a model, production systems count the tokens in the prompt and projected output to verify the total stays within the context window and to log costs per request for billing and monitoring purposes. Libraries like tiktoken for OpenAI models and equivalent tools for other providers are standard dependencies. Chunking strategies for document processing depend entirely on tokenisation. When a document is too long to process in one inference call, it is split into chunks that fit within the context window. Good chunking respects semantic boundaries, such as paragraph or section breaks, rather than splitting at arbitrary token counts. The chunk size is defined in tokens, and the overlap between chunks (repeated content at the boundaries to maintain context) is also measured in tokens. Streaming output at the token level is how LLM APIs deliver progressive responses. Each token is transmitted as it is generated, which is why streaming responses appear word by word rather than all at once. Products implementing streaming need to handle the token stream at the application layer, buffering tokens into displayable words and sentences for the user interface. Cost attribution in enterprise AI products requires accurate token counting per user, per feature, or per customer account. Tracking input and output tokens per request and attributing them to the appropriate billing dimension is standard in multi-tenant AI SaaS products. This data feeds into both billing systems and cost monitoring dashboards. Context window optimisation for long-running agent conversations compresses or summarises earlier parts of a conversation when the growing history approaches the context window limit. Token-aware compression that identifies the minimum representation of earlier context needed to maintain conversation continuity is a practical technique in production conversational AI products.
Related Concepts
Context window is the most practically important concept connected to tokenisation. The context window is measured in tokens, and it determines the maximum amount of text, including both the prompt and the response, that an LLM can process in a single interaction. Every context window management strategy, from chunking to summarisation to retrieval augmentation, depends on understanding how many tokens the content you want to process will consume. Large language models determine the tokenisation scheme used. Different models use different tokenisers with different vocabularies. When switching between models or using multiple models in a product, you cannot assume the same text produces the same token count. Always count tokens using the specific tokeniser for the model you are using. Inference cost is directly determined by token count. Input tokens and output tokens are priced separately at different rates. Output tokens are more expensive because they are generated serially, one at a time. Understanding the token structure of your typical prompts and expected outputs is the foundation of accurate inference cost modelling. Embedding models are also subject to tokenisation, though with different token limits than generative models. Text embedding models like text-embedding-3-small have input token limits per API call, and embedding costs are also billed per token. For bulk indexing of large document collections, token counting is part of both cost planning and chunking strategy. Fine-tuning costs are calculated in training tokens, which is the total number of tokens in your training dataset. A dataset of 500 examples with an average of 800 tokens each is 400,000 training tokens. Understanding this calculation helps estimate fine-tuning costs accurately before committing to a training run.