The word "token" appears in nearly every conversation about AI pricing, model capability, and architectural choices, but the term is rarely defined before it gets used. Reading the major AI providers’ documentation, listening to AI conference talks, or scanning the changelogs for any of the frontier models, you will find tokens treated as common knowledge. They are not. Tokens are a specific technical concept with specific implications, and confusion about what they are produces real cost mistakes, real capacity-planning errors, and real disagreements about model behavior that mostly evaporate once both sides are using the same definition.
This piece walks through what tokens actually are, how they get produced from text, why they matter for the practical decisions you make when building or buying AI capability, and how the tokenizers differ across the major model families. It is a foundational pillar for the Digital Matters AI coverage, intended to be referenced from any later post that uses the term. The goal is that after reading this, you can read any AI pricing page, any model context-window comparison, or any provider’s rate-limit policy and know exactly what is being measured.
The short version is that tokens are the small text chunks that large language models actually process, where one token is approximately three to four characters of English text or roughly three-quarters of an English word on average. The exact size varies by tokenizer, language, and content type. Tokens are the unit that AI providers charge in, the unit that context windows are measured in, the unit that rate limits are enforced in, and the unit that determines whether a given prompt will fit through the model at all. Almost every operational decision in AI workloads ultimately reduces to a question about tokens.
Why models work in tokens rather than words
The natural human unit for text is the word, and a reasonable question is why models do not work in words directly. The short answer is that words are a poor unit for the actual problem the model is solving, which is statistical pattern recognition over text at a level that is finer than whole words but coarser than individual characters.
Working in whole words has two problems. The first is vocabulary size: the set of all distinct words in a language is large (English alone has hundreds of thousands of words in active use, with many more in technical or specialized vocabularies), and the model would need to learn a representation for every one of them, which is expensive in both training data and model capacity. The second is that whole-word vocabularies cannot handle words the model has not seen before. A new technical term, a misspelling, a brand name, or a coined word would have no representation, and the model would fail on it.
Working in individual characters solves both problems but creates a different one. Characters carry very little semantic information on their own. A model working at the character level has to learn long-range patterns just to assemble individual characters into recognizable words, before it can even start to reason about word combinations. This requires much larger model sizes and much more training to achieve the same capability.
Tokens are the compromise. They are units finer than whole words but coarser than characters. A common word like "the" or "and" is typically one token. A long word like "tokenization" might be broken into two or three tokens. A rare word, a brand name, or a non-English word might be broken into many small tokens or even individual characters. The vocabulary size for a typical modern tokenizer is in the range of 100,000 to 200,000 distinct tokens, which is large enough to give common patterns their own representation but small enough to be tractable to train on. The decomposition pattern means that any text, including text the tokenizer has never seen before, can be represented as a sequence of tokens by falling back to smaller and smaller pieces.
How tokenization actually works
The dominant tokenization approach in modern LLMs is called Byte-Pair Encoding, usually abbreviated BPE, with a closely related variant called SentencePiece that some models use. The differences between BPE and SentencePiece are technical and mostly inconsequential for understanding token behavior in practice. The shared idea is the same: start with characters, find the most common adjacent pairs in the training corpus, merge those pairs into single tokens, repeat until the vocabulary reaches the target size.
The process produces a token vocabulary that is data-driven rather than rule-based. Common patterns become single tokens because they appear often in the training data. Uncommon patterns get broken into smaller pieces because the merging process did not get to them within the vocabulary budget. The vocabulary reflects the training corpus: a tokenizer trained on English-heavy text will have efficient single-token representations for common English patterns and inefficient multi-token representations for non-English patterns. A tokenizer trained on multilingual text will be more balanced but may be less efficient on any single language.
A concrete example helps. The English sentence "The cat sat on the mat" tokenizes through OpenAI’s o200k_base tokenizer (used by GPT-5.5) as approximately 6 tokens: "The", " cat", " sat", " on", " the", " mat". The leading space on each token after the first is preserved as part of the token, which is a quirk of how most modern tokenizers handle word boundaries. The same sentence in French ("Le chat est assis sur le tapis") tokenizes as approximately 9 tokens because the tokenizer was trained on less French than English. The same sentence in Japanese tokenizes as substantially more tokens for the same reason, often individual character tokens because the tokenizer has fewer multi-character Japanese patterns in its vocabulary.
This language asymmetry has real operational consequences. The same query content costs more to process in some languages than in others, simply because the tokenizer is more efficient on the languages dominant in its training corpus. Multilingual applications need to plan for the token-count variance across languages they serve.
How many tokens are in a given amount of text
The rule of thumb for English text is that one token is approximately three to four characters or roughly three-quarters of a word on average. A 1,000-word English document is approximately 1,300 to 1,500 tokens. A 100,000-word book is approximately 130,000 to 150,000 tokens. This rule of thumb works for most prose. It breaks down for content with unusual characteristics.
Code tokenizes differently from prose. A typical Python or JavaScript file produces more tokens per character than English prose because code has more punctuation and shorter identifiers that often get split into multiple tokens. A rough working multiplier is that code tokenizes at approximately 1.2 to 1.5 times the rate of equivalent-length English prose.
Tables and structured data tokenize even more densely. CSV data, JSON data, and Markdown tables have repeated delimiters that take their own tokens, and structural punctuation accumulates faster than prose. A spreadsheet-sized table can tokenize to several times the token count of equivalent prose covering the same information.
Non-English text tokenizes differently depending on the language. Languages with Latin alphabets (French, Spanish, German, Italian) typically tokenize at 1.2 to 2 times the English rate for the same information. Languages with non-Latin alphabets (Russian with Cyrillic, Greek, Hebrew, Arabic) often tokenize at 2 to 3 times the English rate. Languages with no clear word boundaries (Chinese, Japanese, Korean) often tokenize at the character level, which can be 3 to 5 times the rate of equivalent English.
Numbers tokenize unpredictably. A long number like "987654321" might be a single token in some tokenizers, three to four tokens in others, or nine individual character tokens depending on what the tokenizer’s vocabulary captured during training. This unpredictability is a known issue and the source of many "the model is bad at math" observations. Models are not necessarily bad at math; they may be bad at the specific number formats their tokenizer has fragmented poorly.
How tokenizers differ across major model families
Each major model family has its own tokenizer. The tokenizers are mostly similar in approach but differ in specifics, and the differences matter for cost comparison and capability evaluation.
OpenAI’s models through GPT-4 used the cl100k_base tokenizer with approximately 100,000 tokens in vocabulary. GPT-5 and GPT-5.5 use the o200k_base tokenizer with approximately 200,000 tokens. The larger vocabulary in o200k_base produces fewer tokens for the same text, which means GPT-5.5 charges per a smaller token count for the same prompt than GPT-4 would have. The price-per-token comparison between models with different tokenizers is misleading without accounting for this; the right comparison is price-per-input-character or price-per-prompt-equivalent.
Anthropic’s Claude models use a tokenizer that was originally derived from the OpenAI cl100k_base approach but has diverged through Claude-specific training. The Claude tokenizer is approximately competitive with o200k_base on English efficiency and slightly different on code and structured data. The practical implication is that the same prompt sent to Claude and to GPT-5.5 will tokenize to slightly different counts, with Claude often producing 5 to 15 percent more tokens for equivalent English content.
Google’s Gemini models use the SentencePiece variant with a vocabulary of approximately 256,000 tokens. The Gemini tokenizer is more efficient for many non-English languages than the OpenAI or Anthropic tokenizers because Google trained it on a more aggressively multilingual corpus. The practical implication is that Gemini’s per-token pricing applies to a smaller token count for non-English prompts than the competitor pricing applies to the same prompts through their tokenizers.
Meta’s Llama models use a SentencePiece tokenizer with a vocabulary around 128,000 tokens (for Llama 4). The Llama tokenizer is moderately efficient on English and less efficient on other languages.
The tokenizer differences mean that you cannot meaningfully compare two models’ token-based pricing without also accounting for how each tokenizer counts. A model that charges $5 per million input tokens but uses a tokenizer that produces twice as many tokens for the same text is more expensive than a model charging $8 per million input tokens with an efficient tokenizer.
Input tokens, output tokens, and why they cost differently
Modern AI providers charge separately for input tokens and output tokens, and the rates are not equal. Output tokens are universally more expensive than input tokens, typically by a factor of 4 to 5. The pricing structure reflects the asymmetric computational cost.
Input tokens are processed in parallel during the model’s encoding step. The model reads the input prompt all at once, computes its attention patterns across the input, and produces an internal representation. The compute cost scales linearly with input length, but the parallelism means the wall-clock time is short and the GPU utilization is efficient.
Output tokens are generated one at a time. The model produces a token, the system feeds it back into the context, the model produces the next token, and so on. The compute cost is similar per output token as per input token, but the serial nature means GPU utilization is less efficient and the wall-clock time scales with output length. The provider passes the higher operational cost through as the higher per-output-token price.
The practical implication is that prompt design matters for cost. A workload that produces long responses to short prompts is dominated by output token cost. A workload that produces short responses to long prompts is dominated by input token cost. The optimization patterns differ. For long-response workloads, the right move is often to reduce response length through prompt engineering or output truncation. For long-prompt workloads, the right move is often to reduce prompt length through summarization or compression of the input.
A specific consequence is that asking the model "explain your reasoning" or "show your work" is meaningfully more expensive than asking for a terse answer, because the explanation is output tokens. The newer "reasoning" features in Opus 4.7 and later, and the equivalent in GPT-5.5 and Gemini 3 Pro, add a third pricing dimension: "thinking tokens" or "reasoning tokens" that are generated internally and may be billed at the output rate even though they are not part of the visible response. The exact billing for thinking tokens varies by provider and is worth checking against the provider’s current pricing documentation when designing workloads that use the feature heavily.
Context windows in token terms
A model’s context window is the maximum number of tokens it can hold in active context for a single request. The number includes both the input prompt and the output response. A model with a 200,000-token context window that has been given a 195,000-token prompt has only 5,000 tokens of room left for the response.
Context windows have grown dramatically through the 2024 to 2026 cycle. GPT-4 launched with a 32,000-token window. GPT-4 Turbo expanded to 128,000 tokens. GPT-5.5 supports 256,000 tokens. Claude Opus 4 launched with 200,000 tokens. Opus 4.7 expanded to 500,000 tokens. Opus 4.8 supports up to 1,000,000 tokens for enterprise customers. Gemini 3 Pro supports 2,000,000 tokens with measured 95+ percent recall at the limit.
The practical implication is that what counts as a "large" prompt has shifted. A workload that needed elaborate chunking and summarization to fit through GPT-4’s 32,000-token window in 2023 fits trivially through any current frontier model’s window in 2026. The architectural patterns built around context-window limits are increasingly obsolete for workloads that fit within the new windows.
The cost implication is that long-context workloads are not free even when the context window is large. A 500,000-token Opus 4.7 prompt costs approximately $7.50 in input tokens at current rates, before any output. A 1,000,000-token Gemini 3 Pro prompt costs approximately $2.50 in input tokens. The capability to send the long prompt does not eliminate the cost of doing so.
A subtle implication is that context-window comparisons across models are also tokenizer-dependent. A 200,000-token context window in a tokenizer that produces 1.5x the tokens of the comparison tokenizer is equivalent to a 133,000-token context window in the more efficient tokenizer. Like pricing, context-window comparison is most meaningful in terms of equivalent input content rather than token count.
Rate limits in token terms
AI providers enforce rate limits in tokens per minute, not requests per minute. The token-based limit reflects the fact that the underlying compute cost is per-token, not per-request. A single very large request that processes a million input tokens places similar load on the provider’s infrastructure as ten thousand small requests that each process a hundred tokens.
The typical rate-limit dimensions are input tokens per minute, output tokens per minute, and requests per minute. The first two are the primary capacity controls; the third is a defense against pathological micro-request patterns. Production applications usually hit the token-based limits before the request-based limit, and the right capacity planning is to monitor the token-based limit and budget accordingly.
Different model tiers have different default limits. The frontier models typically have lower default limits than the cheaper models because the frontier models cost more to serve and the providers want to enforce explicit capacity grants for high-volume use. A new account using GPT-5.5 might be limited to 30,000 input tokens per minute by default, while the same account using GPT-5.5-mini might be limited to 200,000 input tokens per minute, and the cheaper Gemini 3 Flash even higher. Quota increases for the frontier models typically require explicit requests to the provider with business justification.
Counting tokens before sending requests
Several libraries support counting tokens locally without making an API request. This is essential for cost estimation, prompt engineering, and quota planning.
For OpenAI tokenizers, the tiktoken library (Python) or js-tiktoken (JavaScript) provides the canonical implementation. A typical use is to count the tokens in a prompt before sending it: encoding.encode(prompt).length returns the token count. The same library supports the older cl100k_base and the newer o200k_base, so it works for GPT-4 and GPT-5.5 workloads alike.
For Anthropic tokenizers, the anthropic-tokenizer Python library (officially supported) provides a similar interface. The library was released alongside Claude 3 and is the canonical way to count tokens for any Claude model.
For Google’s Gemini tokenizers, the google-generativeai package includes a count_tokens method on the model object that does the counting through the API (using the API endpoint that returns token counts without actually invoking the model). The pure-local alternative is to use the sentencepiece library with the Gemini tokenizer model file, though Google has not historically distributed the tokenizer model file as widely as OpenAI and Anthropic have.
For Llama tokenizers, the transformers Hugging Face library provides the tokenizer through AutoTokenizer.from_pretrained("meta-llama/Llama-4-8B") or similar. This works for any open-weight model whose tokenizer is published on Hugging Face.
The practical pattern for production applications is to count tokens before sending requests, log the counts alongside the requests in observability, and use the historical data for capacity planning and cost forecasting. Token counts are also the basis for prompt-caching decisions, response-truncation decisions, and quality-evaluation work.
Strategies for token efficiency
Several patterns reduce token consumption without sacrificing capability. The first is prompt design. Tight, well-structured prompts produce fewer tokens than verbose prompts and often produce better responses too. Removing redundancy, eliminating examples that the model does not actually need, and shortening descriptions all reduce input cost.
The second is structured output. Asking the model to produce JSON with a specified schema produces shorter responses than asking for prose explanations. The structured output format also reduces the number of subsequent tokens needed to parse or validate the response.
The third is caching. If the same prompt or prompt prefix is sent repeatedly, prompt caching (supported by all major providers as of 2026) can substantially reduce cost. The cached portion of the prompt is billed at a fraction of the normal rate (typically 10 percent of the input rate), and the savings compound for high-volume workloads.
The fourth is tier-aware routing. The cheaper tier of each model family (Claude Haiku, Gemini 3 Flash, GPT-5.5-mini) is sufficient for many workloads that do not require the frontier capability. Routing simple requests to the cheap tier and reserving the expensive tier for hard requests is a standard pattern for cost-sensitive applications.
The fifth is model-family selection. The tokenizer differences across model families mean that some content tokenizes more efficiently in one family than another. Workloads dominated by non-English content may be cheaper through Gemini than through OpenAI even at similar per-token prices, because Gemini’s tokenizer produces fewer tokens for the same content.
The sixth is response length control. Setting an explicit max_tokens parameter on the response prevents the model from over-explaining. The right max_tokens varies by use case, but most applications benefit from setting a cap that is moderately tight rather than leaving it open.
Special tokens
In addition to the text tokens described above, models use a small set of special tokens that have specific structural meaning. These include the beginning-of-sequence token, the end-of-sequence token, the role-marker tokens that distinguish system, user, and assistant messages in a chat context, and tool-use tokens that signal when the model is requesting a tool invocation.
Special tokens count against the token budget like any other token. A typical chat-format prompt has overhead from the role markers that adds approximately 4 to 8 tokens per message. For applications with many short messages, this overhead can be a non-trivial fraction of the total token consumption.
The tool-use tokens deserve specific mention. When a model invokes a tool through the function-calling API, the tool call itself, the tool’s response, and the model’s subsequent processing all count in the token budget. A tool-heavy workflow that invokes ten tools per response has substantially more token overhead than a workflow without tools, even if the final visible response is the same length.
Multimodal tokens
For models that handle images, audio, or video in addition to text, the non-text inputs are also represented in tokens. The conversion is model-specific and depends on the input modality.
For images, the typical pattern is that the model converts the image into a fixed set of "image tokens" using a vision encoder. The token count per image varies by model and by image size: Gemini 3 Pro uses approximately 258 tokens per image for the default resolution, with larger images using more tokens. Claude Opus 4.8 uses approximately 1,500 tokens per image for the default resolution. GPT-5.5 uses approximately 765 tokens per image. These image tokens are billed at the input token rate, so processing images is meaningfully more expensive than processing equivalent-information text.
For audio, the pattern is similar but with substantially more tokens per second of input. A typical conversion is 50 to 100 tokens per second of audio, which means a 5-minute audio file produces 15,000 to 30,000 tokens of input. This is why audio workloads, especially transcription and long-form audio analysis, can run up token bills quickly.
For video, the pattern is image-tokens-per-frame plus optional audio-tokens-per-second. A 1-minute video at one frame per second produces 60 image-token batches plus audio tokens, which can easily reach 50,000 to 100,000 tokens for a minute of input. Video analysis is the most token-intensive multimodal workload by a substantial margin.
Frequently asked questions
How many tokens are in a typical email? A short email (50-100 words) is approximately 70-150 tokens. A medium email (200-300 words) is approximately 300-450 tokens. A long email or thread (1,000 words across multiple messages) is approximately 1,400-1,500 tokens.
How many tokens are in a typical novel? A 90,000-word novel is approximately 120,000-135,000 tokens. Most novels fit within a 200,000-token context window. Long novels (200,000+ words) require longer context windows or chunking.
Why does the same model produce different token counts for the same text on different days? It shouldn’t, with the same tokenizer version. If you see token-count drift, it usually means the provider has updated to a newer tokenizer or that the request is being routed to a slightly different model variant. Both are unusual but happen during major version transitions.
Are tokens the same as words? No. A token is approximately three-quarters of a word on average for English text, but the relationship is not exact. Some common words are single tokens; some words are multiple tokens; some tokens are partial words or punctuation. Word-based estimates are useful for back-of-envelope calculations and not reliable for precise budgeting.
Why does my prompt produce more tokens than I expect? Common causes are: structured formatting (markdown, JSON, code blocks) that adds delimiter tokens, repeated examples that take tokens for the formatting overhead, non-English content that tokenizes less efficiently, and chat-format role-marker overhead. Counting tokens with the tokenizer library before sending often reveals where the unexpected tokens are coming from.
Do tokens used in fine-tuning count the same as tokens used in inference? They are counted similarly but priced differently. Fine-tuning charges per token in the training data, typically at a higher rate than inference, because the compute cost of training is much higher. The exact pricing varies by provider and is on a separate line item from inference pricing.
Can I see the actual tokens my prompt produced? Yes, with the tokenizer library. The encoding.decode(encoding.encode(prompt)) round-trip will reproduce the original text, and encoding.encode(prompt) returns the token IDs that you can inspect individually. Some providers also have a tokenizer playground (OpenAI’s, Anthropic’s) that visualizes the tokenization in a browser interface.
How do tokens interact with prompt caching? Cached portions of a prompt are still counted but billed at a reduced rate (typically 10 percent of the normal input rate). The total token count is unchanged; the cost is reduced. Cache eligibility depends on the provider’s caching rules, which typically require the cached portion to be a prefix of the prompt and to be at least a minimum number of tokens (usually 1,024 tokens or 2,048 tokens for the major providers).
Will the cost-per-token continue to drop over time? Probably, on a per-capability basis. The cost-per-token of frontier models in 2026 is meaningfully lower than the cost-per-token of frontier models in 2024, even though the 2026 models are more capable. The trend is for capability to compound and prices to decline at the same time. Whether a specific application gets cheaper to run over time depends on whether the application can move to newer model versions as they ship.
