
Token: the unit LLMs read, write and bill by
In short: A token is a chunk of text, often part of a word, that language models process and price by, which is why it drives cost and limits.
A token is the basic unit of text a language model reads and generates: a whole word, part of a word, a punctuation mark or a space. Models do not see characters or words directly; a tokenizer splits text into tokens and maps each to a number. Context limits, speed and pricing are all measured in tokens.
Token in plain words
Lego bricks for text. Common words are one brick; rare words, code and many non-English scripts take several.
How it works
Tokenizers learn a vocabulary from training data, typically using byte-pair encoding: frequent character sequences become single tokens. For English prose, OpenAI's rule of thumb is that one token is roughly three-quarters of a word. Code, numbers, and languages such as Hindi or Tamil often need more tokens for the same meaning, which makes them costlier and uses more of the window.
Example
As an illustration, "Bengaluru" might split into several tokens such as "Beng", "al", "uru", while "the" is one. Exact splits differ by model: Anthropic's token counting documentation, read in November 2026, says its newer tokenizer produces about 30 percent more tokens for the same text than earlier models. So use the provider's token-counting tool rather than estimating from word counts when cost matters.
Where you'll meet it
- API pricing per million input and output tokens
- Context window sizes and output limits
- Rate limits expressed in tokens per minute
- Usage dashboards and invoices
What it changes for you
For developers, token counts drive cost, latency and what fits in context, so measure them. For product teams serving Indian languages, budget for higher token use per conversation. For finance, output tokens are usually priced higher than input tokens, so verbose responses cost more than long prompts.
Token vs word
A word is a human unit; a token is the model's unit. Token-to-word ratios vary by language, content type and model.
Common mistake
My view: estimating AI costs from word counts of English test data, then launching in Hindi or with code-heavy inputs. Count tokens on real traffic samples before you set a price.
Related terms
Context window · Token pricing · Tokens per second · Cost per task · Embeddings
From the archive: Price AI work by accepted outcome, not by token
Count tokens on real traffic, in the languages you serve, before you set a price.


