← back to the archiveCover illustration for “Token: the unit LLMs read, write and bill by”
TERMday 172·3d ago·Published ·by Andy Padia

Token: the unit LLMs read, write and bill by

In short: A token is a chunk of text, often part of a word, that language models process and price by, which is why it drives cost and limits.

A token is the basic unit of text a language model reads and generates: a whole word, part of a word, a punctuation mark or a space. Models do not see characters or words directly; a tokenizer splits text into tokens and maps each to a number. Context limits, speed and pricing are all measured in tokens.

Token in plain words

Lego bricks for text. Common words are one brick; rare words, code and many non-English scripts take several.

How it works

Tokenizers learn a vocabulary from training data, typically using byte-pair encoding: frequent character sequences become single tokens. For English prose, OpenAI's rule of thumb is that one token is roughly three-quarters of a word. Code, numbers, and languages such as Hindi or Tamil often need more tokens for the same meaning, which makes them costlier and uses more of the window.

Example

As an illustration, "Bengaluru" might split into several tokens such as "Beng", "al", "uru", while "the" is one. Exact splits differ by model: Anthropic's token counting documentation, read in November 2026, says its newer tokenizer produces about 30 percent more tokens for the same text than earlier models. So use the provider's token-counting tool rather than estimating from word counts when cost matters.

Where you'll meet it

  • API pricing per million input and output tokens
  • Context window sizes and output limits
  • Rate limits expressed in tokens per minute
  • Usage dashboards and invoices

What it changes for you

For developers, token counts drive cost, latency and what fits in context, so measure them. For product teams serving Indian languages, budget for higher token use per conversation. For finance, output tokens are usually priced higher than input tokens, so verbose responses cost more than long prompts.

Token vs word

A word is a human unit; a token is the model's unit. Token-to-word ratios vary by language, content type and model.

Common mistake

My view: estimating AI costs from word counts of English test data, then launching in Hindi or with code-heavy inputs. Count tokens on real traffic samples before you set a price.

Context window · Token pricing · Tokens per second · Cost per task · Embeddings

From the archive: Price AI work by accepted outcome, not by token

Count tokens on real traffic, in the languages you serve, before you set a price.

#field-guide#llm#tokens#cost

Sources

  1. help.openai.com/en/articles/4936856-understanding-and-counting-tokens
  2. platform.claude.com/docs/en/build-with-claude/token-counting
  3. arxiv.org/abs/1508.07909
← older drop
Large language model (LLM): what it is and how it works
newer drop →
Tool calling: how an LLM uses functions and APIs

related drops

explore all 557 drops →
← back to the archiveday 175