Tokens Explained: What LLMs Actually See When You Type

Pramod Dutta · Jun 27, 2026 · 13 min read

You type a sentence into ChatGPT or Claude, hit enter, and a fluent reply streams back. It feels like the model is reading your words the way you do, one after another, left to right. It is not. Before a single layer of the neural network sees your prompt, your text is chopped into pieces called tokens and turned into a list of integers. The model never sees the letter a or the word cat. It sees numbers like [64, 5145, 382]. Everything the model knows about language, everything it predicts, happens in this numeric space. If you have ever wondered why an LLM miscounts the letters in a word, why your API bill is measured in "tokens" instead of words, why some languages cost three times more to process, or why the model sometimes fumbles a simple arithmetic problem, the answer almost always traces back to tokenization. This is the quiet layer that sits between human language and machine math, and understanding it changes how you write prompts, estimate costs, and reason about model behavior. In this article we will pull that layer apart, run real tokenizers on real strings, and build an intuition you can actually use.

What A Token Actually Is

A token is a chunk of text that the model treats as a single unit. It might be a whole word, a piece of a word, a single character, a space plus a word, or a punctuation mark. Tokens are not defined by grammar or dictionaries. They are defined by a statistical process that scanned enormous amounts of text and decided which sequences of characters are common enough to deserve their own entry in a fixed vocabulary.

Think of it like a compression scheme. If the sequence ing shows up millions of times across English text, it is wasteful to spell it out as three separate characters every time. The tokenizer learns to represent ing as one token. Rare sequences, on the other hand, get broken down into smaller and smaller pieces until they bottom out at individual bytes. This is why common words are usually one token while unusual words, technical jargon, or made-up strings get split into several.

Here is the mental model that matters most:

The model has a fixed vocabulary, often between 50,000 and 200,000 entries.
Every entry maps to exactly one integer ID.
Your text is converted into a sequence of these IDs before the model touches it.
The model predicts the next token ID, one at a time, and those IDs get converted back into text you can read.

So when people say a large language model "predicts the next word," that is a friendly simplification. It predicts the next token. Sometimes that token is a word. Often it is a fragment.

Watching A Tokenizer Work

Talk is cheap, so let us run one. The most common tokenizer library in the Python world is tiktoken, which powers OpenAI models, and Hugging Face's transformers library, which exposes tokenizers for open models like Llama and Mistral. Let us start with tiktoken.

import tiktoken

# Load the encoding used by GPT-4o and GPT-4o-mini
enc = tiktoken.get_encoding("o200k_base")

text = "Tokenization is sneaky."
ids = enc.encode(text)

print("Token IDs:", ids)
print("Token count:", len(ids))

# Decode each ID back into its text piece so we can see the split
for token_id in ids:
    piece = enc.decode([token_id])
    print(repr(piece), "->", token_id)

Running this prints something close to the following. The exact IDs depend on the encoding version, but the structure is what we care about:

Token IDs: [30643, 2065, 382, 74997, 13]
Token count: 5

'Token'      -> 30643
'ization'    -> 2065
' is'        -> 382
' sneaky'    -> 74997
'.'          -> 13

Look closely and a few things jump out. The word "Tokenization" became two tokens, Token and ization, because the model learned those as reusable pieces. The word "is" carries a leading space and shows up as ' is', not 'is'. That leading space is part of the token. The period is its own token. Five human-looking words and symbols became five tokens here, but that one-to-one ratio is a coincidence. Change the words and the ratio shifts immediately.

Why "Space Plus Word" Is A Token

The leading-space detail trips up almost everyone the first time they see it. Modern tokenizers based on byte-level Byte Pair Encoding treat the space before a word as part of that word's token. There is a good reason for it. In natural text, most words are preceded by a space, so bundling the space into the token saves the model from spending a separate token on every gap between words.

This has a practical consequence you can feel. The token for ' the' (with a space) and the token for 'the' (without) are different IDs entirely. So a word at the very start of a string can tokenize differently from the same word in the middle of a sentence. Watch this.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

print(enc.encode("hello"))        # start of string, no leading space
print(enc.encode("say hello"))    # 'hello' now has a leading space

You will often see the standalone hello produce a different set of IDs than the hello that follows say, because the second one is really the token ' hello'. This is not a bug. It is a direct result of how the vocabulary was built. When you are debugging why two near-identical prompts behave slightly differently, differences at the token boundary like this are a common culprit.

The Algorithm Underneath: Byte Pair Encoding

Most production tokenizers use a variant of Byte Pair Encoding, usually shortened to BPE. The idea is old and surprisingly simple, and once you see it you will understand why tokens look the way they do.

BPE builds its vocabulary through a training pass over a large text corpus. Here is the core loop in plain terms:

Start by treating every single byte (or character) as its own token. The starting vocabulary is tiny.
Scan the corpus and count how often each adjacent pair of tokens appears next to each other.
Find the most frequent pair. Merge those two tokens into a single new token and add it to the vocabulary.
Repeat steps 2 and 3 thousands of times, each pass adding one new merged token.
Stop when the vocabulary reaches the target size.

Because the most common pairs get merged first, frequent sequences like th, then the, then ' the' climb their way up into single tokens over many merge rounds. Rare sequences never get merged, so they stay as small pieces. This is the whole trick behind why common words are cheap and weird strings are expensive.

You can build a tiny, illustrative BPE from scratch to feel the mechanics. This is not production quality, but it shows the merge loop clearly.

from collections import Counter

def get_pair_counts(tokens):
    """Count how often each adjacent pair appears."""
    pairs = Counter()
    for a, b in zip(tokens, tokens[1:]):
        pairs[(a, b)] += 1
    return pairs

def merge_pair(tokens, pair, new_symbol):
    """Replace every occurrence of `pair` with a single new symbol."""
    merged = []
    i = 0
    while i < len(tokens):
        if i < len(tokens) - 1 and (tokens[i], tokens[i + 1]) == pair:
            merged.append(new_symbol)
            i += 2
        else:
            merged.append(tokens[i])
            i += 1
    return merged

# Start with individual characters
text = "low lower lowest slow slower"
tokens = list(text)

# Run a handful of merge steps
for step in range(5):
    pairs = get_pair_counts(tokens)
    if not pairs:
        break
    best_pair = max(pairs, key=pairs.get)
    new_symbol = "".join(best_pair)
    tokens = merge_pair(tokens, best_pair, new_symbol)
    print(f"Step {step + 1}: merged {best_pair} -> '{new_symbol}'")

print("Final tokens:", tokens)

Run it and you will watch the algorithm discover that l and o love being next to each other, then lo and w, gradually assembling the substring low into a single unit because it recurs across "low", "lower", and "lowest". That is BPE in miniature. Real tokenizers do this over billions of characters with careful handling of bytes so that any possible input, in any language or symbol set, can always be represented.

Different Models, Different Tokenizers

There is no universal tokenizer. Each model family ships with its own vocabulary, trained on its own data mix, and the same sentence can produce different token counts across them. This matters when you compare costs or port a prompt from one provider to another.

Let us compare an older OpenAI encoding with a newer one on the same string.

import tiktoken

text = "The quick brown fox jumps over the lazy dog. 你好世界 🚀"

for encoding_name in ["cl100k_base", "o200k_base"]:
    enc = tiktoken.get_encoding(encoding_name)
    ids = enc.encode(text)
    print(f"{encoding_name}: {len(ids)} tokens")

The cl100k_base encoding powers GPT-3.5 and the original GPT-4, with roughly a 100,000-token vocabulary. The o200k_base encoding powers the GPT-4o family and has about double the vocabulary. A bigger vocabulary can represent more sequences as single tokens, which often means fewer tokens for the same text, especially for non-English scripts. When you run the snippet you will typically see the newer encoding produce a lower token count on that mixed-language string, because its larger vocabulary swallows more of the Chinese characters and emoji into efficient chunks.

For open models, the Hugging Face transformers library gives you the exact tokenizer that ships with a given model.

from transformers import AutoTokenizer

# Load the tokenizer bundled with a specific open model
tok = AutoTokenizer.from_pretrained("gpt2")

text = "Tokenization is sneaky."
encoding = tok(text)

print("Token IDs:", encoding["input_ids"])
print("Tokens:", tok.convert_ids_to_tokens(encoding["input_ids"]))

The classic GPT-2 tokenizer uses a special character to mark leading spaces instead of a literal space, so you will see tokens rendered with a Ġ prefix where a space belongs. It is the same "space is part of the token" idea, just displayed differently. Every tokenizer family has its own conventions for showing this, which is exactly why you should always inspect the real tokenizer for the model you are using rather than assume.

Why Tokens Explain So Many LLM Quirks

Once you internalize that the model reads tokens and not characters, a whole catalog of strange behaviors suddenly makes sense. These are not mysterious failures. They are direct consequences of the tokenization layer.

Counting letters is genuinely hard for the model. When you ask "how many r's are in strawberry," the model does not see s-t-r-a-w-b-e-r-r-y. It sees a couple of tokens, perhaps straw and berry, as opaque integer IDs. The letter-level information is smeared inside those units and is not directly available. The model has to reconstruct spelling from patterns it learned, which is error prone. This is the real reason behind the famous "count the letters" failures, not a lack of intelligence.

Arithmetic can wobble because of how numbers split. Depending on the tokenizer, a number like 1000000 might become one token, or 100 and 0000, or several odd fragments. If the digits do not line up into clean place values as tokens, the model has a harder time doing column arithmetic. Newer tokenizers deliberately split numbers into consistent chunks (often groups of three digits) to help with this, which is itself evidence of how much tokenization shapes capability.

Some languages cost far more than others. Tokenizers are trained mostly on English-heavy data, so English words compress into few tokens. A language with a different script, or one underrepresented in training data, often needs many more tokens to express the same meaning. The same paragraph translated into a low-resource language can consume two or three times the tokens, which means it costs more via the API and eats more of the context window. That is a fairness and cost issue baked right into the tokenizer.

Made-up or glitchy strings behave oddly. Certain rare tokens that appeared in the training vocabulary but almost never in actual text can trigger weird, unstable outputs. These "glitch tokens" exist because the vocabulary was built by an automatic process that occasionally minted tokens for strings the model then barely learned to handle. They are a fingerprint of the tokenization process leaking into behavior.

Tokens, Context Windows, And Your API Bill

Everything about the economics and limits of an LLM is denominated in tokens, so this is where the concept stops being academic and starts affecting your wallet and your architecture.

The context window is the maximum number of tokens the model can consider at once, counting both your input and its output. When a model advertises a 128,000-token context, that budget covers your system prompt, the conversation history, any documents you paste in, and the response. Run over the limit and the earliest content gets truncated or the request is rejected. Because a token is usually less than a full word, you cannot simply count words to know if you will fit. You have to think in tokens.

A rough rule of thumb for English is that one token is about four characters, or roughly three-quarters of a word. So 1,000 tokens is somewhere around 750 words. But treat this only as an estimate. Code, punctuation-heavy text, numbers, and non-English scripts all break the ratio. The only way to know the true count is to run the tokenizer, which is exactly why the libraries above exist.

Here is a small utility that estimates cost from token counts, the kind of helper you end up writing constantly when building on top of these APIs.

import tiktoken

def estimate_tokens_and_cost(text, encoding_name="o200k_base",
                             price_per_million=0.15):
    """Estimate token count and dollar cost for a piece of text."""
    enc = tiktoken.get_encoding(encoding_name)
    token_count = len(enc.encode(text))
    cost = (token_count / 1_000_000) * price_per_million
    return token_count, cost

sample = "Summarize the following report in three bullet points. " * 50
tokens, cost = estimate_tokens_and_cost(sample)

print(f"Tokens: {tokens}")
print(f"Estimated input cost: ${cost:.6f}")

Notice what this makes concrete. Providers bill per million tokens, and input and output are usually priced differently, with output often costing more. If you send a long document to be summarized, you pay for every token going in and every token coming out. Batch prompts, retrieval-augmented context, few-shot examples, and long chat histories all add tokens, and tokens are the meter running the whole time. Engineers who understand tokenization write tighter prompts, trim redundant context, and choose models whose tokenizers are efficient for their language, and those choices show up directly on the invoice.

Practical Habits For Working With Tokens

Knowing the theory is one thing. Here are the habits that turn it into an everyday advantage when you are actually building.

Count before you send. For any feature that handles variable-length input, run the tokenizer and check the count against your model's limit before making the call. Do not guess from word counts.
Inspect the split when behavior is weird. If two similar prompts give different results, decode the token IDs for both and compare. Boundary differences, leading spaces, and unexpected merges are often the cause.
Reserve room for the output. The context window is shared between input and output. If you fill it entirely with input, the model has no room to respond. Leave a generous margin.
Match the tokenizer to the model. Do not estimate GPT-4o costs with a Llama tokenizer or vice versa. Load the exact encoding the target model uses, because counts differ.
Watch non-English and code carefully. These consume more tokens per unit of meaning. If your product serves multiple languages, measure token usage per language rather than assuming English ratios hold.
Prefer structure the tokenizer likes. Clean, common phrasing tokenizes efficiently. Exotic formatting, unusual unicode, and long runs of repeated symbols can inflate counts for little benefit.

None of these habits require deep math. They just require remembering, every time, that the model reads tokens, not characters and not words, and that those tokens are the unit of everything from behavior to billing.

Bringing It Together

Tokens are the hidden alphabet of large language models. Your text is split into pieces from a fixed vocabulary, each piece becomes an integer, and the model does all of its thinking in that numeric space before handing you back readable words. The splitting is done by Byte Pair Encoding, an algorithm that merges the most frequent character sequences into single units, which is why common words are cheap, rare strings are expensive, leading spaces ride along inside tokens, and different models disagree on how many tokens a sentence contains. Once you see this layer clearly, the puzzling behaviors stop being puzzling. Letter counting is hard because letters hide inside opaque tokens. Some languages cost more because their scripts were underrepresented when the vocabulary was built. Your bill is in tokens because tokens are what the model actually processes. And the context window is a token budget you share between what you ask and what you get back.

The best way to lock this in is to run the code in this article yourself. Load tiktoken or a Hugging Face tokenizer, feed it your own sentences, decode the IDs, and watch where the boundaries fall. A few minutes of poking at real tokenizers builds an intuition that no amount of reading can match, and that intuition pays off every time you write a prompt, estimate a cost, or debug a strange output.

If you want to go from understanding tokens to actually building reliable, cost-aware systems on top of language models, that is exactly what our AI Engineering Roadmap course on teachyou.ai is built for. It takes you from these fundamentals through prompt design, retrieval, evaluation, and production deployment, so the ideas here become tools you use to ship real applications. Start with tokens, and keep going.