Tokens and Tokenization in AI, Explained

Here's a small mystery that trips up almost everyone new to AI. You ask a chatbot to do something, and it mentions you've used a certain number of "tokens." Your bill, if you're paying for the tool, is measured in tokens. The amount of text the AI can handle at once is counted in tokens too. So what on earth is a token? It sounds like an arcade coin, but it's actually one of the most useful concepts to understand about how modern AI works, and once you get it, a lot of puzzling behaviour starts to make sense.

The short version: an AI doesn't read words the way you do. Before it can process your text, it chops everything into small pieces called tokens. This single step, known as tokenization, quietly shapes how much things cost, how much an AI can remember, and even why it sometimes stumbles over simple tasks. In this guide we'll explain tokens in plain English, show you why they matter in practice, and help you work with them rather than against them.

What a token actually is

A token is a chunk of text, and it's usually smaller than a whole word. Sometimes a token is a complete short word like "cat" or "the." Often it's a fragment, longer or unusual words get split into pieces. The word "unbelievable," for instance, might become "un," "believ," and "able." Spaces and punctuation count too. A rough rule of thumb in English is that one token is about four characters, and a typical word is a little over one token on average.

Why not just use whole words? Because language is endless, new words, names, typos, and slang appear all the time, and no fixed dictionary could cover them. By breaking text into smaller, reusable pieces, an AI can represent absolutely any text, even words it has never seen, by assembling them from familiar fragments. It's a bit like how a handful of letters can spell any word. This clever splitting is what lets large language models handle the messy reality of human writing.

Tokenization: the chopping step

Tokenization is simply the process of cutting text into those tokens before the AI does anything with them. It happens instantly and invisibly every time you send a message. The model never sees your sentence as a flowing line of words; it sees a list of numbered pieces. Everything the AI does next, all the pattern-matching of deep learning, happens on these tokens, not on the raw text.

Why tokens matter to you

Tokens aren't just a technical detail, they directly affect three things you'll notice when using AI: cost, capacity, and speed. Understanding this turns a confusing meter into a tool you can actually plan around.

That middle row is especially important. The amount of text an AI can consider at once, its context window, is counted in tokens, not pages or words. If you paste in a document longer than the window, the oldest part simply falls out of view, which is why a chatbot can seem to "forget" the start of a very long conversation.

Why AI generates text token by token

When an AI writes a reply, it doesn't compose the whole thing and hand it over. It predicts the most likely next token, adds it, then predicts the next one based on everything so far, and repeats. That's why you often see answers appear word by word, almost as if the AI is typing. Each token is a fresh prediction built on the transformer weighing all the tokens that came before.

This explains a lot of AI's character. It's why responses are fluent and contextual, each token fits what came before. It's also why mistakes can snowball: if an early token sends the answer down a wrong path, later tokens build on that wrong start. And because the model is predicting plausible text rather than verifying facts, it can occasionally produce confident errors, the phenomenon known as hallucination.

The quirks tokens explain

Ever asked an AI how many letters are in a word, or to count something, and watched it fumble a task a child could do? Tokens are often the culprit. Because the model sees "strawberry" as a couple of tokens rather than nine individual letters, character-level questions don't come naturally to it. It's not stupid, it simply never looks at text the way you assumed it did.

Tokenization also explains why some languages cost more to process than others. Languages and writing systems that the model wasn't heavily trained on can get split into many more tokens for the same amount of meaning, making them slower and pricier to handle. And unusual formatting, long strings of numbers, or rare symbols can balloon the token count unexpectedly. These are all consequences of how machine learning systems break text down before they ever start reasoning.

Working smarter with tokens

You can use this knowledge to your advantage. If you're paying per token, trimming rambling prompts saves money without hurting results. If you're feeding in a long document, summarising or splitting it helps you stay within the context window. And if an AI keeps losing track in a marathon conversation, starting a fresh chat clears out the old tokens crowding its memory. None of this requires technical skill, just an awareness of what's happening under the hood.

These foundations also sit beneath the broad, reusable foundation models that power today's tools, and they apply just as much to systems that handle images and audio, the world of multimodal AI, where pixels and sound get tokenised in their own way. For an honest sense of where all this still falls short, our piece on the limits of AI rounds out the picture.

A small idea with big consequences

Tokens are one of those concepts that, once understood, make AI feel far less mysterious. They sit at the very start of everything a model does, and they ripple through cost, capacity, speed, and behaviour. If you're thinking about how these tools might fit into real workflows, where token costs and limits become genuine planning concerns, our guide to an agentic AI business strategy is a useful next step, and you're welcome to get in touch for a closer look.

Frequently asked questions

References

1. Google AI. "Tokenization and Language Models." ai.google.
2. DeepLearning.AI. "How Language Models Process Text." deeplearning.ai.
3. Stanford HAI. "Foundations of Language Models." hai.stanford.edu.