Tokens and context windows, explained

Two words explain a lot of an assistant’s behaviour: tokens and the context window. Once they click, things like “why did it forget what I said earlier?” and “why did it cut off?” stop being mysterious.

What a token is

A model doesn’t read whole words; it reads tokens — short chunks of text. A token is often a word, but long or unusual words split into several, and spaces and punctuation count too. A rough rule for English: about three to four characters per token, or loosely three-quarters of a word. “Unbelievable” might be three tokens; “cat” is one.

You don’t need to count them. You need to know that everything — your prompt, the files you paste, and the model’s reply — is measured in tokens.

What the context window is

The context window is the maximum number of tokens the model can consider at once: your whole conversation so far, plus anything you’ve pasted, plus the answer it’s writing. It’s a working memory, not a hard drive.

The model can only “see” what fits in the context window. Anything older has effectively fallen out of view.

Why it “forgets”

When a long conversation exceeds the window, the earliest parts drop out of the model’s view. It isn’t ignoring you — that text is no longer in front of it. This is why a model can lose a detail you mentioned an hour ago, and why pasting a very long document can push out your actual instructions.

Practical consequences

• Put key instructions near your question, not buried far above, on long threads.
• Summarise instead of pasting everything. Give the model the relevant section, not the entire 80-page report, unless its window genuinely holds it.
• Start a fresh chat for a new task rather than dragging a bloated context along.
• Bigger windows help but cost more and don’t make the model read more carefully — relevance still beats volume, which is also good prompting.

Local models often have smaller windows than hosted ones, which is worth remembering if you run one yourself.