3.2 What Is a Token?
The previous chapter noted that a large language model does not generate an entire answer at once. It processes smaller text pieces step by step. These pieces are called tokens.
A token is not exactly one Chinese character or one English word. A common character may form a piece by itself; an English word may remain whole or be split. Numbers, punctuation, and spaces can also affect splitting. The exact rules belong to the tokenizer used by a model, so different models can count the same sentence differently.
Why should an everyday user know about tokens?
You do not need to count tokens by hand, but they affect three practical matters:
- How much input and output can fit.
- How some services measure usage or cost.
- Why a very long file may need splitting, summarizing, or filtering.
Think of tokens as small items packed into a suitcase. Your question, chat history, uploaded material, and answer all occupy room. Filling it with unrelated old material can crowd out useful facts; making everything extremely short can remove information needed for the task.
A product's character, word, and token counts are different measurements. When an exact allowance matters, use that product's documentation or counting tool rather than the estimate in this teaching lab.