What is a token in artificial intelligence?
A token is the unit of text a language model works with: a short word, a piece of a word or a punctuation mark. The model reads and writes token by token, its context window is measured in tokens and AI APIs charge per token. In English, a token is roughly four characters.
Key takeaways
- A token is a piece of text: a short word, part of a long word or a punctuation mark. The model does not see letters or words, it sees tokens.
- The rule of about four characters per token from OpenAI and Google is built for English: for any other language, or for text full of codes, measure it with each provider's counter.
- Everything that goes into a request, the instructions, the product page and the images, and everything that comes out, the response, is counted in tokens and has a limit: the context window.
- APIs charge per million tokens, with one price for what goes in and another for what comes out.
- The same text does not produce the same tokens in every model: when you switch models, you count again.
If you search for "tokens" you will get cryptocurrencies, NFTs and the access tokens an app asks for to connect to your store. This is about something else: the unit in which a language model reads, writes and bills. It matters to a store as soon as it uses a tool that writes product pages with an LLM (large language model) and its own API key, because the invoice and the limits arrive in tokens, and also when an assistant reads your page, because what fits of it is measured the same way.
How it works
OpenAI explains it with an example: tokens are sequences of characters that show up frequently. The word tokenization is split in two, "token" and "ization," while a short, common word like the is a single token. In a sentence, the first token of each word usually starts with the space before it.[1]OpenAI, API documentation. Key concepts That is why a model sees neither single letters nor whole words: it sees chunks, and long or uncommon words get split into more chunks than short, common ones.
For quick math, both sets of documentation give the same rough rule. OpenAI: a token is about 4 characters or 0.75 words in English text.[1]OpenAI, API documentation. Key concepts Google: a token is about 4 characters, and 100 tokens equal between 60 and 80 English words.[2]Google AI for Developers, Gemini API documentation (updated September 23, 2026). Understand and count tokens Both say it about English. For a catalog in another language, or one full of codes and part numbers, the right figure is the one the counter of the model you use gives you, not the rule.
Not everything that is counted is text. Gemini also turns images, video and audio into tokens: 258 tokens per image if it is no more than 384 pixels on a side; larger ones are divided into 768 × 768 pixel tiles and each costs another 258. Video runs about 100 tokens per second and audio, 32 per second.[2]Google AI for Developers, Gemini API documentation (updated September 23, 2026). Understand and count tokens The product photos you send to a model to describe also count.
And every model slices in its own way. Anthropic warns that the tokenizer of its most recent models produces around 30% more tokens for the same text than that of earlier ones, and asks you to count again with the model you are going to use instead of reusing old counts. It adds that its pre-count is an estimate and that actual usage may differ slightly.[3]Anthropic, Claude documentation. Token counting
Why they count: the limit and the bill
The first reason is room. In a model that generates text, the request and the response together cannot exceed the model's maximum context length.[1]OpenAI, API documentation. Key concepts That maximum is the context window, and it is measured in tokens. Embeddings models, which turn a text into a vector to search by meaning, have theirs: for OpenAI's, the input takes at most 8,192 tokens.[4]OpenAI, API documentation. Vector embeddings[1]OpenAI, API documentation. Key concepts
The second is money. OpenAI publishes its prices per million tokens, with separate columns for input tokens, cached input tokens and output tokens.[5]OpenAI, API documentation. Pricing Anthropic presents its token counter precisely as a way to manage costs and usage limits before sending anything.[3]Anthropic, Claude documentation. Token counting In practice, a long product description costs more twice: when it is read and when it is written.
In a store
A fictional example: a store asks a model, with its own API key, to write the description of a backpack. This is what consumes tokens in that request, and who counts it:
| Piece of the request | What it is | How it is counted |
|---|---|---|
| Instructions | The brand voice and the style rules | Input tokens, with the model's tokenizer |
| The current product page | Title, attributes and description | Input tokens; a long text or one full of codes is split into more chunks |
| Two photos of the product | Images for the model to describe | In Gemini, 258 tokens per small image or per 768 × 768 tile[2]Google AI for Developers, Gemini API documentation (updated September 23, 2026). Understand and count tokens |
| The response | The new description | Output tokens, with their own price[5]OpenAI, API documentation. Pricing |
Multiply that by every product in the catalog and every time you regenerate it, and the bill stops being a technical detail. That is why it makes sense for a tool to limit the context it sends in each request. Clione, for example, works with the model and API key the store configures and, when the store turns it on, attaches to each product's prompt the four most relevant excerpts of its documents, about 1,200 tokens.
What people often get wrong
That a token is a word and that counting words is enough to work out what a catalog will cost. A long word is split into several tokens, like tokenization into two,[1]OpenAI, API documentation. Key concepts and the equivalence of 0.75 words per token that OpenAI and Google give is measured in English.[1]OpenAI, API documentation. Key concepts[2]Google AI for Developers, Gemini API documentation (updated September 23, 2026). Understand and count tokens A budget made in words can come up short.
How to check it
- Paste one of your product pages into OpenAI's tokenizer and see how it splits it: which words stay whole and which get divided. The documentation itself links it for trying specific texts.[1]OpenAI, API documentation. Key concepts
- Count before you send. Anthropic has a free token count that accepts the same content as a request, images and PDFs included, and Gemini has
count_tokens.[3]Anthropic, Claude documentation. Token counting[2]Google AI for Developers, Gemini API documentation (updated September 23, 2026). Understand and count tokens - Read the usage field of each response. Gemini returns there the input, output, reasoning and cached tokens of that request.[2]Google AI for Developers, Gemini API documentation (updated September 23, 2026). Understand and count tokens
- Calculate the cost with the price table, per million tokens and with input and output separately.[5]OpenAI, API documentation. Pricing
- If you switch models, count again. The same text can produce quite a few more tokens with another tokenizer.[3]Anthropic, Claude documentation. Token counting
Related concept: context window
The token is the unit; the context window is the capacity, measured in that unit. A model with a window of a million tokens does not read a million words: it reads a million chunks, and in them fit the instructions, what you give it to read and also the response it writes.[1]OpenAI, API documentation. Key concepts That is why, when an assistant splits your page into fragments to retrieve it, those fragments are measured in tokens, as the entry on RAG (retrieval-augmented generation) explains.
To keep going, the entry on the context window explains what happens when a text does not fit, and the reading path orders the rest of the terms.
Synonyms
- English
- token, LLM token
- Español
- token de IA
Sources
- [1] OpenAI, API documentation. Key concepts. Accessed . ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑
- [2] Google AI for Developers, Gemini API documentation (updated September 23, 2026). Understand and count tokens. Accessed . ↑ ↑ ↑ ↑ ↑ ↑
- [3] Anthropic, Claude documentation. Token counting. Accessed . ↑ ↑ ↑ ↑
- [4] OpenAI, API documentation. Vector embeddings. Accessed . ↑
- [5] OpenAI, API documentation. Pricing. Accessed . ↑ ↑ ↑
Change history
- First English version.
This definition is published under CC BY 4.0. Reuse this definition freely, crediting Clione with a link to this page.
Related terms
Context window
What an LLM's context window is, how many tokens fit in Claude or Gemini, why more context is not better and how much of your product page gets read.
LLM
What an LLM is, how it predicts the text it writes, why it makes up facts and what ChatGPT or Gemini do with your store's product page.
Embeddings
What embeddings are, how they measure similarity between texts with vectors, what OpenAI and Gemini use and what they do and do not say about your products.
Vector search
What vector search is, how it finds products by similarity with embeddings, why it fails on a SKU and what Google's hybrid search adds.
RAG
What RAG is in AI, the paper it comes from, how Google and ChatGPT use it to answer with sources, and what part of your product page ends up in the answer.
Topical authority
What topical authority is, what Google documents and what it doesn't, and how to build it in an online store with categories, product pages and links.
Make your catalog easy for bots to read
Clione fills every product page with what bots read best: the JSON-LD graph with its structured data, the meta description and the FAQs. You review and press Push; then we check your live store so it all stays published.