# What Are Tokens in AI and Why They Count in Your Store

> What a token is in artificial intelligence, how many characters it takes, how OpenAI, Gemini and Claude count them and what they cost on your product pages.

URL: https://clione.ai/glossary/token/
Lang: en

If you search for "tokens" you will get cryptocurrencies, NFTs and the access _tokens_ an app asks for to connect to your store. This is about something else: **the unit in which a language model reads, writes and bills**. It matters to a store as soon as it uses a tool that writes product pages with an [LLM (large language model)](/glossary/llm/) and its own API key, because the invoice and the limits arrive in tokens, and also when an assistant reads your page, because what fits of it is measured the same way.

## How it works

OpenAI explains it with an example: tokens are **sequences of characters that show up frequently**. The word _tokenization_ is split in two, "token" and "ization," while a short, common word like _the_ is a single token. In a sentence, the first token of each word usually starts with the space before it.[^openai-concepts] That is why a model sees neither single letters nor whole words: it sees chunks, and long or uncommon words get split into more chunks than short, common ones.

For quick math, both sets of documentation give the same rough rule. OpenAI: a token is about 4 characters or 0.75 words **in English text**.[^openai-concepts] Google: a token is about 4 characters, and 100 tokens equal between 60 and 80 English words.[^gemini-tokens] Both say it about English. For a catalog in another language, or one full of codes and part numbers, the right figure is the one the counter of the model you use gives you, not the rule.

Not everything that is counted is text. [Gemini](/glossary/gemini/) also turns images, video and audio into tokens: **258 tokens per image** if it is no more than 384 pixels on a side; larger ones are divided into 768 × 768 pixel tiles and each costs another 258. Video runs about 100 tokens per second and audio, 32 per second.[^gemini-tokens] The product photos you send to a model to describe also count.

And every model slices in its own way. Anthropic warns that the tokenizer of its most recent models **produces around 30% more tokens for the same text** than that of earlier ones, and asks you to count again with the model you are going to use instead of reusing old counts. It adds that its pre-count is an estimate and that actual usage may differ slightly.[^anthropic-token-counting]

### Why they count: the limit and the bill

The first reason is room. In a model that generates text, **the request and the response together cannot exceed the model's maximum [context length](/glossary/context-window/)**.[^openai-concepts] That maximum is the context window, and it is measured in tokens. [Embeddings](/glossary/embeddings/) models, which turn a text into a vector to search by meaning, have theirs: for OpenAI's, the input takes at most 8,192 tokens.[^openai-embeddings][^openai-concepts]

The second is money. OpenAI publishes its prices **per million tokens**, with separate columns for input tokens, cached input tokens and output tokens.[^openai-pricing] Anthropic presents its token counter precisely as a way to manage costs and usage limits before sending anything.[^anthropic-token-counting] In practice, a long [product description](/glossary/product-description/) costs more twice: when it is read and when it is written.

## In a store

A **fictional example**: a store asks a model, with its own API key, to write the description of a backpack. This is what consumes tokens in that request, and who counts it:

| Piece of the request                                | What it is                          | How it is counted                                                           |
| --------------------------------------------------- | ----------------------------------- | --------------------------------------------------------------------------- |
| Instructions                                        | The brand voice and the style rules | Input tokens, with the model's tokenizer                                    |
| The current [product page](/glossary/product-page/) | Title, attributes and description   | Input tokens; a long text or one full of codes is split into more chunks    |
| Two photos of the product                           | Images for the model to describe    | In Gemini, 258 tokens per small image or per 768 × 768 tile[^gemini-tokens] |
| The response                                        | The new description                 | Output tokens, with their own price[^openai-pricing]                        |

Multiply that by every product in the catalog and every time you regenerate it, and the bill stops being a technical detail. That is why it makes sense for a tool to limit the context it sends in each request. Clione, for example, works with the model and API key the store configures and, when the store turns it on, attaches to each product's _prompt_ the four most relevant excerpts of its documents, about 1,200 tokens.


That a token is a word and that counting words is enough to work out what a catalog will cost. A long word is split into several tokens, like _tokenization_ into two,[^openai-concepts] and the equivalence of 0.75 words per token that OpenAI and Google give is measured in English.[^openai-concepts][^gemini-tokens] **A budget made in words can come up short.**


## How to check it

1. **Paste one of your product pages into OpenAI's tokenizer** and see how it splits it: which words stay whole and which get divided. The documentation itself links it for trying specific texts.[^openai-concepts]
2. **Count before you send.** Anthropic has a free token count that accepts the same content as a request, images and PDFs included, and Gemini has `count_tokens`.[^anthropic-token-counting][^gemini-tokens]
3. **Read the usage field of each response.** Gemini returns there the input, output, reasoning and cached tokens of that request.[^gemini-tokens]
4. **Calculate the cost with the price table**, per million tokens and with input and output separately.[^openai-pricing]
5. **If you switch models, count again.** The same text can produce quite a few more tokens with another tokenizer.[^anthropic-token-counting]

## Related concept: context window

The token is the unit; the [context window](/glossary/context-window/) is the capacity, measured in that unit. A model with a window of a million tokens does not read a million words: it reads a million chunks, and in them fit the instructions, what you give it to read and also the response it writes.[^openai-concepts] That is why, when an assistant splits your page into fragments to retrieve it, those fragments are measured in tokens, as the entry on [RAG (retrieval-augmented generation)](/glossary/rag/) explains.

To keep going, the entry on the [context window](/glossary/context-window/) explains what happens when a text does not fit, and the [reading path](/learn/ai-search/) orders the rest of the terms.