Caching and tokens
Why stable prompt prefixes decide most of an agent bill, and how thinking models use max_tokens.
Why the prefix decides the bill
An agent sends the same long prefix on every turn: the system prompt, the tool definitions and the conversation so far. The repeated part can be billed as cached input, which costs far less than fresh input.
1.47B
Input tokens in 30 days
93%
Of those input tokens cached
12.8M
Output tokens in 30 days
That is one agent-heavy account on a new plan. Its averages were about 142,000 input tokens and 1,200 output tokens per request. Those figures are our own, from its usage data.
Keep the prefix stable
- Keep the system prompt and tool definitions identical between calls. A reordered tool list is a changed prefix.
- Put timestamps, request IDs and per-run values after the stable part, not in the first lines.
- On glm-5.3-flash a fresh input token costs five times a cached one: $0.15 against $0.03 per million. Every change near the top of the prompt moves tokens from the cheap column into the expensive one.
Do not rewrite history
Append each turn instead of rebuilding the transcript. When a long history has to be shortened, do it on a schedule, for example every so many turns, rather than on every call. A rewrite changes the prompt from the point of the change onward, so everything after that point is fresh input again.
Thinking models and max_tokens
Reasoning models return their thinking in a separate reasoning field, and it draws on the same output budget as the answer.
In our live probe, glm-5.3-flash with max_tokens set to 5 returned empty content with finish_reason “length”. The whole budget went to reasoning, and no answer was written.
- Check finish_reason on every reply. “length” means the answer was cut off or never started.
- Give thinking models a budget with room for the reasoning and the answer. A tiny limit saves nothing if the reply is empty.
Output tokens cost the most
On every priced model, output costs more per million tokens than fresh input. On glm-5.3-flash, output is $0.50 against $0.15 for fresh input, more than three times as much. On deepseek-v4-pro it is $3.96 against $1.32.
In our account, output was under 1% of the input and output tokens together. Each output token still costs more than an input token, so the length of replies matters as well as the size of prompts. Shape replies in the prompt, and use max_tokens only as a backstop.
Read more about where tokens go in Cheaper models.
Check the real numbers
Paste an Ollama Cloud key on the home page. Exact monthly dollars on new plans, estimates on old ones, nothing stored.
Check your own usage