Prompt caching

How a cache hit works

A language model reads every request from the first token. An AI agent partway through a task sends its instructions, its tool definitions and the whole conversation so far on every single call, so on a long task that repeated reading is most of what you pay for. Prompt caching lets the provider skip work it has just done: if the opening of your request matches the opening of a recent one, token for token, that stretch is served from what the provider kept and billed at a cached rate.

Three details decide whether that happens:

• It is a prefix match. The cache covers everything from the first token up to the first difference and nothing after it. Change one character near the top and every token below that point is read, and billed, in full again.
• It expires. Entries survive minutes of inactivity rather than days, and the window differs by provider and model, so a request after a long pause pays full price even when nothing about it changed.
• It has a floor. Below a minimum prompt length nothing is cached at all, and that minimum is set per model. Read it off the provider's current documentation rather than a number copied from an old blog post.

As of September 2026, Anthropic, OpenAI and Google each bill a cache read on their current models at about a tenth of the ordinary input price. Google and OpenAI apply caching automatically; Anthropic caches up to breakpoints you mark in the request and charges a premium to write an entry. A ten-to-one gap is large enough that the cached share of your input, rather than the size of your prompt, often decides what an agent run costs.

Why trimming the context made our agents more expensive

We run long-lived agents that build and edit websites for customers, and one site-building session showed us what unbounded context costs. In just under four hours it made 622 model calls while the conversation it re-read on every call grew to around 310,000 tokens. The text the model actually wrote came to roughly 6% of the bill. Everything else was reading the same history again.

The obvious fix was pruning: before each call, clear out old tool results the agent had already acted on. We measured it on our own test workspace, 366 real model calls in all, and in the stretch where it actually fired, each call cost about 28% more while only around 23,000 tokens came off each prompt. The pruner worked forward from the oldest result, so every call it rewrote a slightly later point inside the prefix the provider had just cached. Uncached input per call went from 2–8 thousand tokens to 5–25 thousand, and at a ten-to-one price ratio that swamped the saving. The arithmetic is worth keeping: a change inside a cached prefix has to remove roughly nine times as many tokens as it invalidates just to break even.

What worked instead was a ceiling rather than a trim. We capped the context an agent may accumulate at 200,000 tokens, so its history is compacted in one pass as it nears that limit instead of being edited a little on every call, and we switched per-call pruning off. One rewrite followed by a long run of calls that only append is the shape caching rewards. The lesson we took is to read cached and uncached input on each call before believing any change saved money, because the prompt size alone told us the opposite of what the bill said.

How to keep your cache hits

Most of the saving comes from ordering, not configuration. If you are building on a model API, four habits cover nearly all of it:

• Stable material first, changing material last. Instructions, tool definitions and reference documents go at the top and stay byte-for-byte identical between calls. A timestamp, a user's name or a request ID near the top of a system prompt quietly defeats the cache on every request.
• Append, do not edit. Add new turns to the end of the conversation. Rewriting, reordering or summarising earlier turns in place invalidates everything after the edit — the mistake our pruning made one step at a time.
• Rewrite rarely, and all at once. If the history has to shrink, do it in a single compaction at a ceiling you chose, and accept one expensive call in exchange for a long cheap run after it.
• Measure the split, not the size. Providers report cached and uncached input separately in each response's usage data. A change that shortens your prompt and raises the uncached count has made things more expensive, whatever the prompt length says.

The numbers above come from running AgentCeres — the AI Growth Officer at agentceres.com — where a roster of specialist agents drafts pages, posts and outreach, and outbound work waits for a person's approval by default. If you are pricing a product built on calls like these, how do I price an AI product covers why a measured per-call cost belongs underneath the price.