What cached tokens do—and what they do not
Cached tokens are input tokens whose eligible computation or context a provider can reuse. Providers may discount reads of that cached input. Generated output is still billed separately, and creating or retaining a cache may carry additional charges.
The table above is generated from the maintained pricing dataset. Compare its input and cached-input columns to see the cache-read discount for each example model. The chart shows ordinary input and output rates, not a cache-hit scenario. These models are billing examples, not a ranking of today’s best models.
Cache-read unit discount = 1 − cached-input rate / ordinary input rate.
That discount applies only to eligible reads. It is not the percentage saved on the entire request. A large output bill, frequent cache misses, short retention, write fees, or storage charges can substantially reduce total savings.
A worked token example
Imagine each request contains 60,000 reusable tokens, 5,000 fresh tokens, and 4,000 output tokens. If the reusable prefix is eligible and hits the cache:
| Billing bucket | No cache | Successful repeat read |
|---|---|---|
| Ordinary input | 65,000 tokens | 5,000 tokens |
| Cached input | 0 tokens | 60,000 tokens |
| Output | 4,000 tokens | 4,000 tokens |
| Pricing data: How we verify → | ||
For the uncached case, multiply input rate by 0.065 and output rate by 0.004. For the repeat read, multiply ordinary input by 0.005, cached input by 0.060, and output by 0.004. Use the generated rate table above; add any cache creation and retention costs separately.
The output quantity does not change. The first request may need to populate the cache, and a repeat request may miss it. Calculate total savings as one minus the cache-aware bill divided by the uncached bill over the same workload, including misses and additional fees.
For broader budgeting, read How to Calculate AI API Costs. Use the token calculator for model comparisons, but do not treat the compact calculator’s fixed uncached split as a cache-aware quote.
Build a cache-friendly prompt and measure the result
Put stable material first: system instructions, fixed policy, tool definitions, response schemas, and reusable examples. Place changing retrieval passages, timestamps, request IDs, and the user question later. Reordering or editing the prefix can prevent reuse.
[stable instructions and policy]
[stable tool schemas and response format]
[reusable examples]
[changing retrieved passages]
[current user request]Do not assume semantic similarity is enough. Use the provider’s documented matching rules, minimum eligible size, retention, and cache controls. Confirm details on the OpenAI pricing page, Anthropic pricing page, and Google AI pricing page, following their official documentation links.
Where it is useful
- Coding assistants: repeated repository instructions and tool schemas can create a reusable prefix; changing file contents still need careful placement.
- Support bots: stable policy and response rules can be reusable while customer details remain request-specific.
- RAG applications: fixed citation instructions can benefit even when the retrieved documents change. Caching does not replace retrieval or ensure factual accuracy.
- Repeated agent calls: shared instructions may qualify, but growing history, tool results, and output remain separate costs.
Measure before assuming a saving
Record total input, cached reads, cache writes where exposed, output, and separately billed services for both initial and repeat requests. Some APIs include cached reads inside total input: subtract them before charging the uncached rate to avoid double-counting.
Track the hit share by tokens, not just by requests. One large miss can outweigh many small hits. Compare representative workloads across the cache’s actual retention period, including deployments that change instructions and quiet periods that allow expiry.
Treat cache isolation and sharing as provider-specific. Do not expose another customer’s data or weaken authorization to improve hit rates. Caching is a billing optimization, not an application memory or access-control system.
If output dominates, focus on shorter responses or model routing as well. If most input changes every request, a smaller model or tighter retrieval may offer more savings than caching. Verify batch compatibility instead of assuming discounts stack.
Alternative model route: For tasks that also work with open models, compare Novita’s hosted catalog against your existing route. Verify the selected endpoint’s own caching support and rates; another provider’s cache behavior does not carry over automatically.
Affiliate disclosure: we may earn a commission through this link, at no extra cost to you.
FAQ
Are cached tokens automatic?
Sometimes. Providers differ on automatic prefix reuse, explicit cache creation, eligibility thresholds, and retention controls.
Do cached tokens reduce output cost?
No. Cache-read discounts apply to eligible input; generated output uses the applicable output rate.
Is the advertised cache discount my total saving?
No. Total savings depend on eligible repeated input, actual hits, output share, cache creation, and storage charges.
Can caching and batch discounts be combined?
It depends on the model and endpoint. Verify the provider’s supported combination and billing rules before forecasting both discounts.