Prompt Caching For Lower LLM Costs
Reuse stable prompt prefixes through provider native caches, inspect cached token usage, and preserve the real-time inference content-stateless boundary.
Prompt caching and cache aware routing overview
Reuse long prompt prefixes. Pay the provider cache rate.
TrustedRouter preserves provider native caching controls, reports cache hit tokens, and settles cached input at the selected route's cache price when the provider publishes one.
TrustedRouter does not keep a second prompt cache. Prompt and output content still terminate inside the attested gateway and are not written to the control plane, billing database, analytics store, logs, or Sentry.
from openai import OpenAI
client = OpenAI(
api_key="sk-tr-v1-...",
base_url="https://api-azure.trustedrouter.com/v1"
)
response = client.chat.completions.create(
model="deepseek/deepseek-v4-pro",
messages=[
{"role": "system", "content": LONG_STABLE_CONTEXT},
{"role": "user", "content": "Summarize section 4."}
],
extra_body={
"provider": {
"only": ["deepseek"],
"allow_fallbacks": False
}
}
)
details = response.usage.prompt_tokens_details
print(details.cached_tokens if details else 0)How it works
Put reusable content first.
Keep tool definitions, system instructions, examples, documents, and conversation history byte stable. Put the changing question at the end.
The selected provider reuses it.
Automatic caching requires no TrustedRouter setting when the selected provider and model support prefix caching. Minimum prompt size, lifetime, and hit rules are provider specific.
Cache usage reaches the ledger.
The attested gateway normalizes provider usage fields. Cached reads, cache writes, uncached input, and output are settled with integer microdollar arithmetic.
Set an explicit cache breakpoint.
On POST /v1/messages, TrustedRouter preserves Anthropic content blocks and their cache_control values through the attested gateway. Put the breakpoint on the last block of the stable prefix.
The first request can include a cache creation charge. Repeated reads are normally cheaper. Check the returned usage rather than assuming every request hit.
from anthropic import Anthropic
client = Anthropic(
api_key="sk-tr-v1-...",
base_url="https://api-azure.trustedrouter.com"
)
response = client.messages.create(
model="anthropic/claude-sonnet-4.6",
max_tokens=600,
system=[{
"type": "text",
"text": LONG_STABLE_CONTEXT,
"cache_control": {"type": "ephemeral"}
}],
messages=[{
"role": "user",
"content": "Summarize section 4."
}]
)
print(response.usage.cache_creation_input_tokens)
print(response.usage.cache_read_input_tokens)Read the cache result
cached_tokens
Read usage.prompt_tokens_details.cached_tokens. The cached count is included within total prompt tokens for OpenAI compatible responses.
cached_tokens
Read usage.input_tokens_details.cached_tokens. TrustedRouter returns the normalized provider reported count.
Creation and read counts
Read cache_creation_input_tokens and cache_read_input_tokens from usage.
Keep a session near its provider cache.
Provider caches are generally scoped to a provider, account, model, and prompt prefix. A fallback to another provider can be a cache miss even when it serves the same model.
For a repeated, cache sensitive workload, use provider.only and optionally disable fallbacks. For higher availability, keep fallbacks enabled and accept that the first request on a new route may rebuild the cache.
For best-effort affinity with fallback, send session_id in the request body or X-Session-ID in the headers. The body takes precedence; prompt_cache_key is the next choice. A successful explicit session prefers its last successful endpoint for ten minutes of inactivity. Explicit provider.order overrides affinity.
Without an explicit session, text conversations can gain affinity after an actual discounted cache hit. The enclave derives an opaque, key-scoped digest from the opening messages using an enclave-only secret. Prompts and outputs stay out of the routing cache. The cache is bounded, process-local, and scoped by API key, model, and gateway region; enclave or control-plane restarts, another instance, or locally admitted lease requests may miss it. Privacy filters always apply, and fallback remains available.
{
"model": "trustedrouter/auto",
"session_id": "research-session-42",
"stream": true,
"messages": [
{"role": "system", "content": "Your stable context"},
{"role": "user", "content": "The next question"}
]
}Use an opaque identifier. Explicit provider.order takes precedence over session affinity.
Provider behavior
Automatic caches
TrustedRouter recognizes standard OpenAI cached token details and provider specific usage from DeepSeek, Kimi, and Gemini adapters. Automatic cache creation and eviction remain controlled by the selected provider.
Current limits
- Cache hits are not guaranteed
- Provider and model minimum token thresholds apply
prompt_cache_retentionreturns501 not_supported_in_alpha- TrustedRouter does not expose provider cache object create, list, or delete APIs
No router side prompt copy.
Provider caching is upstream model processing, so the selected provider's policy still applies. A cache feature does not upgrade an Open route to ZDR or confidential compute.
Use provider.min_privacy = "zdr", trustedrouter/zdr, or trustedrouter/e2e when the workload requires a hard privacy floor. Routing applies that floor before any prompt is sent.