Skip to main content

How prompt caching works in DIAL

Prompt caching reuses the computation a model already performed on the unchanging start of a request, so repeated calls cost less and respond faster. This page explains why caching helps, how DIAL Core makes provider caching reliable across multiple model endpoints, and the trade-offs between automatic and manual caching. It is for developers who build applications on DIAL and want to understand caching before enabling it. To turn caching on and confirm a cache hit, follow the tutorial.

Why prompt caching matters​

Prompt caching, also called context caching, reduces the operational cost and latency of interacting with large language models. Many requests share a large, unchanging prefix — system instructions, tool definitions, or earlier turns of a conversation — and only differ in the final user message. Reprocessing that shared prefix on every call wastes computation and tokens.

By storing and reusing the work done on a shared prefix, caching avoids reprocessing the same content in later requests. The benefit grows with the size of the shared prefix and the frequency of repetition, which makes caching valuable for multi-turn conversations, agent loops, and any workflow that resends large instructions or tool schemas.

How caching works in a model​

A model processes a sequence of input tokens and then generates output tokens one at a time. Each input token changes the model's internal state in a deterministic way. Caching exploits this determinism for sequences that share a prefix:

input sequence 1: [tokenA, tokenB, tokenC, tokenD, ...]
input sequence 2: [tokenA, tokenB, tokenC, tokenE, ...]

The model rolls through the first sequence until the shared prefix ends — [tokenA, tokenB, tokenC] — and saves its state at that point. The second sequence then resumes from the saved state, skipping the redundant computation for the shared prefix.

Developers rarely work with raw token streams. They send chat completion requests instead. The idea carries over directly: adjacent parts of a request, such as tool definitions and messages, map onto adjacent blocks of input tokens. Requests that begin with the same sequence of tools and messages can therefore be served more efficiently.

The single-endpoint problem DIAL Core solves​

Leading providers such as OpenAI, Google, and AWS Bedrock support prompt caching, but each limits the cache to a single endpoint. A cache entry created on one model endpoint is invisible to another. To benefit from the cache, requests that share a prefix must reach the same endpoint.

That is a problem when a deployment spreads load across several upstreams for availability and throughput, because the load balancer would otherwise scatter related requests across endpoints and lose the cache.

DIAL Core closes this gap. It uses hashing to route chat completion requests that share a prefix to the same upstream:

request ──▶ DIAL Core ──▶ hash each prefix ──▶ match in Redis? ─yes─▶ same upstream (cache hit)
                                                       │
                                                       └────────no──▶ pick an upstream, store mapping
  • DIAL Core computes a unique hash for each prefix of an incoming chat completion request.
  • It stores the mapping from each hash to an upstream endpoint and an expiration time in Redis.
  • On the next request, it recomputes the hashes and looks them up. On a match, it routes the request to the same upstream, preserving the provider-side cache.

Cache-availability priority policy​

When DIAL Core finds a matching hash and routes to the corresponding upstream, that upstream might return an error. DIAL Core then faces a choice: retry the same upstream to keep the cache hit, or fail over to another upstream and accept a guaranteed cache miss.

By default, DIAL Core prioritizes availability. If the cached upstream is unavailable or returns an error, the request goes to another upstream even though that guarantees a cache miss. You can change this with the X-DIAL-CACHE-POLICY header on the chat completion request:

  • availability-priority (default) — prioritizes service availability over cache hits. If the cached upstream is unavailable or errors, the request is routed to another upstream. The request succeeds at the cost of cached efficiency.
  • cache-priority — prioritizes cache hits over availability. The request retries on the same upstream even after an initial error. This maximizes cache reuse but risks delayed responses if the upstream stays unavailable.

Automatic versus manual caching​

DIAL supports two caching modes. Automatic caching asks DIAL to decide where prefixes end; manual caching gives you explicit control through cache breakpoints.

Automatic cachingManual caching
EffortMinimal — handled by the systemHigher — you mark cache breakpoints
ControlThe system decides breakpointsYou decide breakpoints
Use caseMulti-turn conversations, general efficiencyCustom workflows, tool-heavy prompts
Provider dependencyRequires automatic caching support from the providerWorks wherever the provider supports caching
FlexibilityLower, system-drivenHigher, user-driven

Automatic caching​

Automatic caching needs no input from the caller. DIAL determines when and where to create cache entries, which suits multi-turn conversations where each turn appends to a growing, stable prefix. It depends on the provider offering automatic caching for the deployed model. Enable it by setting features.autoCachingSupported on the deployment.

Manual caching​

Manual caching suits workflows that need precise control over what is cached — for example, a prompt that mixes tool definitions, long system instructions, and a short user query. Enable it by setting features.cacheSupported on the deployment. You mark where a cacheable prefix ends with a cache breakpoint, and DIAL Core computes hashes only for prefixes that end at a breakpoint.

You mark tool definitions and chat messages with the custom_fields.cache_breakpoint field. A breakpoint may include an expire_at timestamp that sets a time-to-live for the cache entry in DIAL Core and on the model side:

{
    "tools": [
        {
            "type": "function",
            "name": "query_db",
            "parameters": {},
            "custom_fields": {
                "cache_breakpoint": {}
            }
        }
    ],
    "messages": [
        {
            "role": "system",
            "content": "(long instructions)",
            "custom_fields": {
                "cache_breakpoint": {
                    "expire_at": "2026-10-02T15:01:23Z"
                }
            }
        },
        {
            "role": "user",
            "content": "(query)"
        }
    ]
}

Trade-offs and limitations​

Caching only helps when requests genuinely share a prefix. A prefix that changes on every call — for example, a system prompt that embeds the current timestamp — never produces a cache hit and adds hashing overhead for no benefit. Place variable content after the cacheable prefix.

Both modes depend on the underlying model supporting prompt caching. Enabling a caching feature flag on a deployment whose model does not support caching has no effect, and the response usage will show no cached tokens. Confirm support with the provider before relying on caching in production.

Further reading​