Skip to main content

Tutorial: enable and verify prompt caching

In this tutorial, you will enable manual prompt caching on a model deployment, send two chat completion requests that share a long prefix, and confirm from the response that the second request hit the cache. By the end, you will know how to mark a cache breakpoint and how to read cached-token counts from the response usage. This tutorial is for developers who can already call the Unified API; for the concepts behind caching, read How prompt caching works first.

Prerequisites​

  • A running DIAL deployment and its base URL, referred to below as <DIAL_URL>.
  • A DIAL API key, referred to below as <YOUR_DIAL_API_KEY>.
  • A chat model deployment in DIAL whose provider supports prompt caching, referred to below as <DEPLOYMENT_NAME>. Confirm caching support with your provider before starting.
  • Edit access to the DIAL Core config.json (dynamic settings).
  • curl and jq installed.

What you'll do​

  1. Turn on manual caching for the deployment with the cacheSupported flag.
  2. Send a first request with a cache_breakpoint after a long shared prefix and store the cache entry.
  3. Send a second request that shares the same prefix and observe a cache hit in the response usage.

Project structure​

By the end of this tutorial, your working directory will look like this:

dial-prompt-caching/
├── request-1.json   # first request — creates the cache entry
└── request-2.json   # second request — shares the prefix, hits the cache

Step 1: Enable manual caching on the deployment​

In the DIAL Core config.json, find your model deployment under models and set features.cacheSupported to true:

{
  "models": {
    "<DEPLOYMENT_NAME>": {
      "type": "chat",
      "features": {
        "cacheSupported": true
      },
      "upstreams": [
        { "endpoint": "<UPSTREAM_ENDPOINT>", "key": "<UPSTREAM_KEY>" }
      ]
    }
  }
}

cacheSupported is a dynamic setting, so DIAL Core applies it on the next configuration reload without a restart.

Verify: check the DIAL Core logs for a configuration reload entry, or confirm the file is picked up by your reload mechanism. The deployment now accepts cache breakpoints.

Step 2: Create dial-prompt-caching/request-1.json​

Create the working directory and the first request:

mkdir dial-prompt-caching && cd dial-prompt-caching

Create dial-prompt-caching/request-1.json. The system message holds a long, unchanging prefix and ends with a cache_breakpoint. The user message is the only part that changes between requests:

{
  "messages": [
    {
      "role": "system",
      "content": "You are a meticulous financial analyst. Follow these rules for every answer: 1) cite the figures you rely on; 2) state assumptions explicitly; 3) flag any data older than one quarter; 4) never give investment advice; 5) answer in plain language. (Extend this block with the full instruction set your application uses — the longer and more stable the prefix, the larger the caching benefit.)",
      "custom_fields": {
        "cache_breakpoint": {}
      }
    },
    {
      "role": "user",
      "content": "Summarize the key risks in a SaaS company with high net revenue retention but negative free cash flow."
    }
  ]
}

Verify: jq . dial-prompt-caching/request-1.json prints the JSON without errors.

Step 3: Send the first request​

Send the request to your deployment. This populates the cache entry for the shared prefix:

curl "<DIAL_URL>/openai/deployments/<DEPLOYMENT_NAME>/chat/completions?api-version=2024-10-21" \
  -H "Api-Key: <YOUR_DIAL_API_KEY>" \
  -H "Content-Type: application/json" \
  -d @request-1.json | jq '.usage'

Verify: the response returns a completion and a usage object. On this first call, the shared prefix is written to the cache, so cached-read tokens are zero or absent. Depending on the provider, you may see cache-write tokens — for example, Anthropic reports cache_creation_input_tokens.

Step 4: Create dial-prompt-caching/request-2.json​

Create dial-prompt-caching/request-2.json with the same system message and breakpoint, changing only the user message:

{
  "messages": [
    {
      "role": "system",
      "content": "You are a meticulous financial analyst. Follow these rules for every answer: 1) cite the figures you rely on; 2) state assumptions explicitly; 3) flag any data older than one quarter; 4) never give investment advice; 5) answer in plain language. (Extend this block with the full instruction set your application uses — the longer and more stable the prefix, the larger the caching benefit.)",
      "custom_fields": {
        "cache_breakpoint": {}
      }
    },
    {
      "role": "user",
      "content": "Now do the same for a hardware company with strong margins but heavy inventory."
    }
  ]
}

Verify: the system message in request-2.json is byte-for-byte identical to the one in request-1.json. Any difference in the prefix produces a different hash and prevents a cache hit.

Step 5: Send the second request and confirm the cache hit​

Send the second request:

curl "<DIAL_URL>/openai/deployments/<DEPLOYMENT_NAME>/chat/completions?api-version=2024-10-21" \
  -H "Api-Key: <YOUR_DIAL_API_KEY>" \
  -H "Content-Type: application/json" \
  -d @request-2.json | jq '.usage'

Verify: the usage object reports cached input tokens for the shared prefix. The exact field depends on the provider:

  • Anthropic models report cache_read_input_tokens greater than zero.
  • OpenAI models report prompt_tokens_details.cached_tokens greater than zero.

A non-zero cached-token count confirms that DIAL Core routed the second request to the same upstream and that the model reused the cached prefix.

What you learned​

  • How to enable manual caching on a deployment with the cacheSupported flag.
  • How to mark a cacheable prefix with custom_fields.cache_breakpoint.
  • How to read cached-token counts from the response usage to confirm a cache hit.

Further reading​