Prompt caching
This section explains how prompt caching reduces cost and latency in DIAL, and how to turn it on and confirm that it works. It is for developers who already build with DIAL and want to reduce cost and latency in production. You should be comfortable calling the Unified API and configuring model deployments in DIAL Core.
In this section
- How prompt caching works — why prompt caching saves cost and latency, how DIAL Core routes shared prefixes to the same upstream, and the difference between automatic and manual caching.
- Tutorial: enable and verify prompt caching — a hands-on walkthrough that turns on caching for a model deployment, sends two requests with a shared prefix, and confirms the cache hit in the response usage.