Scalability and performance
DIAL scales by adding copies of its stateless services and by adding model capacity behind them, while shared stores hold the state. This page explains which components keep state, how each one scales, and what load tests showed. It is for architects planning capacity who have read State and data lifecycle.
Stateless services, shared state
DIAL Core, DIAL Chat, and the adapters keep nothing on a replica, so any replica can serve any request. Core keeps its state in two shared stores, a Redis-compatible store and blob storage, and does not require a relational database.
| Component | What it keeps | How it scales |
|---|---|---|
| DIAL Core | Nothing on the replica | Add replicas, and autoscale them on CPU |
| DIAL Chat and the adapters | Nothing on the replica | Add replicas |
| Redis-compatible store, such as Valkey | Per-request API keys, and cached resources, including changes not yet written to blob storage | One store shared by all Core replicas; run a managed, replicated service |
| Blob storage | Conversations, prompts, files, applications, and other resources | Scaled by the storage service |
| Optional component databases | Component data, such as DIAL Admin configuration, analytics, and DIAL Scheduler schedules | Per component; see DIAL Stack |
The Redis-compatible store is not a disposable cache. By default, Core caches resources of up to 1 MiB there and writes a change back to blob storage two minutes after its last modification. Until then, blob storage holds the previous version, so losing the store can lose those changes.
Note
Core replicas share load-balancer endpoint state asynchronously, so routing decisions can briefly differ between replicas.
Scale DIAL Core with replicas
Because the replicas share state only through the shared stores, scaling Core means adding replicas. As a baseline, the deployment guides run two copies of each service on different nodes. The services are not resource-intensive and handle thousands of users without further scaling.
To absorb load spikes, autoscale Core on CPU and let the cluster add nodes when pods cannot be scheduled. The Helm chart ships with the Horizontal Pod Autoscaler turned off; Scale DIAL for load shows the settings. Core runs on the JVM, so leave CPU and memory headroom above its resource requests.
Every Core replica uses the same Redis-compatible store. Plan the store's capacity and failover together with the Core fleet, and choose a multi-zone tier of the managed service.
Code Apps scale per app. App Controller runs each Code App as a Knative service, and Knative scales its pods. The bundled service template starts an app at zero pods and allows up to three; a deployment can override the initial, minimum, and maximum scale.
Model capacity is usually the limit
Model-provider quotas and rate limits usually cap throughput before DIAL's own services do. Extra Core replicas do not add model capacity.
Give a model several upstreams, and Core's load balancer spreads requests across them and retries a failed request on another one. See Load balancing and failover. When one endpoint is not enough, scale in this order: more model instances in the same region, a regional quota increase, instances in other regions, then more subscriptions or accounts.
Load test results
Load tests are currently in progress; we will add information about them later.
Measure your own workload
A user count does not describe load. Short classification calls, long streaming answers, and agents that make parallel tool calls use connections, memory, upstream quota, and time differently. Measure concurrent requests and streams, request and response sizes, agent fan-out, and upstream quotas, then compare:
| Measure | What it reveals |
|---|---|
| Gateway overhead and time to first token | Added platform latency versus provider delay |
| Latency percentiles and error rate under load | Whether the complete path meets your service goals |
| CPU, memory, active connections, and store latency | Which component limits capacity |
| Model and tool usage and cost | The effect of retries, longer contexts, and agent loops |
Measure against a controlled upstream first, to separate DIAL Core's overhead from model generation. Then repeat with the real application and processing chain. Record component versions, topology, and test duration with the results.
Further reading
- Scale DIAL for load — Helm autoscaling settings and model upstreams
- Run DIAL with high availability — replicas, failure domains, and a managed cache
- Load balancing and failover — spread a model's traffic across endpoints with retries
- State and data lifecycle — where DIAL keeps state and how to plan recovery
- DIAL Stack — the services and stores a deployment runs