Skip to main content

Scale DIAL for load

Configure DIAL to absorb load spikes by scaling the stateless services automatically and by spreading model traffic across endpoints. This guide covers Horizontal Pod Autoscaler settings in the Helm chart, cluster autoscaling, and why model capacity — not the DIAL services — is usually the real bottleneck. It assumes a working Helm deployment with kubectl and Helm access.

Start from the redundancy in Run DIAL with high availability; scaling adds elasticity on top of fixed replicas.

Prerequisites​

  • DIAL deployed with the dial umbrella chart
  • A metrics server in the cluster (required for CPU- or memory-based autoscaling)
  • A cluster autoscaler or node auto-provisioning, if you want nodes to follow pod demand

Step 1: Right-size the services first​

Size service replicas from representative concurrency, stream duration, request sizes, and dependency latency. Set resource requests and limits so the scheduler and autoscaler have a baseline to reason about:

core:
  resources:
    requests:
      cpu: "1"
      memory: 1Gi
    limits:
      cpu: "4"
      memory: 4Gi

DIAL Core runs on the JVM, so give it headroom above the request for garbage-collection pauses and concurrent request bursts — sizing the limit too close to the request risks CPU throttling or an Out Of Memory kill under load. Treat the values above as a safer starting point, not a target. Tune from observed usage, since the chart ships no fixed sizing table.

Step 2: Enable the Horizontal Pod Autoscaler​

The HPA is disabled by default. Enable it per service and set a target utilization. The autoscaler keeps replicas between minReplicas and maxReplicas:

core:
  autoscaling:
    hpa:
      enabled: true
      minReplicas: 2
      maxReplicas: 6
      targetCPU: 80          # scale out above 80% of the CPU limit
      targetMemory: ""       # optionally scale on memory instead or as well

For finer control, set behavior (scale-up and scale-down stabilization windows) or customRules for custom metrics. The same autoscaling.hpa.* keys apply to DIAL Chat and the adapters.

Step 3: Let the cluster add nodes​

Pods only scale out while the cluster has room. Enable a cluster autoscaler or node auto-provisioning so new nodes are added when pending pods cannot be scheduled, and removed when idle. This is a cluster-level feature; configure it per your platform's deployment guide.

Step 4: Scale model capacity, not only the services​

Model-provider quotas can limit throughput independently of DIAL's CPU. DIAL Core's load balancer distributes traffic across configured upstreams and supports retries. Measure the resulting throughput and HTTP 429 rate for your workload rather than treating extra Core replicas as extra model capacity.

To use it, give each model several upstreams in config.json and, where the provider supports it, raise the per-deployment quota:

{
  "models": {
    "gpt-4": {
      "type": "chat",
      "upstreams": [
        { "endpoint": "https://<DEPLOYMENT_A>/chat/completions" },
        { "endpoint": "https://<DEPLOYMENT_B>/chat/completions" }
      ]
    }
  }
}

In order of preference: request a provider quota increase, add endpoints in more regions, then add more accounts.

Step 5: Apply and watch it scale​

helm upgrade dial dial/dial -f values.yaml --version <CHART_VERSION> --namespace dial
kubectl get hpa -n dial

Generate load and confirm the HPA raises replica counts and that 429 rates stay flat.

Result​

The DIAL services scale out and back automatically with demand, the cluster supplies nodes to match, and model traffic spreads across endpoints with retry — within the configured replica bounds and available upstream capacity. Verify latency and error targets under representative load.

Further reading​