Skip to main content

Run DIAL with high availability

Configure DIAL to reduce disruption from a node, pod, or availability-zone failure. This guide covers the Helm chart settings that make the stateless services redundant, the managed cache choice, and model-endpoint failover. It assumes you deploy DIAL with the umbrella Helm chart and have kubectl and Helm access.

The chart ships single-replica defaults suited to evaluation, not production. High availability is opt-in: you raise replica counts, spread pods across failure domains, and point DIAL Core at a managed, replicated Redis-compatible cache.

Prerequisites​

  • DIAL deployed with the dial umbrella chart on a multi-node, multi-zone cluster
  • A values file you can edit and apply with helm upgrade
  • A managed Redis-compatible service (Amazon ElastiCache, Azure Cache for Redis, or Google Memorystore) for production

Step 1: Run multiple replicas of each service​

DIAL Core, DIAL Chat, and the adapters are stateless, so several copies share load and cover for each other. The chart default is replicaCount: 1. Set at least two for every service:

core:
  replicaCount: 2
chat:
  replicaCount: 2
openai:
  replicaCount: 2

Rolling updates are the default deployment strategy (updateStrategy.type: RollingUpdate), but active-request behavior still needs validation for your deployment.

Step 2: Spread pods across nodes and zones​

Two replicas on the same node or zone do not survive that node or zone failing. Spread them with topology spread constraints or affinity rules:

core:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app.kubernetes.io/name: dial-core

The same keys (topologySpreadConstraints, affinity, nodeSelector, tolerations) exist on every service in the chart.

Step 3: Keep a minimum available during disruptions​

A Pod Disruption Budget stops voluntary disruptions — node drains, autoscaler scale-downs — from taking every replica at once. It is disabled by default (pdb.create: false). Enable it:

core:
  pdb:
    create: true
    minAvailable: 1

Step 4: Use a managed, replicated cache​

The chart bundles a Valkey subchart (valkey.enabled: true) for convenience, but production deployments disable it and use a managed, highly available Redis-compatible cache instead. Set valkey.enabled: false and point DIAL Core at the managed endpoint:

core:
  valkey:
    enabled: false
  env:
    aidial.redis.provider.name: aws-elasti-cache   # or gcp-memory-store, azure-redis-cache
    aidial.redis.clusterServersConfig.nodeAddresses: '["redis://<MANAGED_ENDPOINT>:6379"]'

Choose the managed service's multi-AZ or zone-redundant tier. The store can hold pending resource writes and coordination state. Account for its persistence and failover behavior; see State and data lifecycle.

Step 5: Make model access highly available​

A single model endpoint is a single point of failure and is often rate-limited. In config.json, give each model multiple upstreams. DIAL Core's load balancer spreads requests across them and fails over to the next endpoint when one is unavailable:

{
  "models": {
    "gpt-4": {
      "type": "chat",
      "upstreams": [
        { "endpoint": "https://<REGION_A_ENDPOINT>/chat/completions" },
        { "endpoint": "https://<REGION_B_ENDPOINT>/chat/completions" }
      ]
    }
  }
}

That example fits an OpenAI-compatible upstream, where each region is a distinct URL. Other adapters expose multi-region failover differently, and using the adapter's own mechanism is usually preferable to hand-rolling it with separate upstreams:

  • Amazon Bedrock — cross-region inference is a property of the model ID itself, not the upstreams list: prefix the deployment ID with a region-set code (us., eu., apac., jp., or global.), for example global.<BEDROCK_MODEL_ID>. The adapter then routes across that region set automatically. See AWS deployment scalability.
  • Vertex AI — set upstreams[].extraData.region (and optionally .project) on each upstream instead of a different URL; every entry points at the same adapter endpoint, and the adapter itself calls the given region/project. See GCP deployment scalability.
  • Azure OpenAI — use the endpoint-per-upstreams-entry pattern shown above, one per regional resource, as in the example.

For adapters without a provider-level equivalent, the endpoint-per-upstream pattern above is the general mechanism.

Step 6: Apply and verify​

helm upgrade dial dial/dial -f values.yaml --version <CHART_VERSION> --namespace dial
kubectl get pods -n dial -o wide

Confirm replicas land on different nodes and zones in the -o wide output.

Result​

Each DIAL service runs redundant pods spread across failure domains, a Pod Disruption Budget preserves capacity during maintenance, the cache is a managed replicated service, and models fail over across endpoints. Validate the resulting topology with node/zone failure tests, including active streams and shared dependencies.

Further reading​