Skip to main content

Load balancing and upstreams

An upstream is one provider endpoint behind a model, such as one Azure OpenAI resource in one region, with its own credentials. When a model lists several upstreams, DIAL Core load-balances requests across them and retries another upstream when one fails. Use upstreams to combine regions and subscriptions, and to send traffic to provisioned capacity before pay-per-token capacity. This reference is for platform engineers who configure model capacity; it assumes you know Models.

For why this matters and where its limits are, see Load balancing and failover.

Upstream fields​

Upstreams are defined in models.<name>.upstreams, an array.

FieldDefaultDescription
endpoint—The provider URL for Chat Completions. It also backs Embeddings and Anthropic Messages; prefer interfaces to set those explicitly.
responsesEndpoint—The provider URL for the Responses API.
interfaces—One provider URL per AI API. See Upstream interfaces.
baseUrl—The provider root that completes each interfaces entry without an endpoint.
id—A stable identifier of the upstream. Required when the model serves the Responses API, and on any upstream that declares interfaces.
key—The API key, token, or credential for the provider.
weight1The upstream's share of traffic within its tier. Higher gets more traffic; 0 or below takes the upstream out of routing.
tier0The priority group. Core routes to the lowest tier value first; higher values are used only when every upstream in a lower tier is unavailable.
extraData—Additional data passed to the adapter, as JSON or a string.
secretExtraData—Like extraData, but treated as a secret: encrypted at rest in the platform bucket and never returned by the Configuration API. Core merges it with extraData into one header.

Core passes the chosen upstream to the adapter in the X-UPSTREAM-ENDPOINT, X-UPSTREAM-KEY, and X-UPSTREAM-EXTRA-DATA headers. A deployment without upstreams gets one implicit upstream whose endpoint and ID are its own serving URL.

Tiers and weights​

Core picks the upstream for each attempt with a tiered, weighted balancer:

  1. It takes the tier with the lowest tier value that has an available upstream.
  2. Within that tier, it spreads requests in proportion to weight.
{
  "models": {
    "gpt-4o": {
      "type": "chat",
      "endpoint": "http://model-provider:5000/compat/openai/openai/deployments/gpt-4o/chat/completions",
      "maxRetryAttempts": 3,
      "upstreams": [
        { "id": "ptu-east", "endpoint": "https://<PTU_EAST>.openai.azure.com/openai/deployments/gpt-4o/chat/completions", "key": "<KEY_1>", "tier": 0, "weight": 3 },
        { "id": "ptu-west", "endpoint": "https://<PTU_WEST>.openai.azure.com/openai/deployments/gpt-4o/chat/completions", "key": "<KEY_2>", "tier": 0, "weight": 1 },
        { "id": "paygo", "endpoint": "https://<PAYGO>.openai.azure.com/openai/deployments/gpt-4o/chat/completions", "key": "<KEY_3>", "tier": 1 }
      ]
    }
  }
}

Requests go to the two tier 0 upstreams in a 3:1 ratio. The tier 1 upstream takes traffic only while neither tier 0 upstream is available.

Retries and failover​

RuleBehavior
Attempt limitCore makes at most min(maxRetryAttempts, number of upstreams) attempts per request. maxRetryAttempts defaults to 1, so set it to fail over at all.
Retriable answers429, 502, 503, 504, and any code in retriableErrorCodes. The failed upstream is skipped for the Retry-After seconds it sent, or a default cool-down, and the next attempt goes to another upstream.
Dropped connectionAn upstream that fails to send response headers after receiving the request counts as a 502.
Connect failureWhen Core cannot open a connection, the client gets 502 Failed to connect to upstream server, and no other upstream is tried.
Other errorsA non-retriable answer is relayed to the client as it is.
No upstream leftThe client gets 503 Service is not available when every upstream is cooling down after a 429, with Retry-After when an upstream sent one. Otherwise, 502 No route.

The response carries X-UPSTREAM-ATTEMPTS, the number of attempts Core spent on the request.

Pin a request to an upstream​

  • X-UPSTREAM-ID: a client can send this header with an upstream's id to pin a request to that upstream, for Chat Completions and the Responses API.
  • Responses API: Core routes follow-up requests (retrieve, cancel, delete) to the upstream that handled the original request, using the upstream id.

Prompt cache pinning​

Some providers cache prompts per endpoint. Core hashes the cacheable prefix of a request and sends later requests with the same prefix to the same upstream, so they can hit the provider's cache. The hash order is fixed per API:

APIHashed prefix, in order
OpenAI Chat Completionstools, messages
Anthropic Messagestools, system, messages (per content block)
OpenAI Responsestools, instructions, input

Candidates for Anthropic come from the client's cache_control blocks or from autoCachingSupported; for Responses, from autoCachingSupported only. The older fieldsHashingOrder model field is ignored. See Prompt caching.

Upstream interfaces​

interfaces on an upstream declares one provider URL per AI API, so one model can front a provider that serves several APIs, such as both the OpenAI and the Anthropic API. The keys are the same as on the model: openaiChatCompletions, openaiEmbeddings, openaiResponses, and anthropicMessages.

Each entry can set endpoint, key, extraData, and secretExtraData. Each field falls back independently to the upstream's own field. An entry without endpoint is completed from the upstream's baseUrl plus the API's own path:

InterfacePath appended to baseUrl
openaiChatCompletions/v1/chat/completions
openaiEmbeddings/v1/embeddings
openaiResponses/v1/responses
anthropicMessages/v1/messages

This differs from a model-level base_url, which is joined with the path the request reached Core on.

  • interfaces is a whitelist: an API it does not declare falls back to responsesEndpoint for Responses and to endpoint for every other API.
  • Every declared interface must resolve to a URL, and the upstream must have an id. A model that breaks either rule is rejected.
  • Declaring the same key in extraData and secretExtraData at one level is rejected as ambiguous.
"upstreams": [
  {
    "id": "fireworks",
    "key": "<FIREWORKS_KEY>",
    "extraData": { "region": "us-east-1" },
    "baseUrl": "https://api.fireworks.ai/inference",
    "interfaces": {
      "openaiChatCompletions": {},
      "openaiResponses": {},
      "anthropicMessages": {
        "endpoint": "https://api.fireworks.ai/inference/something-else/v1/messages",
        "key": "<ANTHROPIC_KEY>"
      }
    }
  }
]

Chat Completions resolves to https://api.fireworks.ai/inference/v1/chat/completions and Responses to https://api.fireworks.ai/inference/v1/responses; Anthropic Messages uses its explicit URL and key. All three send the inherited extraData.

Routes​

Routes use upstreams with the same weight and tier fields, and the route's maxRetryAttempts bounds retries the same way.

Further reading​