Skip to main content

Troubleshooting

This page catalogs the failures DIAL operators encounter most often, the HTTP status codes DIAL Core returns, and the resolution for each. It is for DevOps and platform engineers running a deployment who need to map a symptom to a cause quickly. It assumes familiarity with the Operating DIAL sections it links to; it does not re-explain configuration that those pages already document.

How to diagnose​

Three facts about how DIAL is built determine where to look first.

  • DIAL Core and the other services are stateless. A crashed or restarted replica loses nothing durable. Most "lost data" reports are storage or Redis problems, not Core problems.
  • Blob storage holds permanent data — conversations, files, published resources, application state. Storage errors surface as failed reads, failed uploads, or 5xx responses.
  • Redis holds volatile data — caches, sessions, rate-limit counters. Redis being unreachable degrades or blocks requests even when storage is healthy.

First steps​

  1. Read the service logs. Every quick-start and deployment page verifies a healthy start by confirming that the core, chat, and adapter services logged a startup line with no errors. A service that exits immediately or logs a stack trace on boot points at configuration, not load. See Logging for log format and levels.
  2. Identify the HTTP status code returned to the client. The HTTP error code reference below maps each code to its triggers.
  3. Read the error body. DIAL Core returns a structured JSON error (see Error response format); its message field names the specific failure.
tip

When an application or Quick App returns a generic "Tool call failed" with no detail, the raw error is being suppressed. Set display_error_in_stage to true on the application configuration temporarily to surface the underlying error in the chat stage.

Symptom index​

The fastest path from a symptom to the section that resolves it.

SymptomLikely areaSection
A service exits or crashes on startupStartup and deploymentStartup and deployment
Containers fail to bind a portStartup and deploymentStartup and deployment
DIAL Admin will not connect to DIAL CoreStartup and deploymentStartup and deployment
Every request returns 401Authentication and accessAuthentication and access
A user is blocked from a model, app, or file (403)Authentication and accessAuthentication and access
Chat completions time out or hangModels and adaptersModels and adapters
Rising rate of 5xx from an adapter or routeModels and adaptersModels and adapters
Spike in 429 responsesModels and adaptersModels and adapters
File upload or download failsStorage and RedisStorage and Redis
Sessions or rate limits behave erraticallyStorage and RedisStorage and Redis

For the signals to alert on for each of these failure modes, see Alerting.

Startup and deployment​

SymptomLikely causeResolution
A container fails to start with a port-bind errorPorts 8080 (DIAL Core) or 3000 (DIAL Chat) are already in useFree the ports or remap them. See Echo application for the local-setup port requirements.
DIAL Admin will not start or applies an unexpected configuration versionAdmin cannot reach Core to read its config version, or the config.json is incompatible with CoreAdmin falls back to the version in CORE_CONFIG_VERSION when it cannot connect. Verify Core is reachable and the configuration is compatible.
Core exits on boot with a configuration errorA required settings.json section is missing or malformedValidate against the Core configuration reference. Storage, Redis, and at least one identity provider are required.
Adapter fails to start when file storage is enabledDIAL_URL is unset while DIAL_USE_FILE_STORAGE is TrueSet DIAL_URL to the Core URL. See Adapter configuration.

Authentication and access​

SymptomLikely causeResolution
Every request returns 401No identity provider is configured, or the API key is unknownConfigure at least one provider with jwksUrl or userInfoEndpoint. See Identity Providers and JWT authentication.
401 with a valid-looking keyBoth an API key and an Authorization header were sent (Core rejects this as 400), or the Authorization header is malformedSend exactly one credential. See API keys.
403 on a model or applicationThe authenticated user lacks role-based access to that deploymentGrant the role. See Roles and rate limits.
403 on a file attached to a chat completionThe application's per-request key does not cover the attached fileAttach only files associated with the per-request key or directly accessible to the application.
403 when approving a publicationOnly administrators can approve publicationsPerform the approval with an admin account.

Models and adapters​

SymptomLikely causeResolution
Chat completion returns 504The upstream model did not respond in timeCheck upstream health and latency. For streaming, set SSE_HEARTBEAT_INTERVAL on the adapter so it injects ping comments and avoids idle read timeouts.
Chat completion returns 502The upstream is unavailable, or the maximum retry attempts were exceededmaxRetryAttempts defaults to 5 for language models and 1 for applications. Verify the upstream endpoint and adjust retries in Core dynamic settings.
Chat completion returns 503Every retry to the upstreams returned 429, or the application controller or code interpreter is unavailableAdd upstream capacity or raise provider limits; confirm the application controller is running.
Spike in 429 responsesA user exceeded token or request limits per minute, hour, or dayInspect and adjust limits. A Retry-After header indicates the wait. See Roles and rate limits.
Intermittent upstream errors from one providerProvider-side retriable errors are not being retriedSet the provider retry variable: ANTHROPIC_MAX_RETRY_ATTEMPTS, BOTOCORE_MAX_RETRY_ATTEMPTS, or GOOGLE_GENAI_MAX_RETRY_ATTEMPTS (each defaults to 0).

Sustained 5xx or latency from adapters warrants an alert; see Configure metrics and Alerting.

Storage and Redis​

SymptomLikely causeResolution
File upload or download fails; resource reads errorBlob storage is misconfigured or unreachableVerify the provider-specific required fields: S3 needs bucket and region; Azure Blob needs a container; GCS needs bucket and projectId. See Storage.
Upload rejected with 413The content exceeds a size limitmaxUploadedFileSize defaults to 512 MB for multipart/form-data; maxSize defaults to 67 MB for resources. Raise the relevant limit or reduce the payload.
Sessions drop or rate limits behave erraticallyRedis is unreachableRedis is required for caching, sessions, and rate-limit counters. Provide a single-server or cluster configuration. See Redis.
412 on a resource writeA concurrent modification failed the If-Match or If-Unmodified-Since preconditionRe-read the resource and retry the write with the current version.

Losing Redis costs a cache warm-up, not data. Losing blob storage loses permanent data — see Back up and restore DIAL.

HTTP error code reference​

DIAL Core returns the following HTTP status codes. The triggers are authoritative, drawn from the Core error-code catalog.

CodeNameMeaning and common triggers
400Bad RequestClient error: a malformed resource URL, both an API key and an Authorization header supplied, or a folder that cannot be downloaded.
401UnauthorizedInvalid credentials: a bad Authorization header or an unknown API key.
403ForbiddenAuthenticated but lacking permission: no access to a model or application, approving a publication without admin rights, or accessing files attached to a chat completion request.
404Not FoundThe request is authenticated and authorized, but the resource — application, model, file, conversation, or prompt — does not exist.
405Method Not AllowedThe HTTP method is unsupported. Core supports GET, DELETE, POST, PUT, HEAD, and OPTIONS.
409ConflictThe request conflicts with resource state: an application must be stopped before moving, deploying, or deleting it, and started before undeploying it.
412Precondition FailedA conditional request failed its If-Match or If-Unmodified-Since precondition — typically a concurrent resource modification.
413Content Too LargeThe request exceeds maxUploadedFileSize (storage settings, default 512 MB, for multipart/form-data) or maxSize (resource settings, default 67 MB).
415Unsupported Media TypeThe content format is not accepted. Chat completion requests must be application/json.
422Unprocessable EntityThe server failed to receive the request body.
429Too Many RequestsRate limiting: the user exceeded token or request limits over a minute, hour, or day on a chat completion or router request. A Retry-After header may indicate the wait.
500Internal Server ErrorA generic catch-all for an unexpected server condition with no more specific 5xx code.
502Bad GatewayAn upstream is unavailable on a chat completion request or behind a route, or the maximum retry attempts were exceeded.
503Service UnavailableThe server is not ready: all upstream retries returned 429, or the application controller or code interpreter is unavailable.
504Gateway TimeoutThe upstream did not respond in time on a chat completion request.
505HTTP Version Not SupportedThe HTTP version is unsupported. Core supports HTTP/1.1 only.

Error response format​

DIAL Core serializes errors as a JSON object compatible with the OpenAI error shape. Field names are snake_case, and null fields are omitted.

{
"error": {
"message": "human-readable error message",
"display_message": "message safe to show to end users",
"type": "error type",
"param": "offending parameter",
"code": "machine-readable code"
}
}
FieldDescription
messageThe full human-readable error message. The primary field for diagnosis.
display_messageA message safe to surface to end users. Populated when the raw message should not be shown directly.
typeThe error type.
paramThe request parameter that caused the error, when applicable.
codeA machine-readable error code, when applicable.

Further reading​