Skip to main content

Configuration

DIAL Model Provider is configured with environment variables. Every adapter it runs reads the variables it reads when it runs standalone, from one process environment that all adapters share. The process itself adds a few settings. This reference is for platform engineers who deploy DIAL Model Provider; it assumes you know the served paths.

Everything below is checked against DIAL Model Provider 0.1.0-rc.0: its Dockerfile, scripts/docker_entrypoint.sh, scripts/__cacert_entrypoint.sh, and the adapter sources.

Adapter settings​

Each adapter's variables, supported models, and per-deployment settings are on its own pages:

AdapterVariablesModels
OpenAIDIAL OpenAI Adapter configurationSupported upstreams
BedrockDIAL Bedrock Adapter configurationSupported models
Vertex AIDIAL Vertex AI Adapter configurationSupported models
DIAL-to-DIALDIAL-to-DIAL adapter—

Set the variables of every adapter you use on the one DIAL Model Provider container.

Process settings​

VariableDefaultDescription
TIMEOUT_KEEP_ALIVE70How long an idle HTTP keep-alive connection stays open, in seconds. Passed to Uvicorn as --timeout-keep-alive. See Keep-alive timeout.
LOG_LEVELINFOThe level of the aidial_model_provider, aidial_sdk, aidial_adapter_anthropic, and uvicorn loggers, so one level applies to every adapter in the process. AIDIAL_LOG_LEVEL, which the standalone Bedrock and Vertex AI adapters read, is not read.
USE_SYSTEM_CA_CERTS—Any non-empty value trusts the PEM *.crt files mounted at /certificates, in addition to the system CA bundle. See Private certificate authorities.

Keep-alive timeout​

DIAL Core keeps idle connections in its pool for client.keepAliveTimeout seconds, 60 by default. DIAL Model Provider must keep them open longer, so that Core is always the side that closes first. Keep TIMEOUT_KEEP_ALIVE at Core's client.keepAliveTimeout plus 10 seconds:

DIAL Core client.keepAliveTimeoutTIMEOUT_KEEP_ALIVE
60 (default)70 (default)

The default covers an unmodified DIAL Core, so change it only when you change client.keepAliveTimeout. Uvicorn's own default of 5 seconds is not safe. DIAL Model Provider would close idle connections before Core expires them, and a request Core writes into a closed connection fails with Connection was closed.

Private certificate authorities​

To trust a private CA for outbound TLS, set USE_SYSTEM_CA_CERTS and mount a directory of PEM *.crt files at /certificates:

docker run --rm \
  -e USE_SYSTEM_CA_CERTS=1 \
  -v "$PWD/certs:/certificates:ro" \
  epam/ai-dial-model-provider:0.1.0-rc.0

At startup, the container combines the system CA bundle with those certificates in a file under /tmp and points SSL_CERT_FILE at it, so /tmp must be writable.

Shared variables​

The adapters share one environment, so a variable that several adapters read takes one value for all of them. When you merge the environments of standalone adapters, check these variables. Where the defaults differ, the value you set applies to every adapter that reads it.

VariableRead byDefaults
DIAL_URLOpenAI, Bedrock, Vertex AI, DIAL-to-DIAL—
HTTP_MAX_CONNECTIONSOpenAI, Vertex AI, DIAL-to-DIALOpenAI 1000, Vertex AI 250, DIAL-to-DIAL 1000
HTTP_MAX_KEEPALIVE_CONNECTIONSOpenAI, Vertex AI, DIAL-to-DIALOpenAI 100, Vertex AI 75, DIAL-to-DIAL 100
HTTP_POOL_TIMEOUTOpenAI, Vertex AI, DIAL-to-DIAL10
HTTP_TRANSPORTOpenAI, Vertex AIaiohttp
THREAD_POOL_SIZEOpenAI, BedrockOpenAI min(32, CPUs + 4), Bedrock 512
CLAUDE_DEFAULT_MAX_TOKENSOpenAI, Bedrock, Vertex AI1536
ANTHROPIC_MAX_RETRY_ATTEMPTSBedrock, Vertex AI0
ACCESS_TOKEN_EXPIRATION_WINDOWOpenAI, Vertex AI10
AWS_DEFAULT_REGIONBedrock, Vertex AI—

The table lists the variables found in the adapter sources of 0.1.0-rc.0; it is not exhaustive.

Region variables​

The standalone Bedrock adapter also accepts the deprecated DEFAULT_REGION for the AWS region. In DIAL Model Provider, that name belongs to the Vertex AI adapter, as the deprecated form of GCP_DEFAULT_REGION. So the Bedrock adapter reads only AWS_DEFAULT_REGION. When it needs a default region and the variable is not set, it fails with AWS_DEFAULT_REGION env variable is not set.

Example​

DIAL Model Provider in Docker Compose, serving Bedrock and Vertex AI models with credentials from the environment:

model-provider:
  image: epam/ai-dial-model-provider:0.1.0-rc.0
  environment:
    DIAL_URL: http://core:8080
    LOG_LEVEL: INFO
    AWS_DEFAULT_REGION: us-east-1
    AWS_ACCESS_KEY_ID: <AWS_ACCESS_KEY_ID>
    AWS_SECRET_ACCESS_KEY: <AWS_SECRET_ACCESS_KEY>
    GCP_PROJECT_ID: <GCP_PROJECT_ID>
    GCP_DEFAULT_REGION: us-central1
    GOOGLE_APPLICATION_CREDENTIALS: /etc/gcp/key.json
  volumes:
    - ./gcp-key.json:/etc/gcp/key.json:ro

Further reading​