DIAL Evaluations configuration

The evaluation services take their settings from environment variables, and depend on settings in DIAL Admin, DIAL Core, and the identity provider. This page maps each setting to its place, lists the steps that connect DIAL Admin to the services, and gives a Docker Compose example. It is for platform engineers who add Evaluation to a DIAL deployment that already runs DIAL Admin with sign-in.

Where each setting lives

Place	What it sets	Reference
Admin Frontend environment variables	`DIAL_EVAL_API_URL`, and the sign-in whose token Admin forwards	Admin Frontend evaluation settings
Framework backend environment variables	The databases, DIAL Core URL and API key, token validation, metric providers, run limits, and telemetry	Framework Backend
Metrics service environment variables	The DIAL Core URL and API key, the judge models, concurrency, and logging	Metrics
DIAL Core dynamic configuration	The API keys of both services, the roles that can use the judge and embeddings models, and the application route for Application mode	Keys, Roles, Application routes
Identity provider	Tokens for DIAL Admin users with an audience and roles the backend accepts	Sign in to DIAL Admin

Connect DIAL Admin

Create API keys in DIAL Core. Give each service a key, or share one key, as the advanced sample does. Give the key a role that is listed in the userRoles of the judge and embeddings models, or leave userRoles unset on those models.
```
{
  "keys": {
    "<DIAL_EVALUATION_API_KEY>": {
      "project": "DIAL-EVALUATION",
      "role": "default"
    }
  }
}
```
Run Evaluation Metrics with DIAL_URL, DIAL_API_KEY, and the judge models: EVAL__METRICS__COMMON__SUPPORTED_MODELS and EVAL__METRICS__COMMON__DEFAULT_MODEL. The built-in list names six specific deployment IDs; replace it with deployments your Core has. See Judge models.
Run the framework backend with both databases, DIAL_CORE_URL, DIAL_EF_API_KEY, METRIC_PROVIDERS_DIAL_BASE_URL set to the metrics service, and METRIC_PROVIDERS_SYNC_ENABLED=true. Without the sync, the backend doesn't load the metrics service's catalog, and Add metric doesn't offer its metrics.
Set token validation on the backend for the identity provider that signs administrators in to DIAL Admin. PROVIDERS_<ID>_ISSUER must equal the iss claim of the tokens, which is the issuer URL the browser uses. PROVIDERS_<ID>_JWKSETURI must be reachable from the backend. PROVIDERS_<ID>_AUDIENCES must contain the tokens' aud, and PROVIDERS_<ID>_ALLOWEDROLES the administrators' roles. See Identity providers.
Point DIAL Admin to the backend. Set DIAL_EVAL_API_URL on Admin Frontend to the backend's base URL, without /api/v1. Keep Admin's sign-in on: without NEXTAUTH_URL, Admin sends no token, and the backend accepts requests only with CONFIG_REST_SECURITY_MODE=none.
Check the result. The Evaluation menu appears in DIAL Admin. In a test suite, Add metric lists the metrics, and Try Out returns the target's answer.

Troubleshooting

Symptom	Cause
No Evaluation menu in DIAL Admin	`DIAL_EVAL_API_URL` isn't set on Admin Frontend.
The Evaluation screens fail to load	The backend rejects the token: its issuer, signature, or audience doesn't match a provider. If the backend logs `Access denied for issuer`, the user has none of the provider's `ALLOWEDROLES`.
Add metric offers no metrics	`METRIC_PROVIDERS_SYNC_ENABLED` is `false`, or `METRIC_PROVIDERS_DIAL_BASE_URL` doesn't reach the metrics service.
A deployment is missing from the targets	The signed-in user's token can't use that deployment in DIAL Core.
LLM metrics fail	The judge model doesn't exist in Core, or the metrics service's key can't use it.
Ragas: Answer Relevancy fails	The embeddings deployment `text-embedding-ada-002` doesn't exist in Core. See Embeddings model.
A long run fails partway through	The user's token expired. Use Application mode.
The backend doesn't start	`DIAL_EF_API_KEY` is unset, or both datasources point to the same database and schema.
Run costs are missing	The backend isn't connected to the Analytics data-access service; see Run costs.

Docker Compose example

This example is adapted from the advanced setup sample. It adds the two services and their database to a Compose project that already runs DIAL Core as core, and DIAL Admin with Keycloak sign-in as in the Admin Frontend example. Replace <JUDGE_MODEL> with a chat model deployed in Core.

services:
  evaluation-postgres:
    image: postgres:17.5
    environment:
      POSTGRES_DB: evaluation_db
      POSTGRES_USER: evaluation
      POSTGRES_PASSWORD: ${EVALUATION_POSTGRES_PASSWORD:?Set a database password}
    volumes:
      - evaluation-postgres-data:/var/lib/postgresql/data
      # Creates the second database, evaluation_analytics_db, on first start
      - ./evaluation/init-db.sql:/docker-entrypoint-initdb.d/init-db.sql:ro

  evaluation-metrics:
    image: epam/ai-dial-admin-evaluation-metrics:0.3.0
    environment:
      DIAL_URL: http://core:8080
      DIAL_API_KEY: ${DIAL_EVALUATION_API_KEY:?Set the Core API key of the evaluation services}
      EVAL__METRICS__COMMON__SUPPORTED_MODELS: '["<JUDGE_MODEL>"]'
      EVAL__METRICS__COMMON__DEFAULT_MODEL: <JUDGE_MODEL>
      EVAL__APP__MAX_CONCURRENT_EVALUATIONS: "2"
      # No OpenTelemetry collector in this example
      OTEL_TRACES_EXPORTER: none
      OTEL_METRICS_EXPORTER: none
      OTEL_LOGS_EXPORTER: none

  evaluation-backend:
    image: epam/ai-dial-admin-evaluation-framework-backend:0.5.0
    ports:
      - "127.0.0.1:8060:8080"
    environment:
      POSTGRES_META_DATASOURCE_URL: jdbc:postgresql://evaluation-postgres:5432/evaluation_db
      POSTGRES_META_DATASOURCE_USERNAME: evaluation
      POSTGRES_META_DATASOURCE_PASSWORD: ${EVALUATION_POSTGRES_PASSWORD}
      POSTGRES_ANALYTICS_DATASOURCE_URL: jdbc:postgresql://evaluation-postgres:5432/evaluation_analytics_db
      POSTGRES_ANALYTICS_DATASOURCE_USERNAME: evaluation
      POSTGRES_ANALYTICS_DATASOURCE_PASSWORD: ${EVALUATION_POSTGRES_PASSWORD}
      DIAL_CORE_URL: http://core:8080
      DIAL_EF_API_KEY: ${DIAL_EVALUATION_API_KEY}
      METRIC_PROVIDERS_DIAL_BASE_URL: http://evaluation-metrics:5000
      METRIC_PROVIDERS_SYNC_ENABLED: "true"
      METRIC_PROVIDERS_SYNC_CRON: "0 */5 * * * *"
      CONFIG_REST_SECURITY_MODE: oidc
      # The issuer in the tokens, as the browser and Admin see it
      PROVIDERS_KEYCLOAK_ISSUER: ${KEYCLOAK_ISSUER_URL:?Set the shared Keycloak issuer URL}
      # The same realm's keys, as this container reaches them
      PROVIDERS_KEYCLOAK_JWKSETURI: ${KEYCLOAK_JWKS_URL:?Set the Keycloak JWKS URL}
      PROVIDERS_KEYCLOAK_AUDIENCES: dial-admin-console
      PROVIDERS_KEYCLOAK_ROLECLAIMS: roles
      PROVIDERS_KEYCLOAK_PRINCIPALCLAIM: sub
      PROVIDERS_KEYCLOAK_ALLOWEDROLES: ConfigAdmin,admin
    depends_on:
      - evaluation-postgres
      - evaluation-metrics

  admin-frontend:
    environment:
      DIAL_EVAL_API_URL: http://evaluation-backend:8080

volumes:
  evaluation-postgres-data:

evaluation/init-db.sql creates the analytics database:

CREATE DATABASE evaluation_analytics_db;

Add the <DIAL_EVALUATION_API_KEY> key from step 1 to Core's configuration, with the same value as DIAL_EVALUATION_API_KEY.
KEYCLOAK_ISSUER_URL is the issuer that Admin Frontend uses in AUTH_KEYCLOAK_HOST. The advanced sample sets it to http://keycloak.localhost:8058/realms/dial, and the JWKS URL to http://keycloak:8058/realms/dial/protocol/openid-connect/certs.
The sample also sets EVAL__METRICS__RAGAS__EMBEDDINGS_MODEL, which Metrics 0.3.0 doesn't apply; see Embeddings model.
Port 8060 publishes the backend's Swagger UI on the host. The metrics service is published nowhere, because it has no authentication.

Where each setting lives​

Connect DIAL Admin​

Troubleshooting​

Docker Compose example​

Where each setting lives

Connect DIAL Admin

Troubleshooting

Docker Compose example