Skip to main content

DIAL Evaluations

The evaluation services run the test suites that the Evaluation section of DIAL Admin defines, and score the answers. Evaluation Framework Backend stores datasets, test suites, and runs, and sends each test request through DIAL Core. Evaluation Metrics calculates the scores. This reference is for platform engineers who deploy the two services next to DIAL Admin. For the Evaluation screens, see Evaluation overview.

info
ComponentRepositoryReleasesImage tags
Framework BackendGitHubReleasesDocker Hub
MetricsGitHubReleasesDocker Hub

The two services​

ServiceRepositoryImageWhat it does
Evaluation Framework Backendai-dial-admin-evaluation-framework-backendepam/ai-dial-admin-evaluation-framework-backendStores datasets, test suites, and runs in PostgreSQL. Runs test suites against models, applications, and MCP tools through DIAL Core, sends the answers to the metric providers, and keeps the scores. A Spring Boot service.
Evaluation Metricsai-dial-admin-evaluation-metricsepam/ai-dial-admin-evaluation-metricsPublishes the metric catalog and calculates metric scores. Rule-based metrics run in the service; LLM metrics call a judge model through DIAL Core. A stateless FastAPI service.

Evaluation is a Preview feature of DIAL Admin. The metrics the service ships with are listed in Available metrics.

How the services fit together​

  1. DIAL Admin to the framework backend. Setting DIAL_EVAL_API_URL on Admin Frontend turns on the Evaluation menu. Admin's server sends every evaluation request to the backend's /api/v1 API with the signed-in user's token.
  2. Token validation. The backend accepts the token only when its issuer, audience, and roles match one of the backend's identity providers. See Security.
  3. Calls to DIAL Core. The backend calls Core with the same user token to list deployments, run Try Out, run test suites, and call MCP tools. A user can evaluate only the deployments their token can reach. The backend's own API key, DIAL_EF_API_KEY, is used for the test case files it keeps in Core's storage.
  4. Metric catalog. A sync job reads GET /metrics from each metric provider and stores the metric declarations that Admin's Add metric step lists. The job is off by default.
  5. Scoring. After the target answers, the backend sends each metric to its provider's POST /evaluate. LLM metrics call the judge model through Core with the metrics service's own DIAL_API_KEY, so judge calls use that key's roles and limits.
  6. Results. Scores and timing are stored in the backend's analytics database. Run cost figures come from the Analytics data-access service, when the backend is connected to one.

A run uses the token of the user who started it. If a long run outlasts that token, it fails. The optional Application mode runs test suites through a DIAL Core application route with a per-request key instead.

Dependencies​

ServiceNeeded byWhy
DIAL CoreBoth servicesHosts the deployments under test and the judge and embeddings models, and stores test case files. Each service needs a Core API key; the advanced sample uses one key for both.
PostgreSQLFramework backendTwo databases, or two schemas: meta for datasets, test suites, and runs, and analytics for results and scores. The backend applies its own migrations at startup.
Identity providerFramework backendIssues the tokens Admin forwards. It is the provider that signs administrators in to DIAL Admin.
DIAL Admin—The interface for datasets, test suites, and runs. It connects only to the framework backend.
Analytics data-access serviceFramework backend, optionalRun cost figures, set with DIAL_ADAS_URL.
Grafana with TempoFramework backend, optionalTrace links in runs, set with GRAFANA_BASE_URL.

Deployment​

ServicePortEndpoints
Framework backend8080The REST API under /api/v1, health checks at /api/v1/health, /api/v1/health/ready, and /api/v1/health/live, and Swagger UI at /swagger-ui.html
Framework backend9464Spring Boot Actuator, with Prometheus metrics at /actuator/prometheus
Evaluation Metrics5000GET /metrics, POST /evaluate, GET /health, and API docs at /docs

Neither the dial/dial nor the dial/dial-admin Helm chart deploys the evaluation services. Run them from their images, as in the Docker Compose example and the advanced setup sample, which pins Framework Backend 0.5.0 and Metrics 0.3.0 with Admin Frontend 0.22.1.

Warning

Evaluation Metrics has no authentication. Any client that reaches it can run LLM metrics with its DIAL Core API key. Keep it on an internal network that only the framework backend can reach.

Warning

The framework backend has no per-user authorization yet. Every user it accepts can read and change all datasets, test suites, and runs.

eval-cli​

The framework backend's repository also contains eval-cli, a command-line tool that runs test suites from a pipeline. It copies the selected test suites on the backend, fetches their test cases, runs them against a target deployment through DIAL Core, and imports the results as a run. No image is published; build it from the repository at a release tag. It calls the backend with an Api-Key header, so the backend needs API-key authentication. See the eval-cli README for its commands and settings.

Evaluation elsewhere in the docs​