Skip to main content

Evaluation Metrics configuration

Evaluation Metrics takes its settings from environment variables. This reference lists them for release 0.3.0, with the defaults from the release's source. It is for platform engineers who run the service for the framework backend; the metrics it calculates are listed in Available metrics.

Endpoints​

The service listens on port 5000.

EndpointDescription
GET /metricsThe metric catalog: each metric's name, description, configuration and input schemas, and outputs. The framework backend's sync job reads it.
POST /evaluateCalculates one metric for one result.
GET /healthHealth check.
/docs, /redoc, /openapi.jsonAPI documentation.

Warning

The service has no authentication. Any client that reaches it can run LLM metrics with its DIAL Core API key. Keep it on an internal network that only the framework backend can reach.

DIAL Core connection​

VariableDefaultDescription
DIAL_URL—Required. DIAL Core URL for the judge and embeddings calls. The service doesn't start without it.
DIAL_API_KEY—DIAL Core API key for those calls. Without it, the service starts with a warning, and only rule-based metrics work.

Judge and embeddings calls go through Core's Unified API to the deployment ID named in the metric's configuration. The key's role must be allowed to use those deployments; see Roles.

Judge models​

The LLM metrics are in three groups: DeepEval, Ragas, and DIAL RAG Eval. In each group, the model parameter of a metric's Configuration in DIAL Admin offers the group's supported models, with the default model preselected.

VariableDefaultDescription
EVAL__METRICS__COMMON__SUPPORTED_MODELSSee belowJSON array of Core deployment IDs offered as judge models in all groups, such as '["gpt-4o"]'.
EVAL__METRICS__COMMON__DEFAULT_MODELgemini-3.1-flash-lite-previewJudge model preselected in all groups. If it isn't in the group's list, the first listed model is used, with a warning.
EVAL__METRICS__DEEPEVAL__SUPPORTED_MODELSThe common listSupported models of the DeepEval metrics only.
EVAL__METRICS__DEEPEVAL__DEFAULT_MODELThe common defaultDefault model of the DeepEval metrics only.
EVAL__METRICS__RAGAS__SUPPORTED_MODELSThe common listSupported models of the Ragas metrics only.
EVAL__METRICS__RAGAS__DEFAULT_MODELThe common defaultDefault model of the Ragas metrics only.
EVAL__METRICS__AIDIAL_RAG_EVAL__SUPPORTED_MODELSThe common listSupported models of the DIAL RAG Eval metrics only.
EVAL__METRICS__AIDIAL_RAG_EVAL__DEFAULT_MODELThe common defaultDefault model of the DIAL RAG Eval metrics only.

The built-in list is gemini-2.5-flash-lite, gpt-5-nano-2025-08-07, anthropic.claude-haiku-4-5-20251001-v1:0, gemini-3.1-flash-lite-preview, gpt-5-mini-2025-08-07, and gpt-5.4-mini-2026-03-17. These are deployment IDs, so set the list to deployments that exist in your DIAL Core.

Embeddings model​

Ragas: Answer Relevancy also calls an embeddings deployment. It is the metric's embeddings_model configuration parameter, with the default text-embedding-ada-002. Deploy an embeddings model with that ID in Core, or set the parameter when you add the metric to a test suite.

Note

The service's README and the advanced setup sample set EVAL__METRICS__RAGAS__EMBEDDINGS_MODEL. Release 0.3.0 reads that variable but doesn't apply it, so it doesn't change the default.

Concurrency​

VariableDefaultDescription
EVAL__APP__MAX_CONCURRENT_EVALUATIONS10Evaluations calculated at once. Further requests wait.
EVAL__APP__MAX_QUEUE_BACKLOG100Requests that may wait. When the queue is full, the service answers 503.
EVAL__METRICS__AIDIAL_RAG_EVAL__MAX_CONCURRENCY8Parallel LLM calls within one DIAL RAG Eval evaluation.
EVAL__METRICS__DEEPEVAL__VERBOSE_MODEfalsePrint DeepEval's intermediate steps to standard output. The API responses don't change.

The framework backend sends at most METRIC_EVALUATION_DEFAULT_CONCURRENCY_PER_PROVIDER evaluations, 5 by default, to one provider at a time; see Metric evaluation.

Logging and telemetry​

VariableDefaultDescription
LOG_LEVELINFOLog level: CRITICAL, ERROR, WARNING, INFO, or DEBUG.
LOG_LEVEL_OVERRIDE{}Log levels of individual modules, as JSON, such as {"uvicorn.access":"INFO"}.
DIAL_SDK_LOG_FORMATtexttext or json.
DIAL_SDK_TEXT_LOG_FORMAT'%(levelprefix)s | %(asctime)s | %(name)s | %(process)d | %(message)s'Format of text logs.
DIAL_SDK_JSON_LOG_FORMAT'{"level": "%(levelname)s", "time": "%(asctime)s", "logger": "%(name)s", "process": "%(process)d", "message": "%(message)s"}'Format of JSON logs.

OpenTelemetry is set up through the DIAL SDK and takes the standard OTEL_* variables. Without a collector, set OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, and OTEL_LOGS_EXPORTER to none, as the advanced sample does.

The image also sets two variables: DEEPEVAL_TELEMETRY_OPT_OUT=1, which turns DeepEval's telemetry off, and TIKTOKEN_CACHE_DIR=/app/tiktoken_cache, where the tokenizer files are bundled, so nothing is downloaded at runtime.

Further reading​