Evaluation overview

The Evaluation section of DIAL Admin lets you measure how well a model, an application or an MCP tool works, and check that it keeps working after you change it. You describe what to test in a dataset, choose how to score the answers with metrics, run the test suite against a deployment, and read or compare the results — all in the Admin Panel.

What you can do

• Benchmark a model or an application on your own test cases: accuracy, answer relevance, faithfulness to retrieved context, or any custom criteria.
• Evaluate RAG applications: score retrieval and answer quality with ready-made metrics.
• Test MCP tools: call a tool of an MCP toolset with test arguments and check the result.
• Test multi-step behavior: multi-turn dialogues, and chains of up to eleven requests per test case where each request uses the results of the previous ones.
• Catch regressions: re-run the same test suite after a change, compare two runs side by side, and follow score trends across runs. A score threshold marks which test cases and runs pass.
• Analyze and share results: drill down to a single test case, see cost and timing, open traces in Grafana, and export results to CSV.
• Automate: run test suites from a pipeline with eval-cli, which executes them against a target deployment and imports the results back into the Runs list.

How it works

DIAL Admin (Evaluation screens)
        │  Evaluation API
        ▼
Evaluation framework backend ──► DIAL Core ──► evaluated model, application or MCP tool
        │                            ▲
        │ metric requests            │ judge model calls
        ▼                            │
Evaluation metrics service ──────────┘

Results are stored in PostgreSQL (suites, datasets, runs and analytics).

1. You work in DIAL Admin. It sends requests to the evaluation framework backend.
2. For each test case, the evaluation framework backend sends the request from the test suite to the target deployment through DIAL Core, and extracts the response fields you defined.
3. It then asks the evaluation metrics service to calculate the metrics. Metrics that use a language model as a judge call a model deployed in DIAL Core.
4. Scores, timing and cost are stored, and DIAL Admin shows them in runs.

The Evaluation menu appears in DIAL Admin when the Evaluation API URL is configured. See DIAL Admin evaluation settings. To deploy and configure the evaluation services, see DIAL Evaluations.

Components and repositories

Component	Repository	Role
DIAL Admin	ai-dial-admin-frontend	The Test Suites, Runs and Datasets screens.
Evaluation framework backend	ai-dial-admin-evaluation-framework-backend	Stores datasets, test suites and runs, executes runs against deployments, calls the metrics service and keeps the results. The repository also contains eval-cli.
Evaluation metrics service	ai-dial-admin-evaluation-metrics	Calculates the metrics. It provides the metrics listed in Available metrics and reports its metric list to the backend.
DIAL Core	ai-dial-core	Hosts the evaluated deployments and the judge models, and proxies MCP calls.

Concepts

Concept	Description
Dataset	A reusable collection of test cases with a defined schema. Datasets are independent of test suites and can be shared across many of them. Test cases you add directly in a test suite stay private to it until you publish them as a dataset. See Datasets.
Test case	One row of test data in a dataset: the input for the target and, optionally, the expected output. Fields can be string, number, integer, boolean, object, array or file. A test case can be multi-turn.
Metric	A scoring function that evaluates a response, for example Faithfulness, Answer Relevancy or Exact Match. A metric is attached to a test suite, where its inputs are bound to test case fields, response fields or constants.
Test suite	Ties together a target (a model, an application or an MCP tool), a dataset, a request template and metrics. It supports the OpenAI Chat Completions, OpenAI Responses and Anthropic Messages APIs. See Test suites.
Run	One execution of a test suite. It records per-test-case scores, an overall score, status, timing, cost and traces. Runs can be stopped, compared and exported. See Runs.

Typical workflow

1. Create a dataset with the test cases and their schema, or start from a test suite and add test cases manually. See Datasets.
2. Create a test suite: select the target and its method, link the dataset, and define the request template and response columns. See Test suites.
3. Add metrics to the test suite, bind their inputs, and set the score settings and the score threshold. See Metrics.
4. Run the test suite. Use Try Out on one test case first to check the request.
5. Read the results on the run Summary and Extraction result tabs. See Reading run results.
6. Improve and repeat: change the application, prompt or model, run again, then compare runs and watch the trends.

Common patterns

Goal	Approach
Check an application after a change	Keep one test suite, run it before and after the change, and compare the two runs. A drop in the heat map or in the metric statistics shows the regression.
Evaluate a RAG application	Extract the answer and the retrieved context from the response into response columns, add retrieval and faithfulness metrics, and bind their inputs to those columns and to the expected answer in the dataset.
Judge answers by your own criteria	Add the G-Eval metric with a description of the criteria or a list of evaluation steps.
Check exact or structured output	Use rule-based metrics such as Exact Match, Regex Match or DeepDiff. They need no judge model.
Test an agent's tool calls	Use the Tool Call Accuracy and Tool Call F1 metrics to compare the tool calls the agent makes with the expected ones.
Test a multi-turn dialogue	Use per-turn schema fields in the dataset, and a metric Condition such as turn.last to score only the final answer.
Gate a release	Set a score threshold in the test suite, and look at Runs Passed Threshold on the Trends tab. Run the suites from a pipeline with eval-cli.