Skip to main content

Configure tracing with OpenTelemetry

This guide enables distributed tracing across DIAL so you can follow a single request from DIAL Core through an adapter, interceptor, and application. It is for DevOps engineers who already run DIAL and have an OTLP-compatible collector or tracing backend. For the conceptual model, see Observability.

Prerequisites​

  • A running DIAL deployment (Docker Compose or Kubernetes)
  • An OpenTelemetry Collector, or a tracing backend that accepts OTLP (such as Jaeger, Grafana Tempo, or Zipkin)
  • Access to set environment variables on each DIAL component you want to trace

How tracing flows across components​

DIAL Core assigns each request a trace context and propagates it to downstream components. Core emits the following identifiers, which DIAL Analytics also stores with each usage row, letting you join a trace to its usage data:

IdentifierMeaning
trace_idThe OpenTelemetry trace ID for the whole request
core_span_idThe span ID generated by DIAL Core for a call
core_parent_span_idThe span ID of the caller that invoked core_span_id
execution_pathA /-separated call stack, for example app1/app2/model1

A request that fans out across applications and models produces one trace with nested spans, so you can see which leg was slow or failed.

Choose an OTLP transport​

OpenTelemetry defines three OTLP transports, and DIAL components don't all support the same ones:

TransportPortSet with
gRPC4317OTEL_EXPORTER_OTLP_PROTOCOL=grpc
HTTP/protobuf4318OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
HTTP/JSON4318OTEL_EXPORTER_OTLP_PROTOCOL=http/json

Set OTEL_EXPORTER_OTLP_PROTOCOL explicitly rather than relying on a component's default. DIAL Chat, for example, does not support the gRPC transport and requires http/json on port 4318 — see Configure DIAL Chat. The sections below give a working configuration for each component.

Configure DIAL Core​

DIAL Core reads the standard OpenTelemetry environment variables. Set the gRPC transport and the traces-specific OTLP endpoint to enable the trace exporter:

OTEL_SERVICE_NAME=dial-core
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://<YOUR_COLLECTOR_HOST>:4317

Note

DIAL Core needs the signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to enable traces — setting only the generic OTEL_EXPORTER_OTLP_ENDPOINT is not enough in current versions.

note

DIAL Core is a Java service. These variables are read by the OpenTelemetry Java agent, not by config.json. How the agent jar is attached to the JVM is set at the image or deployment level — confirm the attachment mechanism for your deployment before relying on these variables.

Configure DIAL Chat​

DIAL Chat is a Node.js service. It does not support the gRPC OTLP transport — point it at the collector's HTTP/JSON endpoint on port 4318 instead of the gRPC port used by DIAL Core:

OTEL_SERVICE_NAME=dial-chat
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=http://<YOUR_COLLECTOR_HOST>:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/json

Set OTEL_SDK_DISABLED=true to turn the Node.js instrumentation off entirely.

Configure DIAL Admin​

Admin Frontend uses the Node.js OpenTelemetry SDK with HTTP/JSON exporters. Point it at your collector's HTTP endpoint; see DIAL Admin telemetry settings for its variables and defaults. The optional Admin Backend is a separate Java service; its instrumentation does not configure the frontend.

Configure adapters and applications​

Adapters and applications built on the DIAL SDK are Python services. They read these variables:

OTEL_RESOURCE_ATTRIBUTES=service.name=<YOUR_SERVICE_NAME>
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_ENDPOINT=http://<YOUR_COLLECTOR_HOST>:4317
OTEL_PYTHON_LOG_CORRELATION=true

When tracing is enabled, the SDK instruments outgoing HTTP calls (requests, aiohttp, httpx, urllib) and FastAPI request handling, so adapter-to-provider calls appear as child spans. OTEL_PYTHON_FASTAPI_EXCLUDED_URLS excludes paths such as health checks from tracing.

For the full SDK telemetry surface — TelemetryConfig, programmatic setup, and every supported variable — see the SDK telemetry reference.

Correlate traces with logs​

Set OTEL_PYTHON_LOG_CORRELATION=true on SDK-based components to inject trace_id and span_id into log lines. With correlation on, you can pivot from a slow trace to the exact log records emitted during that request.

Result​

Each request produces a single trace spanning every configured component. Open your tracing backend and search by trace_id to see the span tree, including adapter calls to upstream providers.

warning

DIAL does not expose its own trace-sampling configuration. Sampling is governed by the standard OpenTelemetry agent and SDK environment variables (for example OTEL_TRACES_SAMPLER), which are general OTEL behavior rather than a DIAL feature. Validate sampling against your collector and backend.

Further reading​