Skip to main content

Configure alerting

This guide shows how to attach Prometheus alert rules to DIAL through the Helm chart and offers recommended alerts for common failure modes. It is for DevOps engineers running DIAL on Kubernetes with the Prometheus Operator. For the conceptual model, see Observability.

Prerequisites​

  • DIAL deployed with the dial-core Helm chart
  • Prometheus Operator running in the cluster
  • Metrics enabled so there are series to alert on

The alerting mechanism​

The dial-core chart can render a Prometheus Operator PrometheusRule from values. Enable it and supply your rules:

metrics:
  enabled: true
  prometheusRule:
    enabled: true
    rules: []          # your alert and recording rules go here
warning

DIAL ships no default alert rules — rules is empty out of the box — and the source repositories document no recommended thresholds. Everything below is general guidance, not a DIAL default. Treat the examples as starting points and tune the metric names and thresholds to the series your deployment actually exposes.

These failure modes are worth alerting on for a production DIAL deployment:

Failure modeSignal to watch
Upstream model errorsRising rate of 5xx responses from adapter or Core HTTP metrics
Adapter timeoutRequest latency for an adapter exceeding its configured timeout
Storage unavailableErrors on object-storage or Redis calls; failed health checks
Token or rate-limit exhaustionSpike in 429 responses
Degraded latencyHigh request-duration percentiles on Core or an adapter

Example PrometheusRule​

The rule below alerts on an elevated HTTP error rate. The metric name is illustrative — replace http_server_requests_total and its labels with the actual series your instrumentation exposes (see the metric-name caveat).

metrics:
  prometheusRule:
    enabled: true
    rules:
      - alert: DialHighErrorRate
        expr: |
          sum(rate(http_server_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_server_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "DIAL 5xx error rate above 5% for 10 minutes"

For usage and cost alerts — for example a spike in token consumption — use the usage data in DIAL Analytics rather than the runtime-health metrics above.

Result​

Prometheus evaluates your rules and routes firing alerts through your Alertmanager configuration.

Further reading​