DIAL Analytics environment variables
This reference covers environment variables for platform engineers operating DIAL Analytics. It assumes the service connections are configured.
Defaults
Defaults come from the application.yml and configuration documentation. — means no configured value. Actual defaults containing localhost are reproduced unchanged; container deployments need reachable service names.
Container environment
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
USE_SYSTEM_CA_CERTS | — | Any non-empty value (1, true, yes) imports certificates mounted into /certificates/ at startup. |
JAVA_OPTS | — | Extra JVM arguments appended to the launch command. |
DEBUG_OPTS | — | JVM debug arguments, applied before JAVA_OPTS. |
SPRING_CONFIG_ADDITIONAL_LOCATION | — | External YAML/properties location overriding any property here. |
Server
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
HTTP_PORT | 5000 | Port ADAS serves the API on. |
MANAGEMENT_SERVER_PORT | — | Separate port for the /actuator endpoints (probes, info, Prometheus); unset serves them on HTTP_PORT. |
Scheduled task execution
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
SCHEDULING_POOL_SIZE | 4 | Threads shared by the blob poller, the daily sync, the ingest retry sweep, and the level-file poll; below three a slow job delays another. Takes effect at startup. |
ClickHouse connection
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
CLICKHOUSE_HOST | localhost | Required for deployment. ClickHouse host. |
CLICKHOUSE_PORT | 8123 | ClickHouse HTTP interface port. |
CLICKHOUSE_DATABASE | — | Default database for both connections. Blank means default. |
CLICKHOUSE_USER | default | Required for deployment. Read-only user. The default is the local image's passwordless default user. |
CLICKHOUSE_PASSWORD | — | Required for deployment. Password for the read-only user. Supply from the deployment's secret store. |
ClickHouse writer credential
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
CLICKHOUSE_WRITER_USER | default | Required for deployment. User holding the write and DDL grants. |
CLICKHOUSE_WRITER_PASSWORD | — | Required for deployment. Password for the writer user. Supply from the deployment's secret store. |
ClickHouse schema migrations
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
CLICKHOUSE_MIGRATIONS_ENABLED | true | Run the ClickHouse migrations at startup; true also when the property is absent. Set false when the schema is provisioned out of band. A migration failure aborts startup. |
PostgreSQL table catalog
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
POSTGRES_HOST | localhost | Required for deployment. Catalog host. |
POSTGRES_PORT | 5432 | Catalog port. |
POSTGRES_DATABASE | testdb | Required for deployment. Catalog database name. |
POSTGRES_OPS | — | JDBC URL query string appended after ?, for example sslmode=require&ApplicationName=adas. |
POSTGRES_DATASOURCE_USERNAME | postgres | Required for deployment. Catalog user; needs read/write on the catalog tables, and DDL when migrations run. |
POSTGRES_DATASOURCE_PASSWORD | postgres | Required for deployment. Catalog password. Supply from the deployment's secret store. |
POSTGRES_MIGRATIONS_ENABLED | true | Run the PostgreSQL migrations at startup. Set false when the schema is provisioned out of band. A migration failure aborts startup. |
Blob source
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
BLOB_PROVIDER | azureblob | jclouds provider id, azureblob or aws-s3. filesystem is test-scoped and unavailable at runtime. |
BLOB_BUCKET | logs | Required; must match the producer. Container or bucket name holding the logs. |
BLOB_IDENTITY | — | Required for shared-key authentication. Azure storage account name, or AWS access key id; blank with BLOB_CREDENTIAL selects Azure managed identity / AWS workload identity. |
BLOB_CREDENTIAL | — | Required for shared-key authentication. Azure account key, or AWS secret access key, from the deployment's secret store; blank with BLOB_IDENTITY selects managed identity / workload identity. |
BLOB_ENDPOINT | — | Required for managed identity and S3. Endpoint override, https://<STORAGE_ACCOUNT>.blob.core.windows.net or https://s3.<REGION>.amazonaws.com. Blank uses the provider default, which Azure AD auth cannot resolve and which signs S3 for us-east-1. |
Query execution budget
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
QUERY_BUDGET_TIMEOUT | 20s | Wall-clock ceiling per request, shared by a row-mode query's include_total count. At least 1s. |
QUERY_BUDGET_MAX_MEMORY | 2GB | Hard memory ceiling per query per node. Keep below the ClickHouse container's memory limit. |
QUERY_BUDGET_SPILL_THRESHOLD | 1GB | Aggregation-state size at which a query spills to disk instead of growing. Must be strictly below max-memory; 0 disables spilling. |
Query deduplication
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
FORCE_FINAL_QUERIES | true | Migration switch: true reads every user-facing query FINAL whatever it says; false honours the query's FINAL. |
Saved queries
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
SAVED_QUERIES_MAX_BODY_BYTES | 65536 | Maximum combined serialized UTF-8 size of a saved query's query, sql, time and chart payload, and of each dashboard inline query's query, sql and chart; larger is rejected with 422. |
Arrow Flight SQL endpoint
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
FLIGHT_ENABLED | false | Master switch. When false no listener is opened and no Flight components are created. |
FLIGHT_PORT | 32010 | gRPC port for the listener; 0 binds an ephemeral port (used by tests). |
FLIGHT_BATCH_ROWS | 1024 | Flush an Arrow record batch to the client after this many rows. |
FLIGHT_BATCH_MAX_BYTES | 4194304 | Byte threshold for flushing: this much buffered Arrow memory in bytes, whichever comes first. |
FLIGHT_MAX_CONCURRENT_STREAMS | 4 | Cap on concurrently executing result streams; excess requests fail fast with retryable RESOURCE_EXHAUSTED. |
FLIGHT_REQUEST_MEMORY_LIMIT_BYTES | 67108864 | Hard bound on Arrow memory a single request may allocate, via a per-request child allocator. |
FLIGHT_MAX_ROWS | 10000 | Most rows this transport returns from ClickHouse, replacing the shared 1000-row maximum for Flight only. A larger result without an explicit LIMIT at or below the cap is refused with INVALID_ARGUMENT, not truncated. Raising it trades JVM heap for reach. |
Table catalog limits
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
TABLES_MAX_COLUMNS_PER_TABLE | 256 | Maximum exposed columns per table, enforced at create, on catalog import, and when a PATCH adds columns. |
TABLES_MAX_TABLES | 1000 | Maximum table definitions in the catalog, enforced at create and on catalog import. |
TABLES_MAX_WRITE_BATCH_ROWS | 10000 | Maximum rows accepted in one POST /v1/tables/{name}/rows batch, and the chunk size an OTLP export is written in. |
OTLP sinks
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
OTEL_SINK_ENABLED | true | Serve the OTLP surface; true also when the property is absent. When false every /v1/otel/** endpoint answers 503 otel_disabled. |
OTEL_MAX_ROWS_PER_REQUEST | 50000 | Maximum rows one export may flatten into (one per log record, span, or metric data point); larger is rejected with 422 before anything is written. |
OTEL_DEFAULT_RETENTION_DAYS | 30 | Row-expiry TTL in days given to a sink's landing tables when the create request omits retention_days. 0 creates them with no TTL. |
Catalog transfer
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
CATALOG_IMPORT_MAX_FILE_SIZE | 20MB | Maximum size of the uploaded bundle file. |
CATALOG_IMPORT_MAX_REQUEST_SIZE | 25MB | Maximum size of the whole multipart request, all parts included. |
Pipeline read budget
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
PIPELINE_BUDGET_TIMEOUT | 120s | Wall-clock ceiling per pipeline statement, and per materialize call; the relayout's writer hold waits this plus 15 s. Keep below the runner's HTTP read timeout. |
PIPELINE_BUDGET_MAX_MEMORY | 2GB | Hard memory ceiling per statement per node, relayout slices (~1 GiB) included. Size it for the heaviest legitimate read, as above. |
Enrichment scan
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
ENRICHMENT_SCAN_SAFETY_LAG | 5s | Rows newer than now64(3) − safety-lag are excluded from every enrichment read—scan, group-rows and materialize—and from the drained horizon a dependent rollup waits on, which raising it holds back equally. |
ENRICHMENT_SCAN_DEFAULT_LIMIT | 1000 | Page size applied when a scan request omits limit. At least 1. |
ENRICHMENT_SCAN_MAX_LIMIT | 10000 | Upper bound on a requested limit; larger is rejected with 400. At least 1. |
Enrichment group assembly
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
ENRICHMENT_GROUP_FETCH_DEFAULT_ROWS | 200 | Members returned when a request omits limit. At least 1. |
ENRICHMENT_GROUP_FETCH_MAX_ROWS | 2000 | Upper bound on a requested limit (400 if larger) and on a group pipeline's trigger.member_select.limit (422 when armed or previewed). At least 1. |
ENRICHMENT_GROUP_FETCH_MAX_BYTES | 8MB | Serialized-size ceiling for the returned members; exceeding it trims the group and sets truncated. |
Enrichment Class-1 materialize
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
ENRICHMENT_MATERIALIZE_DEFAULT_LIMIT | 10000 | Window row count applied when a materialize request omits limit. At least 1. |
ENRICHMENT_MATERIALIZE_MAX_LIMIT | 100000 | Upper bound on a requested limit; larger is rejected with 400. At least 1. |
Enrichment provenance reconciliation
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
ENRICHMENT_PROVENANCE_RECONCILE_ENABLED | true | Adds the pipeline_generation provenance column to enrich targets that lack it, at startup. |
Enrichment scan cursor keyring
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
ENRICHMENT_CURSOR_ACTIVE_KEY_ID | k1 | Id of the keyring key that signs newly issued cursors. Must match an entry in the keyring, or startup fails. |
ENRICHMENT_CURSOR_KEY_ID | k1 | Stable id of the first keyring key, embedded in issued cursors. |
ENRICHMENT_CURSOR_SECRET | — | Required for deployment. HMAC secret for that key, from the deployment's secret store. Blank is accepted and logs scan-cursor integrity is NOT enforced. |
Aggregate pipelines
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
AGGREGATE_ENABLED | true | Master switch for the refresh scheduler. When false nothing runs on a schedule; on-demand refresh and the whole API stay available. |
AGGREGATE_TICK_CRON | 0 * * * * * | How often the scheduler looks for pipelines whose next run is due. Bounds the finest cadence any single pipeline can achieve. |
AGGREGATE_SCHEDULER_POOL_SIZE | 1 | Threads on the aggregate tick's own scheduler, separate from SCHEDULING_POOL_SIZE. Raising it lets ticks overlap; refreshes within one tick stay serial. At least 1. |
AGGREGATE_SAFETY_LAG | 5s | Rows newer than now64(3) − safety-lag are excluded from a refresh window. |
AGGREGATE_WINDOW_LIMIT | 100000 | Maximum input rows one refresh probes. A run that fills it reports has_more and the next tick continues from the advanced cursor. At least 1. |
AGGREGATE_PREVIEW_ROWS | 5 | Rows returned by the dry-run sample (POST /v1/pipelines/{name}/preview), which writes nothing. At least 1. |
AGGREGATE_PREVIEW_TIMEOUT | 30s | Wall-clock ceiling for the preview; exceeding it refuses it with 422 pipeline_preview_budget_exceeded. At least 1s. |
AGGREGATE_PREVIEW_MAX_MEMORY | 2GB | Hard memory ceiling for the preview, per query per node. Keep below the ClickHouse container's memory limit. |
AGGREGATE_PREVIEW_SPILL_THRESHOLD | 1GB | Aggregation-state size at which the preview spills to disk. Must be strictly below max-memory; 0 disables spilling. |
Data sync schedule
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
SYNC_CRON | 0 0 1 * * * | Spring cron for the daily run. |
SYNC_DAY_OFFSET | 1 | Days back from today to target; 1 is yesterday. |
SYNC_BATCH_SIZE | 10000 | Rows per INSERT batch. At least 1. |
SYNC_MAX_BATCH_BYTES | 134217728 | Byte ceiling per INSERT batch, flushing on whichever bound is hit first. Budget this per target per in-flight run. Between 1 MiB and 357913941. |
SYNC_MAX_LINE_BYTES | 67108864 | Max decompressed bytes of one log line; a longer line is skipped and counted as malformed. Between 1024 and 1073741824. |
SYNC_PATH_PREFIX | logs | Required; must match the producer. Base path within the container preceding the date directory. Surrounding / are ignored. |
SYNC_DATE_PATH_PATTERN | yyyy/MM/dd | Required; must match the producer. java.time pattern for the date directory; yyyy-MM-dd for a single-directory producer, blank for one that writes no date directory at all. Must distinguish dates. |
SYNC_FILE_NAME_PREFIX_PATTERN | 'date='yyyy-MM-dd | Required; must match the producer. java.time pattern for the blob file name's leading text, before the embedded epoch. Blank is legal under a date directory; without one it must carry the date. |
SYNC_VALIDATE_SCHEMA | true | Reconcile each target's mapping, the ingest ledger table, and the ingest failure ledger table against the live schema at startup and fail fast on drift. Set false for local runs without ClickHouse. |
SYNC_INLINE_FILES_DROP_TRUNCATED | true | Cut a payload body DIAL Core truncated at 4 MiB back to before the inline file the cut landed in. |
SYNC_INLINE_FILES_MAX_SIZE | — (no limit) | Remove inline files whose stored base64 is longer than this (256KB, 1MB; a bare number is bytes); 0 removes all. Negative fails startup. |
SYNC_REQUEST_BODY_MODE | whole | whole stores complete bodies. whole-with-path adds a hidden parent path and supports rollback from delta. delta stores new chat messages without shared parent system keys and tools. Others fail startup. |
SYNC_TABLE | dial_usage_log | First ingest target's table—the usage log; its retention is SYNC_USAGE_LOG_TTL_DAYS. |
SYNC_MAPPING_FILE | classpath:mapping/dial-usage-log.yaml | First target's mapping file—the match-all mapping. |
SYNC_PAYLOAD_TABLE | dial_usage_log_payload | Second ingest target's table—the payload enrichment holding the five verbatim-content columns; its retention is SYNC_PAYLOAD_TTL_DAYS. |
SYNC_PAYLOAD_MAPPING_FILE | classpath:mapping/dial-usage-log-payload.yaml | Second target's mapping file—match-all, shadowing the usage log one row per event_id. Drop this target to stop capturing raw content. |
SYNC_PAYLOAD_TTL_DAYS | — (keeps the live TTL) | Payload retention in days, applied at startup. Unset preserves the live TTL (initially 90 days); -1 removes it. Values 0, below -1, or above usage-log retention fail startup. Not applied (still validated, against the other only when both are set) when CLICKHOUSE_MIGRATIONS_ENABLED=false. |
SYNC_USAGE_LOG_TTL_DAYS | — (keeps the live TTL) | Usage-log retention in days, applied at startup. Unset preserves the live TTL (initially one year); -1 removes it. Values 0, below -1, or below payload retention fail startup. Not applied (still validated, against the other only when both are set) when CLICKHOUSE_MIGRATIONS_ENABLED=false. |
SYNC_RATE_TABLE | rate_analytics | Third ingest target's table. |
SYNC_RATE_MAPPING_FILE | classpath:mapping/rate-analytics.yaml | Third target's mapping file, emitting only /rate POST records. Drop this target to stop populating rate_analytics. |
Near-real-time poller
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
POLL_ENABLED | true | Master switch; true also when the property is absent. When off, ingestion proceeds only via the scheduled run. |
POLL_CRON | 0 * * * * * | Spring cron controlling how often a poll cycle runs. |
POLL_OVERLAP | 5m | How far below the watermark to keep re-considering blobs, absorbing out-of-order writes and clock skew. |
POLL_RESTART_LOOKBACK | 1h | How far back the watermark is set at startup, so a restart re-scans a bounded recent window. |
POLL_PREVIOUS_DAY_GRACE | 1h | How long after UTC midnight to also scan the previous day's prefix. day mode only; ignored in band mode. |
POLL_FINALIZATION_LAG | 15m | band mode only: how far below the cutoff the listed slice reaches back. Set at or above the longest request/stream duration; blobs lagging further are recovered by the daily run. |
POLL_LISTING_MODE | band | band narrows each listing to a recent time slice; day lists the whole active day and makes no assumption about the key format. |
Data-sync backfill
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
BACKFILL_ENABLED | true | Master switch for the backfill endpoints. When false a trigger is rejected with 503 backfill_disabled. |
BACKFILL_MAX_DAYS | 366 | Maximum inclusive span in days one request may cover; a longer range is rejected with 400. |
Data-sync failure retry
Source: configuration reference.
| Variable | Default | Description |
|---|---|---|
SYNC_RETRY_ENABLED | true | Whether the sweep attempts due failures; failures are recorded and listed either way. |
SYNC_RETRY_CRON | 0 */15 * * * * | Spring cron controlling how often a sweep runs. |
SYNC_RETRY_INITIAL_BACKOFF | 15m | Wait after a subject's first transient failure, doubled after each further one; positive. |
SYNC_RETRY_MAX_BACKOFF | 24h | Cap on that wait; not below initial-backoff, or startup fails. |
SYNC_RETRY_MAX_ATTEMPTS | 10 | Transient failures after which a subject is parked until an operator requeues it; at least 1. |
SYNC_RETRY_MAX_PER_SWEEP | 20 | Most blobs and days one sweep attempts; at least 1. |
Ingestion and retention behavior
The poller and daily reconciliation share the same blob layout and target mappings.
Target dates and pipeline schedules use UTC. The service crons (SYNC_CRON, POLL_CRON, SYNC_RETRY_CRON, and AGGREGATE_TICK_CRON) fire in the JVM's default time zone, which is UTC only while the container's time zone is left unset. Disabling POLL_ENABLED leaves daily reconciliation and operator backfills available.
The poller's watermark is in memory; restart lookback controls its recovery window.
POLL_FINALIZATION_LAG must cover the longest request or stream; daily reconciliation recovers older arrivals.
Azure shared-key authentication needs both account name and key.
Otherwise Azure uses DefaultAzureCredential, a blob endpoint, and a managed identity with Storage Blob Data Reader access.
The storage account must have hierarchical namespace disabled. SAS tokens are unsupported.
For S3, the endpoint must select the correct region; credentials can come from workload identity or an access-key pair.
sync.targets is a list of table/mapping pairs. An external YAML file replaces the list when adding or removing targets.
At least one target is required. Each accepted record can produce a row in several targets.
blob.overrides supplies additional jclouds properties when needed.
Warning
The shipped payload target stores raw content. Removing that target stops future capture; it does not delete stored content. Unset retention variables preserve live TTLs: new tables start with 90 days for payloads and one year for usage logs. Positive retention changes can expire existing rows.
-1removes the TTL;0is invalid.
Payload retention cannot exceed usage-log retention. Retention changes require ClickHouse migrations to be enabled.
SYNC_REQUEST_BODY_MODE=delta stores only new chat messages; whole-with-path supports rollback from that format.
The signed cursor keyring can be replaced with enrichment.cursor.keys in external YAML.
Rotation adds a new key, selects it through active-key-id, and retains old verification keys while cursors remain in use.
An empty cursor secret leaves cursor integrity unenforced.
Resource budgets
Query and preview spill thresholds must remain below their memory ceilings; 0 disables spilling.
Budget violations reject the request rather than sample or truncate query results.
Pipeline statement timeouts must remain below the runner's ADAS_READ_TIMEOUT_MS.
A group fetch can issue several statements, so its HTTP timeout also needs headroom for every heavy-column read.
Aggregate scheduling runs in ADAS; SQL and LLM enrichment scheduling runs in the separate runner.
Further reading
- Security and observability — companion settings reference
- DIAL Admin analytics settings — frontend feature switches and service URLs