DIAL Log Parser
DIAL Log Parser converts DIAL Core request logs into Parquet datasets partitioned by deployment and date. This reference is for platform engineers with existing Core log files and writable dataset storage. The output supports offline analysis and evaluation.
How parsing works
The parser recursively lists the input directory, matches file names, and selects files for one date. It normalizes request records and writes deployment/date partitions. The implementation derives the partition date from the filename, rather than the request timestamp.
An existing deployment field is retained. When absent, the parser extracts it from /openai/deployments/<DEPLOYMENT>/chat/completions or /openai/deployments/<DEPLOYMENT>/embeddings. Rows without a deployment are removed, and invalid Arrow batches are skipped with logging. See batch processing.
Log Parser is a finite batch job, with no HTTP port or health endpoint. Scheduling belongs to the deployment's job scheduler. It writes Parquet rather than populating DIAL Analytics, whose own ingestion reads raw logs into ClickHouse. The Enrichment Runner does not consume this dataset as its pipeline input.
Output schema
The Arrow schema defines the output fields. Other raw-log fields are not part of the declared output schema.
| Field | Type and content |
|---|---|
deployment | String partition key, stored in the directory path |
date | date32 partition key, from the log filename |
apiType | Nullable string identifying the API type |
chat | Nullable struct containing string id |
project | Nullable struct containing string id |
user | Nullable struct containing string id and title |
token_usage | Nullable struct: integer completion_tokens, prompt_tokens, and total_tokens; floating-point deployment_price and price |
execution_path | Nullable list of strings |
trace | Nullable struct containing string trace_id, core_span_id, and core_parent_span_id |
request | Nullable struct containing string protocol, method, uri, time, and body |
response | Nullable struct containing string status, upstream_uri, and body |
assembled_response | Nullable JSON string, including an assembled streaming response when present in the source |
question | Nullable string from the final message, only when that message has role user and string content |
answer | Nullable value from the first response choice's message.content |
Missing schema fields are filled with nulls. Invalid JSON in request bodies or assembled responses produces a warning and a null extracted value. A non-object response root also yields a null answer. See question and answer extraction.
Note
The upstream README calls the partition field
deployment_name. The code writesdeployment; dataset readers must use that name.
Dependencies and deployment
| Dependency | Purpose |
|---|---|
| Stored DIAL Core logs | Input JSON records in filenames matching the configured pattern; gzip files are supported |
| Input storage access | Read and list access to local files or the configured filesystem backend |
| Output storage access | Write access, plus deletion rights for partitions replaced during a rerun |
| External scheduler | Optional recurring execution; the image processes one date and exits |
The image is epam/ai-dial-log-parser. Its Dockerfile uses Python 3.12 and runs as appuser, UID 1001. The package dependencies include fsspec, PyArrow, and S3 support. Other filesystem protocols require their implementation to be available in the runtime image.
Configuration, container examples, rerun behavior, and dataset verification are in Configuration.
Warning
Output contains request and response bodies and can include personal data. Restrict dataset access and retention independently of the source logs. Reruns replace matching output partitions without a transactional rollback; keep source logs available for recovery.