DIAL Log Parser configuration

DIAL Log Parser reads configuration from command-line options and DIAL_LOG_PARSER_* environment variables. This reference is for platform engineers with readable Core logs and a writable Parquet output location. It assumes familiarity with the output schema and storage permissions.

Precedence

The command defines five options, with automatic environment binding. Explicit command-line values take precedence over environment values; defaults apply when neither supplies a value. There is no service configuration file.

Environment variables

Variable	Default	Description
`DIAL_LOG_PARSER_INPUT`	—	Required input directory or filesystem URL. Files are listed recursively.
`DIAL_LOG_PARSER_OUTPUT`	—	Required output dataset directory or filesystem URL. Matching deployment/date partitions can be replaced.
`DIAL_LOG_PARSER_DATE`	Yesterday in the process's local date	Date to process, formatted `YYYY-MM-DD`. Computed at process initialization; set explicitly for retries or historical runs.
`DIAL_LOG_PARSER_DEBUG`	`false`	Enables debug logging. Debug output can include malformed request or response bodies.
`DIAL_LOG_PARSER_FILENAME_REGEX`	Pattern below	Regex matched against the basename of each file. Capture group 1 must be an ISO date.

The default filename pattern is defined in the source:

date=(\d{4}-\d{2}-\d{2})(\d+)-(\w{8}-\w{4}-\w{4}-\w{4}-\w{12}).log(.gz)?

This is the literal expression, including its unescaped dots. It uses Python re.match, not fullmatch. Custom patterns must retain the date as their first capture group; filenames without a match are skipped with a warning. The date is parsed with datetime.date.fromisoformat; invalid matched dates fail processing. See filename parsing.

Command-line options

Option	Environment equivalent	Value
`-i`, `--input`	`DIAL_LOG_PARSER_INPUT`	Input directory or URL
`-o`, `--output`	`DIAL_LOG_PARSER_OUTPUT`	Output directory or URL
`-d`, `--date`	`DIAL_LOG_PARSER_DATE`	`YYYY-MM-DD`
`--debug`	`DIAL_LOG_PARSER_DEBUG`	Flag; no argument
`--filename-regex`	`DIAL_LOG_PARSER_FILENAME_REGEX`	Regex string
`--help`	—	Displays usage and exits

The container entrypoint already invokes python -m aidial_log_parser.parse_logs; options follow the image name. A Python environment with the published package can invoke that same module directly.

Storage credentials

Filesystem URLs are resolved through fsspec in the parser. Local paths and s3:// URLs use implementations included in the published image. Backend credentials follow the selected filesystem's configuration; they are not parser options.

Variable	Default	Description
`AWS_ACCESS_KEY_ID`	—	S3 access-key identity when using explicit AWS credentials
`AWS_SECRET_ACCESS_KEY`	—	S3 secret access key, supplied from the deployment's secret store
`AWS_SESSION_TOKEN`	—	Session token when the supplied credentials are temporary

The S3 filesystem credential chain can supply credentials instead of this explicit set. The identity needs input listing and reads, plus output writes and deletion permissions for replaced partitions. Mounted local directories must be accessible to container UID 1001.

Container examples

Replace <VERSION> with an image tag from Docker Hub in both examples. This local example assumes ./logs contains source files and ./parsed-logs already exists with write access for UID 1001. Choose an input date present in the filenames.

docker run --rm \
  --mount type=bind,source="$(pwd)/logs",target=/input,readonly \
  --mount type=bind,source="$(pwd)/parsed-logs",target=/output \
  epam/ai-dial-log-parser:<VERSION> \
  --input /input --output /output --date <YYYY-MM-DD>

For S3, this example forwards credentials already set in the calling environment:

docker run --rm \
  -e AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY \
  -e AWS_SESSION_TOKEN \
  epam/ai-dial-log-parser:<VERSION> \
  --input s3://<INPUT_BUCKET>/<LOG_PREFIX>/ \
  --output s3://<OUTPUT_BUCKET>/<DATASET_PREFIX>/ \
  --date <YYYY-MM-DD>

A recurring deployment uses a batch scheduler to invoke the container once per date. Explicit dates make reruns independent of the scheduler's timezone and retry time. The container has no persistent listener or built-in schedule.

Reruns and data handling

The writer uses PyArrow existing_data_behavior="delete_matching". Under PyArrow's replacement rules, each encountered partition directory is deleted before new output is written. Partitions absent from a rerun's output are not a complete snapshot replacement.

Warning

Rerunning against an incomplete source can replace a partition with fewer rows. A failed write can leave partially replaced output. Preserve the raw logs and recover by rerunning the complete date into a controlled output location. Debug logging can print malformed payload contents; restrict access to job logs.

The input is read without deletion. Output has no parser-managed retention, encryption policy, or access-control API. Storage policies must protect both raw and parsed content.

Verification and troubleshooting

Successful writes log each file's size and row count. Verify dataset partitions and extracted fields after the run; process completion alone does not prove every source row was retained.

The upstream tests read the directory with typed deployment and date partition keys:

import pyarrow as pa
import pyarrow.dataset as ds

dataset = ds.dataset(
    "<PARQUET_DATASET_PATH>",
    format="parquet",
    partitioning=ds.partitioning(
        pa.schema([
            pa.field("deployment", pa.string()),
            pa.field("date", pa.date32()),
        ])
    ),
)
print(dataset.head(10, columns=["deployment", "date", "question", "answer"]))

Symptom	What to check
No selected files	Requested date, input prefix, filename pattern, and input list permissions
`FileNotFoundError: Input directory … does not exist`	The input location is missing or lists no entries; the check runs before any date filter
Files skipped with invalid-name warnings	The regex matches the basename and captures its ISO date first
Rows removed for invalid deployment	Provide `deployment` or a recognized chat-completions or embeddings URI
Null question	The last message must have role `user` and string content
Null answer	Check the assembled response JSON and its first choice's `message.content`
Permission errors	Storage credentials, output deletion rights, and local ownership for UID `1001`
Fewer output rows after a retry	Matching partitions are replaced, not appended; check source completeness

Precedence​

Environment variables​

Command-line options​

Storage credentials​

Container examples​

Reruns and data handling​

Verification and troubleshooting​

Further reading​