Storage

DIAL Core keeps its permanent data in blob storage: user files, conversations, prompts, applications, toolsets, skills, and the configuration written through the Configuration API. These static settings choose the blob store and how Core addresses it, and this page lists the permissions each provider needs. It is for platform engineers who provision storage for Core.

Core also needs Redis for volatile data; see Redis.

Settings

Setting	Default	Required	Description
`storage.provider`	`filesystem`	Yes	The blob store provider: `s3`, `aws-s3`, `azureblob`, `google-cloud-storage`, or `filesystem`.
`storage.endpoint`	—	Depends on the provider	The endpoint URL, for S3-compatible stores and Azure Blob Storage.
`storage.identity`	—	Depends on the provider	The access key or principal. Can be unset for `filesystem`, `aws-s3`, and `google-cloud-storage`.
`storage.credential`	—	Depends on the provider	The secret key. Can be unset for `filesystem`, `aws-s3`, and `google-cloud-storage`.
`storage.bucket`	`dial`	No	The bucket name.
`storage.createBucket`	`true`	No	Whether Core creates the bucket at startup. Creating a bucket needs more permissions than those listed below.
`storage.prefix`	—	No	A base prefix for every stored resource, so several environments (dev, pre-prod, prod) can share one bucket. Must not contain path separators or invalid characters.
`storage.maxUploadedFileSize`	`536870912`	No	The maximum size of an uploaded file, in bytes. A larger upload answers `413`.
`storage.overrides.*`	`jclouds.filesystem.basedir: data`	No	Key-value pairs that override provider settings, such as jclouds properties.

Provider requirements

AWS S3

Core supports user credentials and temporary credentials through IAM roles for service accounts.

User credentials: set storage.identity to the access key ID and storage.credential to the secret access key.
Temporary credentials: set up the pod in EKS and leave storage.identity and storage.credential unset.

The bucket needs these permissions:

{
  "Statement": [
    {
      "Action": [
        "s3:PutObjectAcl",
        "s3:PutObject",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucket",
        "s3:GetObject",
        "s3:GetBucketLocation",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::<YOUR_DIAL_CORE_BUCKET>/*",
        "arn:aws:s3:::<YOUR_DIAL_CORE_BUCKET>"
      ]
    }
  ],
  "Version": "2012-10-17"
}

Google Cloud Storage

Core supports user credentials and temporary credentials through application default credentials.

User credentials: set storage.credential to the path of the service account's private key JSON file and leave storage.identity unset. Alternatively, set storage.credential to the private key in PEM format and storage.identity to the client email address.
Temporary credentials: set up Workload Identity for the pod, leave storage.identity and storage.credential unset, and set the jclouds credential type:
```
{
  "storage": {
    "overrides": {
      "jclouds.oauth.credential-type": "bearerTokenCredentials"
    }
  }
}
```

The bucket needs at least Storage Bucket Viewer (roles/storage.bucketViewer) and Storage Object User (roles/storage.objectUser).

Azure Blob Storage

The storage account must have the hierarchical namespace feature turned off; Core does not support folders in Azure Blob Storage. Core supports user credentials and temporary credentials through Azure AD Workload Identity.

User credentials: set storage.identity to the service principal ID and storage.credential to its secret.

Temporary credentials: set up Azure Workload Identity for the pod, leave storage.identity and storage.credential unset, and override these properties:

{
  "storage": {
    "endpoint": "https://<STORAGE_ACCOUNT>.blob.core.windows.net",
    "overrides": {
      "jclouds.azureblob.auth": "azureAd",
      "jclouds.oauth.credential-type": "bearerTokenCredentials"
    }
  }
}

Assign the Storage Blob Data Contributor role on the storage account to the managed identity Core uses.

S3-compatible and filesystem storage

For an S3-compatible store such as MinIO, set storage.provider to s3 with an endpoint and keys:

{
  "storage": {
    "provider": "s3",
    "endpoint": "http://minio:9000",
    "identity": "<ACCESS_KEY>",
    "credential": "<SECRET_KEY>",
    "bucket": "dial"
  }
}

The filesystem provider stores blobs on a local path and suits local evaluation:

{
  "storage": {
    "provider": "filesystem",
    "bucket": "dial",
    "createBucket": true,
    "overrides": {
      "jclouds.filesystem.basedir": "/opt/data"
    }
  }
}

Settings​

Provider requirements​

AWS S3​

Google Cloud Storage​

Azure Blob Storage​

S3-compatible and filesystem storage​

Further reading​