Skip to main content

Configure roles and rate limits

This guide shows how to define roles and attach usage limits to them. A role groups together resource access and consumption limits, and applies identically whether the caller authenticated with an API key or a JWT. Limits live in the roles section of config.json.

Prerequisites​

  • A running DIAL Core instance with write access to config.json.
  • At least one key or JWT claim that resolves to the role you want to limit.

How limits combine​

  • If a limit is not set for a role, the default role's limit applies. If the default role does not set it either, the value is unlimited.
  • All limits apply in parallel — a request must satisfy every applicable limit.
  • Within a metric, a lower window is capped by the higher one. For example, an unset requestHour is still bounded by requestDay.
  • When a request is rejected, the caller is told which limit it violated.

Step 1: Set request limits​

Request limits cap how many calls a role may send to a deployment per time window. Define them under roles.<role>.limits.<deployment>.

{
  "roles": {
    "myRole": {
      "limits": {
        "chat-gpt-35-turbo": {
          "requestHour": "100000",
          "requestDay": "10000000"
        }
      }
    }
  }
}

Step 2: Set token limits​

Token limits cap throughput in tokens per minute, day, week, or month, using a floating window. Add them to the same limits.<deployment> block.

{
  "roles": {
    "myRole": {
      "limits": {
        "chat-gpt-35-turbo": {
          "minute": "10000",
          "day": "10000000",
          "week": "50000000",
          "month": "200000000"
        }
      }
    }
  }
}

Step 3: Set cost limits​

Cost limits cap spending in USD across all models available to the role, so budgets hold regardless of which model is used. Define them under roles.<role>.costLimit.

{
  "roles": {
    "myRole": {
      "costLimit": {
        "minute": 10.00,
        "day": 100.00,
        "week": 500.00,
        "month": 2000.00
      }
    }
  }
}

Step 4: Set sharing limits (optional)​

Sharing limits restrict how a role may share private resources. invitation_ttl is the link lifetime in hours; max_accepted_users caps how many users can accept it. Define them per resource type under roles.<role>.share.

{
  "roles": {
    "myRole": {
      "share": {
        "APPLICATION": {
          "invitation_ttl": "24",
          "max_accepted_users": "10"
        },
        "FILE": {
          "invitation_ttl": "24",
          "max_accepted_users": "10"
        }
      }
    }
  }
}

Defaults are 72 hours for invitation_ttl, and for max_accepted_users, 10 for APPLICATION and unlimited for other resource types.

Result​

The role myRole now enforces request, token, cost, and sharing limits. DIAL Core reloads config.json automatically, so the limits take effect within about 60 seconds. Exceeding any limit returns an error naming the violated constraint. For every field and default, see the Roles reference.

Further reading​