Skip to main content

Usage limits and cost control

DIAL meters caller consumption and rejects requests when applicable allowances are exhausted. This page explains token, request, and spending limits for architects and operators. Limits are associated with roles, so read the authentication and access-control model first.

Why limits exist​

An open gateway to language models is also an open path to runaway spend and resource contention. Limits let an organization manage consumption, prevent overuse, and allocate model capacity across users and applications. Roles provide reusable policies for users and API keys; assigning the same role to several users does not by itself create a shared team budget.

Token limits govern token volume, request limits govern call counts, and cost limits govern recorded spending. These controls address different concerns and can apply together.

Token rate limiting​

Token limits count input and output tokens per deployment, across minute, day, week, and month windows. Administrators can assign different allowances to different roles, such as a larger allowance for production use and a smaller one for experimentation.

The minute window is floating: usage ages out as time passes. Daily, weekly, and monthly windows reset at fixed calendar boundaries. The deployment-wide rate limit schedule sets the time zone, the first day of the week, and the reset time. Without an override, boundaries use midnight UTC and weeks begin on Monday.

Request rate limiting​

Request limits count calls to a deployment independently of their token volume. The role's requestHour limit uses a floating window; requestDay uses a fixed calendar window. A caller can therefore reach a request limit even when its token allowance is not exhausted.

See the Roles reference for the token and request fields.

Cost-based rate limiting​

The role's costLimit caps recorded spending in USD across models, using minute, day, week, and month windows. Core uses the configured model pricing to calculate consumption. Token allowances alone do not express a consistent monetary allowance because models have different prices.

For model calls by users authenticated through an identity provider, spending limits can select budgets instead:

  • off: the role's costLimit applies.
  • shadow: Core checks and records budgets, but only the role's costLimit can reject a call on spending grounds.
  • enforce: budgets replace the role's costLimit; a model call without a budget that can pay is rejected.

API-key callers continue to use the role's costLimit in all three modes. Token and request limits still apply alongside the selected spending control.

How limits are enforced​

Core checks recorded consumption before forwarding a model request. A request must pass the effective token and request limits and the selected spending control. A rejection identifies the constraint that was reached. These are consumption checks, not a reservation of the exact cost of the next completion.

When several eligible roles define a limit, Core takes the largest value for each metric and time window. For deployment token and request limits, eligible roles are the caller's roles that match the deployment's userRoles, when that field is set. Cost limits use the caller's roles. If no role defines the relevant limits, the default role applies; without a configured default, usage is unlimited. Assigning an additional restrictive role therefore does not necessarily lower an existing allowance.

Monitoring and analytics​

DIAL Analytics ingests request analytics for consumption monitoring and dashboards. Core writes an analytics entry per request; analytics and tracing settings describe the recorded data and optional enrichment.

Administrators can compare actual usage with allowances and adjust policies to match demand. Logs and analytics are additional copies of potentially sensitive request data; their access and retention belong in the data protection model.

Further reading​