Skip to main content

Run DIAL with a self-hosted vLLM model

In this tutorial, you will run DIAL locally and connect it to a Hugging Face model served by vLLM, a high-throughput inference engine. By the end, you will chat with a self-hosted model through DIAL Chat at http://localhost:3000. No cloud account or model API key is required — vLLM downloads the model from Hugging Face the first time you start the stack.

Prerequisites​

  • Docker Engine with Docker Compose v2.20.0 or higher. See the Docker documentation.
  • Free TCP ports 3000 (DIAL Chat), 8080 (DIAL Core), and 8000 (vLLM) on your machine.
  • Several gigabytes of free disk space for the downloaded model.

What you'll build​

A local DIAL stack connected to vLLM. vLLM serves a Hugging Face model through an OpenAI-compatible API, so DIAL Core routes to it directly — no adapter is needed. DIAL exposes the model as a single chat deployment you select in DIAL Chat.

Project structure​

The dial-docker-compose/vllm directory looks like this:

vllm/
├── docker-compose.yml
└── core/
    └── config.json

The docker-compose.yml adds a single vllm service to the shared common.yml base:

include:
  - path: ../common.yml
    env_file: ./.env

services:
  vllm:
    image: vllm/vllm-openai:v0.13.0
    ports:
      - "8000:8000"
    command:
      - "${VLLM_CHAT_MODEL:-Qwen/Qwen2.5-Coder-0.5B-Instruct-GPTQ-Int4}"
      # Must match the DIAL deployment ID in core/config.json
      - "--served-model-name"
      - "dial-chat-deployment-id"
    environment:
      - VLLM_CPU_KVCACHE_SPACE=${VLLM_CPU_KVCACHE_SPACE:-1}
      - VLLM_USE_FLASHINFER_SAMPLER=0
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 120s

Step 1: Get the Docker Compose files​

Clone the DIAL repository and change into the vllm directory:

git clone https://github.com/epam/ai-dial.git
cd ai-dial/dial-docker-compose/vllm

Verify: the directory contains docker-compose.yml and a core/ directory.

Step 2: Choose a model to run​

vLLM serves chat models from Hugging Face. The setup defaults to the lightweight Qwen/Qwen2.5-Coder-0.5B-Instruct-GPTQ-Int4 model, which is small enough to run without a GPU.

To use a different model, create a .env file in the vllm directory and set the model name:

VLLM_CHAT_MODEL=Qwen/Qwen2.5-Coder-0.5B-Instruct-GPTQ-Int4
CPU-only machines

The default vllm/vllm-openai:v0.13.0 image targets GPU hosts. To run on an x86 CPU, switch the image in docker-compose.yml to a pre-built CPU image, for example public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.13.0. See the vLLM CPU installation guide for available images.

Step 3: Launch DIAL​

Start the stack, downloading and serving the model:

docker compose up --abort-on-container-exit

A lightweight Hugging Face model is a few gigabytes, so the first run may take several minutes depending on your connection and the model size.

Verify: the vllm service passes its health check (curl -f http://localhost:8000/health returns success) and the logs show the model is being served.

Step 4: Chat with the model​

  1. Open http://localhost:3000 in your browser.
  2. Open the Catalog and select the Self-hosted chat model deployment.
  3. Send a prompt.

Verify: the model responds. Your local DIAL deployment is now serving a Hugging Face model through vLLM.

How it fits together​

The deployment ID in core/config.json must match the --served-model-name passed to vLLM. DIAL Core routes the deployment straight to vLLM's OpenAI-compatible endpoint:

"dial-chat-deployment-id": {
    "type": "chat",
    "displayName": "Self-hosted chat model",
    "endpoint": "http://vllm:8000/v1/chat/completions"
}

Because vLLM speaks the OpenAI protocol, no adapter sits between DIAL Core and the model.

What you learned​

  • How to run DIAL with a self-hosted Hugging Face model served by vLLM
  • How to select a model with the VLLM_CHAT_MODEL variable and switch to a CPU image
  • Why the DIAL deployment ID must match vLLM's --served-model-name

Further reading​