Skip to main content

Run DIAL with a self-hosted Ollama model

In this tutorial, you will run DIAL locally and connect it to a self-hosted model served by Ollama. By the end, you will chat with an open-source model through DIAL Chat at http://localhost:3000, with everything running on your own machine. No cloud account or model API key is required — Docker downloads the model the first time you start the stack.

Prerequisites​

  • Docker Engine with Docker Compose v2.20.0 or higher. See the Docker documentation.
  • Free TCP ports 3000 (DIAL Chat), 8080 (DIAL Core), and 11434 (Ollama) on your machine.
  • Several gigabytes of free disk space for the downloaded model.

What you'll build​

A local DIAL stack connected to Ollama, which serves open-source models through an OpenAI-compatible API. You choose which model to download — a chat model, a vision model, an embedding model, or all three — and DIAL Core exposes each one as a deployment you can select in DIAL Chat.

Project structure​

The dial-docker-compose/ollama directory looks like this:

ollama/
├── docker-compose.yml
├── core/
│   └── config.json
└── ollama_setup/
    ├── app.py
    ├── Dockerfile
    └── requirements.txt

The docker-compose.yml adds three services to the shared common.yml base: the Ollama server, an ollama-setup job that pulls the models you request, and the OpenAI adapter that fronts the vision model.

Step 1: Get the Docker Compose files​

Clone the DIAL repository and change into the ollama directory:

git clone https://github.com/epam/ai-dial.git
cd ai-dial/dial-docker-compose/ollama

Verify: the directory contains docker-compose.yml, a core/ directory, and an ollama_setup/ directory.

Step 2: Choose a model to run​

Ollama supports a wide range of open-source models. Decide which modality you need — a text chat model, a multimodal vision model, or an embedding model — then pick a model from Ollama Search.

These models have been tested with this setup:

ModalityTested models
Chatllama3.1:8b-instruct-q4_0, mistral:7b-instruct-q4_0, phi3.5:3.8b-mini-instruct-q4_0, gemma2:2b-instruct-q4_0
Visionllava:7b-v1.6-mistral-q4_0, llava-phi3:3.8b-mini-q4_0
Embeddingnomic-embed-text:137m-v1.5-fp16, bge-m3:567m-fp16

Step 3: Set the models in .env​

Create a .env file in the ollama directory and set a variable for each model you want. Set only the ones you need — a model that isn't set won't be downloaded.

OLLAMA_CHAT_MODEL=llama3.1:8b-instruct-q4_0
OLLAMA_VISION_MODEL=llava:7b-v1.6-mistral-q4_0
OLLAMA_EMBEDDING_MODEL=nomic-embed-text:137m-v1.5-fp16

The ollama-setup service reads these variables and pulls the matching models when the stack starts.

Verify: the .env file exists in the ollama directory and contains at least one OLLAMA_*_MODEL variable.

Step 4: Launch DIAL​

Start the stack, pulling and loading the models you requested:

docker compose up --abort-on-container-exit

A lightweight Ollama model is a few gigabytes, so the first run may take several minutes depending on your connection and the model size.

Verify: the ollama-setup service prints The Ollama server is up and running. once the models are loaded.

Step 5: Chat with the model​

  1. Open http://localhost:3000 in your browser.
  2. Open the Catalog and select the deployment that matches your model:
    • Self-hosted chat model for OLLAMA_CHAT_MODEL
    • Self-hosted vision model for OLLAMA_VISION_MODEL
  3. Send a prompt.

Verify: the model responds. Your local DIAL deployment is now serving an open-source model.

note

Vision models in this setup do not stream their responses and are more computationally expensive than chat models, so a reply can take minutes. The embedding model is available under the deployment name embedding-model and is called through http://localhost:8080/openai/deployments/embedding-model/embeddings.

How it fits together​

The chat and embedding models point directly at Ollama's OpenAI-compatible endpoint, because Ollama speaks the OpenAI protocol natively. The vision model is routed through the OpenAI adapter, which handles image attachments:

"chat-model": {
    "type": "chat",
    "displayName": "Self-hosted chat model",
    "endpoint": "http://ollama:11434/v1/chat/completions"
},
"vision-model": {
    "type": "chat",
    "displayName": "Self-hosted vision model",
    "endpoint": "http://adapter-openai:5000/openai/deployments/vision-model/chat/completions",
    "inputAttachmentTypes": ["image/png", "image/jpeg"],
    "upstreams": [
        { "endpoint": "http://ollama:11434/v1/chat/completions", "key": "dummy-key" }
    ]
}

Note

This sample runs the standalone DIAL OpenAI Adapter, which will be deprecated. For your own deployments, use DIAL Model Provider, which serves the same adapter under /compat/openai. To move a configuration, see Migrate from the standalone adapters.

What you learned​

  • How to run DIAL with a self-hosted model and no cloud dependency
  • How to select Ollama models through the .env file
  • How DIAL Core routes a model directly to Ollama, or through the OpenAI adapter for vision

Further reading​