Skip to main content

Tutorial: build a RAG app

In this tutorial, you will build a Retrieval-Augmented Generation (RAG) Custom App that answers questions about uploaded text files. The app receives a document as an attachment, splits it into chunks, stores them in a local vector index, and uses an LLM to generate grounded answers. You will learn how to handle file attachments, call upstream models, and use streaming responses. Basic familiarity with the Getting started tutorial and the translator app tutorial is assumed.

Prerequisites​

What you will build​

A RAG application that:

  1. Accepts a text file attachment from the user.
  2. Splits the file content into chunks and creates an in-memory vector index.
  3. On each question, retrieves relevant chunks and sends them as context to an LLM.
  4. Streams the LLM's grounded answer back to the user.
User (with file) → DIAL Core → RAG App → [chunk + embed + retrieve] → DIAL Core → LLM → RAG App → DIAL Core → User

Step 1: Create the project directory​

mkdir -p dial-rag-app/rag-app
mkdir -p dial-rag-app/core
cd dial-rag-app

Final layout:

dial-rag-app/
├── docker-compose.yml
├── core/
│   └── config.json
├── sample.txt
└── rag-app/
    ├── app.py
    ├── requirements.txt
    └── Dockerfile

Step 2: Create a sample document​

Create sample.txt in the project root. This is the document you will upload to test the app:

DIAL is an open-source, enterprise-grade AI orchestration platform developed by EPAM.
It provides a Unified API compatible with the OpenAI format for models, applications, and adapters.
DIAL Core is the central routing component that handles authentication, rate limiting, and request forwarding.
DIAL Chat is the web-based user interface for interacting with models and applications.
Custom Apps are containerized services built with the DIAL SDK that implement the chat completions endpoint.
Quick Apps are no-code orchestrators that combine a model, instructions, and tools without writing code.
Code Apps are Python applications written and deployed directly in the DIAL Chat browser-based editor.
The DIAL SDK is a Python framework for building applications and adapters for DIAL.
Interceptors are middleware components that can modify requests and responses passing through DIAL Core.
Adapters translate external AI provider APIs into the DIAL Unified API format.

Step 3: Implement the RAG app​

Create rag-app/app.py:

import os
from typing import Optional

import json
import httpx
import uvicorn

from aidial_sdk import DIALApp
from aidial_sdk.chat_completion import ChatCompletion, Request, Response

DIAL_URL = os.environ.get("DIAL_URL", "http://core:8080")
MODEL_DEPLOYMENT = os.environ.get("MODEL_DEPLOYMENT", "chat-model")
OLLAMA_MODEL = os.environ.get("OLLAMA_MODEL", "llama3.2:1b")
CHUNK_SIZE = int(os.environ.get("CHUNK_SIZE", "200"))
TOP_K = int(os.environ.get("TOP_K", "3"))


def split_into_chunks(text: str, chunk_size: int) -> list[str]:
    words = text.split()
    chunks = []
    for i in range(0, len(words), chunk_size):
        chunk = " ".join(words[i : i + chunk_size])
        if chunk.strip():
            chunks.append(chunk)
    return chunks


def simple_similarity(query: str, chunk: str) -> float:
    query_words = set(query.lower().split())
    chunk_words = set(chunk.lower().split())
    if not query_words:
        return 0.0
    return len(query_words & chunk_words) / len(query_words)


def retrieve(query: str, chunks: list[str], top_k: int) -> list[str]:
    scored = [(chunk, simple_similarity(query, chunk)) for chunk in chunks]
    scored.sort(key=lambda x: x[1], reverse=True)
    return [chunk for chunk, score in scored[:top_k] if score > 0]


class RagApplication(ChatCompletion):
    def __init__(self):
        self.document_chunks: dict[str, list[str]] = {}

    async def chat_completion(
        self, request: Request, response: Response
    ) -> None:
        last_message = request.messages[-1]
        user_text = last_message.content or ""

        attachment_content = await self._extract_attachment(request)

        if attachment_content:
            chunks = split_into_chunks(attachment_content, CHUNK_SIZE)
            self.document_chunks[request.jwt or "default"] = chunks
            with response.create_single_choice() as choice:
                choice.append_content(
                    f"Document indexed. Split into {len(chunks)} chunks. "
                    f"Ask me a question about it."
                )
            return

        chunks = self.document_chunks.get(request.jwt or "default", [])
        if not chunks:
            with response.create_single_choice() as choice:
                choice.append_content(
                    "No document uploaded yet. Please attach a text file "
                    "and send it to index its content."
                )
            return

        relevant_chunks = retrieve(user_text, chunks, TOP_K)
        context = "\n\n---\n\n".join(relevant_chunks)

        answer = await self._ask_model(context, user_text)
        with response.create_single_choice() as choice:
            choice.append_content(answer)


    async def _ask_model(
        self, context: str, user_text: str
    ) -> str:
        """Call a language model through DIAL Core."""
        url = (
            f"{DIAL_URL}/openai/deployments/"
            f"{MODEL_DEPLOYMENT}/chat/completions"
        )
        
        headers = {
            "Content-Type": "application/json",
            }

        payload = {
            "model": OLLAMA_MODEL,
            "messages": [
                {
                    "role": "system",
                    "content": (
                        "You are a helpful assistant. Answer the user's "
                        "question based ONLY on the following context. If the "
                        "answer is not in the context, say you don't know.\n\n"
                        f"Context:\n{context}"
                    ),
                },
                {"role": "user", "content": user_text},
            ],
        }

        async with httpx.AsyncClient() as client:
            resp = await client.post(
                url, headers=headers, json=payload, timeout=120.0
            )
            resp.raise_for_status()
            data = resp.json()
            return data["choices"][0]["message"]["content"]

    async def _extract_attachment(self, request: Request) -> Optional[str]:
        last_message = request.messages[-1]
        if not last_message.custom_content:
            return None
        attachments = last_message.custom_content.attachments
        if not attachments:
            return None

        attachment = attachments[0]
        if attachment.data:
            return attachment.data
        if attachment.url:
            return await self._download_file(request, attachment.url)
        return None

    async def _download_file(
        self, request: Request, url: str
    ) -> Optional[str]:
        import httpx

        if not url.startswith(("http://", "https://")):
            url = f"{DIAL_URL}/v1{url}" if url.startswith("/") else f"{DIAL_URL}/v1/{url}"

        headers = {}
        if request.jwt:
            headers["api-key"] = request.jwt
        async with httpx.AsyncClient() as client:
            resp = await client.get(url, headers=headers)
            if resp.status_code == 200:
                return resp.text
        return None


app = DIALApp(dial_url=DIAL_URL, propagate_auth_headers=True)
app.add_chat_completion("rag-app", RagApplication())

To run locally with uvicorn:

uvicorn app:app --host 0.0.0.0 --port 5000

Key concepts in this code:

  • File attachments arrive in custom_content.attachments on the last message. Each attachment has either inline data or a url pointing to DIAL file storage.
  • Streaming responses use the OpenAI SDK's stream=True parameter. Each chunk's content is appended to the response via choice.append_content().
  • Per-request key propagation (propagate_auth_headers=True) ensures the upstream LLM call is authorized automatically.
  • Simple keyword-based retrieval is used here to keep the tutorial focused on the DIAL integration pattern. In production, you would use vector embeddings (e.g., FAISS, Chroma) for semantic retrieval.

Step 4: Add dependencies​

Create rag-app/requirements.txt:

aidial-sdk[httpx]==0.27.0
uvicorn==0.34.0
httpx==0.28.1

Step 5: Create the Dockerfile​

Create rag-app/Dockerfile:

FROM python:3.11-alpine

WORKDIR /app
COPY . /app
RUN pip install --no-cache-dir -r requirements.txt

EXPOSE 5000
CMD ["python", "app.py"]

Step 6: Configure DIAL Core​

Create core/config.json:

{
  "models": {
    "chat-model": {
      "type": "chat",
      "displayName": "Local LLM (Ollama)",
      "endpoint": "http://ollama:11434/v1/chat/completions"
    }
  },
  "applications": {
    "rag-app": {
      "displayName": "RAG Document Q&A",
      "description": "Upload a text file, then ask questions about its content",
      "endpoint": "http://rag-app:5000/openai/deployments/rag-app/chat/completions",
      "inputAttachmentTypes": ["text/plain"],
      "maxInputAttachments": 1
    }
  },
  "keys": {
    "dial_api_key": {
      "project": "TEST-PROJECT",
      "role": "default"
    }
  },
  "roles": {
    "default": {
      "limits": {
        "chat-model": {},
        "rag-app": {}
      }
    }
  }
}

The inputAttachmentTypes field tells DIAL Chat to show the file upload button for this application and restrict uploads to plain text files. maxInputAttachments limits it to one file per message.

Step 7: Write the Docker Compose file​

Create docker-compose.yml:

services:
  rag-app:
    build: rag-app
    depends_on:
      - core
      - ollama
    environment:
      DIAL_URL: "http://core:8080"
      OLLAMA_MODEL: "llama3.2:1b"
      MODEL_DEPLOYMENT: "chat-model"
      CHUNK_SIZE: "200"
      TOP_K: "3"

  ollama:
    image: ollama/ollama:0.3.10
    ports:
      - "11434:11434"
    volumes:
      - ./.ollama:/root/.ollama

  # ========= DIAL =========
  core:
    image: epam/ai-dial-core:0.25.1
    platform: linux/amd64
    environment:
      aidial.config.files: '["/opt/config/config.json"]'
      aidial.storage.overrides: '{ "jclouds.filesystem.basedir": "data" }'
      aidial.redis.singleServerConfig.address: "redis://redis:6379"
      aidial.identityProviders.test.rolePath: 'roles'
      aidial.identityProviders.test.disableJwtVerification: 'true'
      aidial.encryption.secret: 'salt'
      aidial.encryption.key: 'password'
    ports:
      - "8080:8080"
    volumes:
      - ./core:/opt/config
    depends_on:
      - redis

  themes:
    image: epam/ai-dial-chat-themes:0.10.0
    platform: linux/amd64

  chat:
    image: epam/ai-dial-chat:0.26.0
    platform: linux/amd64
    ports:
      - "3000:3000"
    depends_on:
      - themes
      - core
    environment:
      NEXTAUTH_SECRET: "secret"
      THEMES_CONFIG_HOST: "http://themes:8080"
      DIAL_API_HOST: "http://core:8080"
      DIAL_API_KEY: "dial_api_key"
      ENABLED_FEATURES: "conversations-section,prompts-section,top-settings,top-clear-conversation,top-chat-info,top-chat-model-settings,empty-chat-settings,header,footer,request-api-key,report-an-issue,likes,conversations-sharing,input-files,attachments-manager,custom-applications,marketplace"

  redis:
    image: redis:7.2.4-alpine3.19
    command: >
      redis-server
      --maxmemory 256mb
      --maxmemory-policy volatile-lfu
      --save ""
      --appendonly no

Step 8: Start the stack and pull a model​

docker compose up --build -d

Wait for containers to start, then pull a model into Ollama:

docker compose exec ollama ollama pull llama3.2:1b

Verify: Confirm Ollama responds:

docker compose exec ollama ollama run llama3.2:1b "Say hello"

Step 9: Test in DIAL Chat​

  1. Open http://localhost:3000.
  2. Open the Catalog and select RAG Document Q&A.
  3. Click the attachment icon (paperclip) and upload the sample.txt file you created in Step 2.
  4. Send the file. The app responds with "Document indexed. Split into X chunks."
  5. Ask a question: What is DIAL Core?

Verify: The app responds with an answer grounded in the uploaded document, such as: "DIAL Core is the central routing component that handles authentication, rate limiting, and request forwarding."

Step 10: Test via the API​

Upload the document and ask a question in two requests:

curl http://localhost:8080/openai/deployments/rag-app/chat/completions \
  -H "Content-Type: application/json" \
  -H "api-key: dial_api_key" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Index this document",
        "custom_content": {
          "attachments": [
            {
              "type": "text/plain",
              "title": "sample.txt",
              "data": "DIAL is an open-source AI orchestration platform. DIAL Core handles routing. DIAL Chat is the UI."
            }
          ]
        }
      }
    ]
  }'

Then ask a question:

curl http://localhost:8080/openai/deployments/rag-app/chat/completions \
  -H "Content-Type: application/json" \
  -H "api-key: dial_api_key" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Index this document",
        "custom_content": {
          "attachments": [
            {
              "type": "text/plain",
              "title": "sample.txt",
              "data": "DIAL is an open-source AI orchestration platform. DIAL Core handles routing. DIAL Chat is the UI."
            }
          ]
        }
      },
      {
        "role": "assistant",
        "content": "Document indexed. Split into 1 chunks. Ask me a question about it."
      },
      {
        "role": "user",
        "content": "What does DIAL Core do?"
      }
    ]
  }'

Verify: The response contains an answer grounded in the document content.

Step 11: Stop the stack​

docker compose down

What you learned​

  • How to handle file attachments in a Custom App using custom_content.attachments.
  • How to stream responses from an upstream LLM back to the user via choice.append_content().
  • How to configure inputAttachmentTypes and maxInputAttachments in DIAL Core to enable file uploads for your app.
  • The RAG pattern: ingest → chunk → retrieve → generate, all within a single DIAL Custom App.
  • How per-request key propagation secures app-to-model communication automatically.

Production considerations​

This tutorial uses keyword matching for retrieval to keep the focus on DIAL integration. For production RAG applications:

  • Use vector embeddings (via an embedding model deployed in DIAL or an external service) for semantic retrieval.
  • Use a persistent vector store (FAISS, Chroma, Weaviate, Qdrant) instead of in-memory storage.
  • Implement document chunking strategies appropriate for your content type.

Further reading​