Skip to main content

Multimodality

Multimodality is the ability to work with more than one kind of content — text, images, audio, and video — in a single conversation or application. This page explains what multimodality means in DIAL, how the platform delivers it through models, files, and applications, and how multimodal capabilities combine into richer solutions. It is written for developers who want to understand the model before building with it.

What multimodality means in DIAL​

Most real tasks are not text-only. A user might upload a chart and ask for an explanation, request a generated image, transcribe a recording, or produce a short video from a prompt. DIAL supports these by connecting to models that handle different media types and by giving every application a consistent way to receive and return files.

Two platform features make this work together:

  • The Unified API carries non-text content as attachments on a message. A user or an application can pass files in as input and receive files as output.
  • File storage holds those files in a per-user bucket governed by the same access control as every other resource, so multimodal inputs and outputs respect permissions automatically.

Because the protocol and storage are shared, a model that returns an image and an application that consumes one interoperate without any special integration.

Media travels in both directions as attachments. The adapter calls the provider's dedicated API for each modality and can read input files from, and upload generated files to, DIAL file storage through DIAL Core.

Working across modalities through models​

Connection to models is handled by adapters, which translate each provider's API into the Unified API. This lets an application target a capability — "generate an image," "describe this page" — rather than a specific vendor format. DIAL covers the common modality directions:

  • Text-to-text — the familiar chat and completion tasks.
  • Image understanding (image-to-text) — answering questions about an uploaded image or document page.
  • Image generation (text-to-image) — producing images from a prompt.
  • Audio — speech-to-text transcription and text-to-speech synthesis.
  • Video — text-to-video, image-to-video, and video understanding, where supported by the provider.

The exact models available for each direction change as providers release new ones. Supported providers is the authoritative, current list; treat it as the source of truth rather than memorizing model names.

Building multimodal applications​

Applications consume these capabilities the same way they consume any model. With the DIAL SDK you can build an application that accepts a file, calls a multimodal model, and returns text or another file — for example, an application that reads a scanned invoice and returns structured data. Because the application speaks the Unified API, it becomes a reusable agent like any other.

Orchestrating multiple modalities​

The most capable solutions combine several single-modality agents behind one entry point. A Quick App orchestrator can route each request to the right specialist: a retrieval agent for a factual question, a text-to-image application for an illustration request, an image-understanding model for an uploaded photo. The user interacts with one conversation while, underneath, the orchestrator selects the appropriate model or application per task.

This is multimodality and the agentic platform working together: shared file handling and a shared protocol let modality-specific building blocks compose into a single multimodal experience.

Further reading​