Multimodal AI

In short

Multimodal AI is artificial intelligence that can take in, and sometimes produce, more than one kind of data, such as text, images, audio and video. You can show it a photo and ask a question about it in words.

Also called: Multimodal model, Multimodal machine learning, Multimodal LLM, MLLM
Topic: The basics
Level: Beginner
First described: One of the earliest examples of multimodal research was audio-visual speech recognition, combining sound with lip movements, in work by Yuhas, Goldstein and Sejnowski in 1989¹.

A "modality" is a type of information: written or spoken language, pictures and video, or sounds. Multimodal machine learning builds models that can process and relate information from several of these at once¹.

OpenAI's 2023 GPT-4 report described a "large-scale, multimodal model which can accept image and text inputs and produce text outputs"². Google's Gemini models, announced the same year, were built to understand image, audio, video and text³.

One approach is to turn every input into the same currency the model already uses for text. Anthropic's documentation says Claude views images in 28 by 28 pixel patches, each counted as a visual token⁵. Another key idea, from OpenAI's 2021 CLIP paper, is to learn from pictures paired with their captions: CLIP was trained on 400 million image and text pairs collected from the internet⁴.

Taking in several kinds of data does not mean producing them. Claude, for example, can analyse images but cannot generate or edit them⁵.

Multimodal models also make mistakes. Anthropic warns that Claude may hallucinate when reading low-quality, rotated or very small images, and gives only approximate counts of many small objects⁵.

Example

You photograph a confusing parking sign, upload it to a chatbot and ask "Can I park here at 6pm on a Saturday?" The model reads the sign from the image and answers in text.

Why it matters

It is why AI assistants can now read a screenshot, describe a photo or listen to a voice note, not just reply to typed messages. It also means images and recordings you share are processed by the model too.

Questions people ask

Is ChatGPT multimodal?

Yes. Its underlying model became multimodal with GPT-4, which OpenAI described in 2023 as accepting image and text inputs and producing text outputs².

What is the difference between multimodal AI and a large language model?

A large language model works with text. A multimodal model also handles other types of data such as images, audio or video¹. Many of today's leading chatbot models, such as Gemini, are both³.

Can multimodal AI understand video?

Some models can. Google's Gemini family was designed for image, audio, video and text understanding³. Claude accepts images, including several in one request⁵.

Related terms

Tools that use it

Sources

Back to the glossary