Multimodality refers to an AI system's ability to jointly process and integrate information from multiple data types (modalities) — such as text, image, audio and video. Unlike unimodal models, which are limited to a single data type, a multimodal model can understand different inputs in combination and produce outputs in various formats.
Technically, the different modalities are usually mapped into a shared representation space (embeddings), so the model can relate them to one another — for example describing an image, generating an image from text or answering a question about a chart. A model can thus, for instance, take a photo of a plate of cookies and output a matching recipe.
Multimodality substantially broadens the scope of generative AI — from analyzing medical images and document understanding to autonomous systems that combine camera, sensor and text data. Current state-of-the-art models are increasingly designed to be natively multimodal.