A model that reads and looks at the same time. Show it a photograph, a chart, a screenshot or a scanned form and ask questions about it. It is the capability behind document processing, accessibility descriptions and agents that can operate a screen.
Technical view
Combines a vision encoder, typically a vision transformer, with a language model, connected by a projection layer or cross-attention, trained on image-text pairs and then instruction-tuned on visual tasks. Images consume substantial context, scaling with resolution and tiling strategy, which is the main cost driver. Strong on description and open-ended questions, weaker on precise spatial reasoning, counting, and reading dense small text.
Related
Part of AI Terminology Index, section: Modalities and Applications.