---
title: "Multimodal Integration"
description: "Trace visual representations, token budgets and integration depth from pixels to language computation."
---

> Documentation Index
> Fetch the complete documentation index at: https://latentlit.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Multimodal Integration

import FocusCanvas from '../../../../components/FocusCanvas.astro';
import ArchitectureAtlas from '../../../../components/architectures/ArchitectureAtlas.astro';
import SourceFigure from '../../../../components/architectures/SourceFigure.astro';

Attaching an image does not tell us how a model uses it. Follow the evidence:
**what the encoder preserves, how many representations reach the language side,
and where they enter the computation**. A vision encoder, connector, and training
recipe jointly define that interface.[^clip][^blip][^llava][^qwen]

## Four interfaces, four different jobs

<a id="figure-1"></a>

**Four interfaces with different jobs**

**CLIP · match representations**: Separate image and text encoders produce comparable representations. CLIP alone is not a generative language decoder.

**BLIP-2 · query the image**: Learned queries read vision features. Projected query outputs become visual prompts during generative pretraining.

**LLaVA · project the features**: A connector maps image features into the language embedding space. Original LLaVA uses a linear projection; 1.5 uses an MLP.

**Qwen3-VL · add depth routes**: Merged visual tokens enter the input; selected intermediate vision features also enter early decoder layers through DeepStack.

The number of drawn cells is illustrative. Training and frozen-backbone choices depend on the model and stage.

**Figure 1.** Matching images and text is different from conditioning a language decoder. Independent comparison of the named interfaces; feature counts are illustrative.[^clip][^blip][^llava][^qwen]

**CLIP** learns comparable image and text representations with contrastive
supervision. It can provide a visual backbone or retrieval representation;
CLIP alone does not generate a language response.[^clip]

**BLIP-2** learns queries that read frozen vision features. In its generative
stage, projected query outputs condition a frozen language model.[^blip]

**LLaVA** maps vision features directly into language embedding space.
The original uses a linear projection; LLaVA-1.5 uses an MLP connector.[^llava][^llava15]

**Qwen3-VL** adds another route: merged input tokens plus intermediate vision
features injected at early decoder depths through DeepStack.[^qwen]
That changes where evidence enters, not merely how many input tokens are added.

The interfaces in [Figure 1](#figure-1) are useful comparisons, not a sequence in which each interface makes
the previous one obsolete. Training data, task, resolution and budget remain
part of the result.

## A query bottleneck makes the budget explicit

<a id="figure-2"></a>

![BLIP-2 learns an interface between frozen visual and language backbones.](/figures/model-studies/blip-2/generative-bridge.webp)

**Figure 2.** Learned queries between frozen backbones. Reproduced from Li et al., ICML 2023 Figure 3; unchanged figure crop, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).[^blip]

**Image description:** BLIP-2 uses a Q-Former and projection to condition a frozen causal decoder or frozen encoder-decoder language model from frozen image features.

The snowflakes in [Figure 2](#figure-2) mark frozen weights. The upper branch
conditions a causal decoder; the lower branch shows an encoder and decoder
variant. The same connector idea can therefore cross different language
model contracts. The paper uses 32 learned queries: the language input need
not include every encoder feature.[^blip]

A bottleneck controls computation but can omit detail the question requires.
That is a design concern to test, not a claim that fewer tokens always cause
lower accuracy.

## Count patches, merged tokens, and views separately

For image height H, width W and square patch side P:

$$
N_{\mathrm{patch}}=\frac{H}{P}\frac{W}{P},\qquad N_{\mathrm{merged}}=\frac{N_{\mathrm{patch}}}{r^2}
\tag{1}
$$

The first count assumes divisible dimensions with no padding or crops. The
second assumes every r × r patch neighborhood becomes one output token.
It is a local geometry calculation, not a complete processor specification.
Actual models may add class tokens, delimiters, crops, frame grouping or
different resamplers.[^vit][^qwen2]

<a id="figure-3"></a>

**Resolution grows a budget before merging reduces it**

**A 4 × 4 patch grid**: Sixteen patch features merged in 2 × 2 neighborhoods become four language-side visual tokens.

**Twice the height and width**: An 8 × 8 grid has 64 patches. The same merging rule gives 16 visual tokens: four times the original count.

Toy geometry at a fixed patch size, without crops, padding, class tokens or delimiters. This is not a latency measurement.

**Figure 3.** Twice the height and width produces four times the visual budget at the same merge ratio. Original geometry example, not a timing result.[^vit][^qwen2]

[Figure 3](#figure-3) isolates the geometry. Qwen2-VL merges neighboring visual features; Gemma 3 uses a compact fixed
token budget per encoded view, with extra crops when needed. More views can
therefore increase the aggregate budget even when the count per view is fixed.[^qwen2][^gemma]
For video, sampling changes the number and coverage of frames before
position encoding begins. See [Position and long context](/architectures/evolution/position-and-long-context/#images-and-video-need-coordinates-not-just-indices).

## Architecture and training must agree

| Question | Why it changes the interpretation |
| --- | --- |
| What is frozen, and at which stage? | A small trained interface and joint backbone training have different adaptation budgets |
| What is supervised? | Matching, captioning, instruction following and grounded coordinates teach different behavior |
| What is compressed or omitted? | Fine print, small objects or brief events may disappear before language computation |
| Where do features enter? | Input projection, cross attention and depth additions create different routes |

This is a comparison framework. The [InternVL 3](/architectures/models/internvl-3/),
[GLM vision](/architectures/models/glm-4-v/), and
[Gemma 3](/architectures/models/gemma-3/) studies provide additional training
and processor contrasts. Do not transfer one checkpoint's frozen stages,
token count or resolution rules to an entire family.

## Check whether the answer depends on the evidence

Compare correct, absent, swapped and contradictory images. Include small
text, chart values, spatial relations, and questions requiring several regions.
Control resolution, view count, data and training budget when testing an interface.

These are proposed tests, not measurements here. Selected mechanisms
rechecked October 10, 2026. Continue with [Qwen3-VL](/architectures/models/qwen3-vl/)
for original framework artwork and a concrete DeepStack implementation trace.

[^clip]: Alec Radford et al. [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/html/2103.00020v1), 2021, section 2.
[^blip]: Junnan Li, Dongxu Li, Silvio Savarese and Steven Hoi. [BLIP-2](https://arxiv.org/html/2301.12597v3), ICML 2023, sections 3.1 to 3.3 and Figure 3. Figure source and crop are retained in the [asset record](/figures/model-studies/blip-2/SOURCES.md).
[^llava]: Haotian Liu et al. [Visual Instruction Tuning](https://arxiv.org/html/2304.08485v2), 2023, sections 4.1 and 4.2.
[^llava15]: Haotian Liu et al. [Improved Baselines with Visual Instruction Tuning](https://arxiv.org/abs/2310.03744v2), 2023, connector and training changes.
[^qwen]: Qwen Team. [Qwen3-VL Technical Report](https://arxiv.org/html/2511.21631v1), 2025, section 2: merger and DeepStack.
[^vit]: Alexey Dosovitskiy et al. [An Image is Worth 16x16 Words](https://arxiv.org/html/2010.11929v2), 2021 revision, section 3.1.
[^qwen2]: Qwen Team. [Qwen2-VL](https://arxiv.org/html/2409.12191v1), 2024, visual feature merging and dynamic resolution. Equation (1) and Figure 3 are local geometry examples.
[^gemma]: Gemma Team. [Gemma 3 Technical Report](https://arxiv.org/html/2503.19786v1), 2025, visual encoder, 256 tokens per view and pan-and-scan.

Source: https://latentlit.com/architectures/evolution/multimodal-integration/index.mdx
