Attaching an image does not tell us how a model uses it. Follow the evidence: what the encoder preserves, how many representations reach the language side, and where they enter the computation. A vision encoder, connector, and training recipe jointly define that interface.1234
Four interfaces, four different jobs
CLIP · match representations
Separate image and text encoders produce comparable representations. CLIP alone is not a generative language decoder.
BLIP-2 · query the image
Learned queries read vision features. Projected query outputs become visual prompts during generative pretraining.
LLaVA · project the features
A connector maps image features into the language embedding space. Original LLaVA uses a linear projection; 1.5 uses an MLP.
Qwen3-VL · add depth routes
Merged visual tokens enter the input; selected intermediate vision features also enter early decoder layers through DeepStack.
The number of drawn cells is illustrative. Training and frozen-backbone choices depend on the model and stage.
Figure 1. Matching images and text is different from conditioning a language decoder. Independent comparison of the named interfaces; feature counts are illustrative.1234
CLIP learns comparable image and text representations with contrastive supervision. It can provide a visual backbone or retrieval representation; CLIP alone does not generate a language response.1
BLIP-2 learns queries that read frozen vision features. In its generative stage, projected query outputs condition a frozen language model.2
LLaVA maps vision features directly into language embedding space. The original uses a linear projection; LLaVA-1.5 uses an MLP connector.35
Qwen3-VL adds another route: merged input tokens plus intermediate vision features injected at early decoder depths through DeepStack.4 That changes where evidence enters, not merely how many input tokens are added.
The interfaces in Figure 1 are useful comparisons, not a sequence in which each interface makes the previous one obsolete. Training data, task, resolution and budget remain part of the result.
A query bottleneck makes the budget explicit
Figure 2. Learned queries between frozen backbones. Reproduced from Li et al., ICML 2023 Figure 3; unchanged figure crop, CC BY 4.0.2
The snowflakes in Figure 2 mark frozen weights. The upper branch conditions a causal decoder; the lower branch shows an encoder and decoder variant. The same connector idea can therefore cross different language model contracts. The paper uses 32 learned queries: the language input need not include every encoder feature.2
A bottleneck controls computation but can omit detail the question requires. That is a design concern to test, not a claim that fewer tokens always cause lower accuracy.
Count patches, merged tokens, and views separately
For image height H, width W and square patch side P:
The first count assumes divisible dimensions with no padding or crops. The second assumes every r × r patch neighborhood becomes one output token. It is a local geometry calculation, not a complete processor specification. Actual models may add class tokens, delimiters, crops, frame grouping or different resamplers.67
A 4 × 4 patch grid
Sixteen patch features merged in 2 × 2 neighborhoods become four language-side visual tokens.
Twice the height and width
An 8 × 8 grid has 64 patches. The same merging rule gives 16 visual tokens: four times the original count.
Toy geometry at a fixed patch size, without crops, padding, class tokens or delimiters. This is not a latency measurement.
Figure 3. Twice the height and width produces four times the visual budget at the same merge ratio. Original geometry example, not a timing result.67
Figure 3 isolates the geometry. Qwen2-VL merges neighboring visual features; Gemma 3 uses a compact fixed token budget per encoded view, with extra crops when needed. More views can therefore increase the aggregate budget even when the count per view is fixed.78 For video, sampling changes the number and coverage of frames before position encoding begins. See Position and long context.
Architecture and training must agree
| Question | Why it changes the interpretation |
|---|---|
| What is frozen, and at which stage? | A small trained interface and joint backbone training have different adaptation budgets |
| What is supervised? | Matching, captioning, instruction following and grounded coordinates teach different behavior |
| What is compressed or omitted? | Fine print, small objects or brief events may disappear before language computation |
| Where do features enter? | Input projection, cross attention and depth additions create different routes |
This is a comparison framework. The InternVL 3, GLM vision, and Gemma 3 studies provide additional training and processor contrasts. Do not transfer one checkpoint’s frozen stages, token count or resolution rules to an entire family.
Check whether the answer depends on the evidence
Compare correct, absent, swapped and contradictory images. Include small text, chart values, spatial relations, and questions requiring several regions. Control resolution, view count, data and training budget when testing an interface.
These are proposed tests, not measurements here. Selected mechanisms rechecked October 10, 2026. Continue with Qwen3-VL for original framework artwork and a concrete DeepStack implementation trace.
References
-
Alec Radford et al. Learning Transferable Visual Models From Natural Language Supervision, 2021, section 2. ↩ ↩2 ↩3
-
Junnan Li, Dongxu Li, Silvio Savarese and Steven Hoi. BLIP-2, ICML 2023, sections 3.1 to 3.3 and Figure 3. Figure source and crop are retained in the asset record. ↩ ↩2 ↩3 ↩4 ↩5
-
Haotian Liu et al. Visual Instruction Tuning, 2023, sections 4.1 and 4.2. ↩ ↩2 ↩3
-
Qwen Team. Qwen3-VL Technical Report, 2025, section 2: merger and DeepStack. ↩ ↩2 ↩3
-
Haotian Liu et al. Improved Baselines with Visual Instruction Tuning, 2023, connector and training changes. ↩
-
Alexey Dosovitskiy et al. An Image is Worth 16x16 Words, 2021 revision, section 3.1. ↩ ↩2
-
Qwen Team. Qwen2-VL, 2024, visual feature merging and dynamic resolution. Equation (1) and Figure 3 are local geometry examples. ↩ ↩2 ↩3
-
Gemma Team. Gemma 3 Technical Report, 2025, visual encoder, 256 tokens per view and pan-and-scan. ↩
