Skip to content

Qwen3-VL

A checkpoint-specific study of adaptive vision, DeepStack feature injection, interleaved positions and staged multimodal post-training.

Updated View as Markdown

Qwen3-VL asks a useful question: can a language model receive richer visual evidence without making the input sequence longer? DeepStack answers by adding visual features at multiple decoder depths, rather than serializing each feature level as more tokens.12

Its distinguishing changes are:

  • Depth as an interface. Intermediate vision features supplement the main visual input at the first three language layers.
  • Better positional allocation. Interleaved M-RoPE distributes spatial and temporal rotary components across frequencies, rather than separate bands.
  • Explicit video time. Timestamp text precedes each temporal visual group, giving the model a readable time signal.
  • A broader training recipe. Long context pretraining, distillation and task specific rewards accompany the architecture changes.1

The public dense and expert families make this an instructive system to study, not just a leaderboard entry. This page selects the dense Qwen3-VL-8B-Instruct checkpoint and pinned Transformers implementation. It does not substitute the Thinking variant, a larger MoE model, or an unverified runtime trace. Sources checked October 10, 2026.32

The model in numbers

Quantity Scope and meaning
Dense 8B family Qwen3 language backbone plus a visual encoder and mergers. The family label is not an independently measured total parameter count.
Vision initialization: SigLIP2-SO-400M Reported backbone for this family size; adapted for dynamic resolution. The smaller 2B and 4B variants use another vision backbone.
Decoder: 36 layers, width 4,096 Selected artifact, with 32 query heads and eight key and value heads.
Context: 262,144 positions Selected configuration; shared by text, visual tokens, timestamps and formatting.
Patches: 16 × 16 pixels; 2 × 2 spatial merge Distinct from Qwen2-VL’s 14 pixel patches. Temporal patches still combine two frames.
Default image budget: 64 to 16,384 merged visual tokens Derived from this artifact’s pixel bounds and 32 pixel merged stride.
Pretraining stages: 67B, about 1T, about 1T, 100B tokens Stage budgets, not unique examples or a disclosed per checkpoint compute measurement.
Standard SFT data: about 1.2M samples One third text only, two thirds image or video paired with text. This is not the count of all pretraining or RL data.
Training FLOPs and checkpoint accelerator hours Not disclosed in the inspected report.

Report level scale and checkpoint configuration are kept separate here. Neither the report’s infrastructure reaching 10,000 GPUs nor a model card context extension claim supplies an 8B hardware cost or deployment guarantee. 134

One input path, additional depth connections

The main path dynamically resizes and normalizes images, groups pixels into patches, encodes them, and compresses each 2 × 2 feature group. The resulting vectors replace visual placeholders beside text. DeepStack has a second path: three intermediate encoder levels have dedicated mergers, whose outputs are added to the corresponding visual positions after the first three decoder layers.12

Pan on smaller screensOpen full size ↗

Figure 1. Qwen3-VL’s multimodal framework. Reproduced from Qwen Team, report Figure 1.5

The vision encoder also adapts its geometry: 2D rotary positions and interpolated learned absolute positions let a pretrained SigLIP2 encoder handle variable patch grids. These vision-side positions are distinct from the language decoder’s interleaved M-RoPE. In the pinned implementation, vision attention is full spatial attention within each temporal tube, not causal attention over the complete video; the language decoder integrates the ordered groups. Dynamic resolution is not a fixed 448-pixel resize, and it does not mean an unlimited pixel budget.124

One visual input, several integration depths
Image or sampled frames → vision encoder
Main input path
  1. Final visual features
  2. 2 × 2 feature merger
  3. Visual embeddings + text embeddings
  4. Language decoder stream
DeepStack side path
  1. Features from three encoder levels
  2. Three dedicated mergers
  3. Aligned visual features at language width
  4. Add at early decoder depths
Layer 1 output+ level A features
Layer 2 output+ level B features
Layer 3 output+ level C features

Additions affect visual positions, not every text position. A, B and C label three feature levels, not checkpoint layer indices. Output additions are checked in Transformers v4.57.1; the rest is a report level schematic.

Figure 2. Visual input plus early depth connections. Original schematic, with residual additions checked in the pinned implementation.12

Read Figure 1 from the inputs upward: different images produce different sequence lengths, video groups carry timestamp text, and DeepStack has a separate route into decoder depth. Its token counts illustrate the report’s example inputs, not a fixed budget for the selected 8B checkpoint. Figure 2 isolates the residual addition contract that the family overview leaves implicit.12

Figure 2’s side path is addition, not concatenation. The selected visual encoder has 27 blocks at width 1,152. Zero based block indices 8, 16 and 24 produce the three supplementary features. Their dedicated mergers reduce spatial length fourfold and project to width 4,096. The implementation adds these vectors only where the visual position mask is true. Text positions do not receive direct additions, and the stream continues through subsequent decoder blocks.32

The final feature merger normalizes before grouping, while the dedicated DeepStack mergers use normalization after grouping. All use an MLP with a GELU activation. This detail matters when reproducing the interface: identical shapes do not imply identical learned transformations.2

A concrete visual token calculation

A 512 × 768 pixel image is already divisible by the processor’s 32 pixel factor and falls within the selected pixel bounds. It yields a 32 × 48 patch grid, or 1,536 visual encoder features, and 384 main visual tokens after merging. Image boundary markers add two more positions, before text and chat formatting. A still image is repeated to fill the two frame temporal kernel. 34

Nmain=TgHgWg4,Hg=H/16,Wg=W/16(1)N_{\mathrm{main}}=T_g\frac{H_gW_g}{4},\qquad H_g=H/16,\quad W_g=W/16 \tag{1}

For this image, the side path supplies three additional arrays, each with 384 feature vectors. It does not create 1,152 extra context positions. It does add projections, feature storage and residual operations. A shorter input than feature concatenation is not a claim of zero overhead.2

For a video with four sampled frames at 512 × 512 pixels, two temporal tubes produce 512 merged visual tokens. The processor also inserts two timestamp strings and two pairs of boundary markers. Their text token count depends on the tokenizer and timestamps, so 512 is not the whole sequence length. This is an architecture derivation, not a processor execution or benchmark. 4

Positions and timestamps are different signals

Qwen2-VL groups temporal, height and width rotary channels into separate frequency bands. Qwen3-VL interleaves these components, so the axes are less segregated by rotary frequency. The selected configuration uses sections 24, 20 and 20 within each half head, with an interleaving flag and a rotary base of 5,000,000.132

Video time now appears as ordinary text, such as a formatted seconds string, before each temporal group. In the inspected code, the video grid is split into those groups; the group’s local temporal coordinate is zero, while its spatial coordinates and sequence offset still matter. Timestamps carry elapsed time; rotary positions still locate features in the mixed sequence. These are not interchangeable signals.124

Pan on smaller screensOpen full size ↗

Figure 3. Position channels and timestamp text solve different problems. Original schematic; the 24/20/20 allocation is checkpoint specific.132

The code uses height frequencies at slots 1, 4, …, 58, width at 2, 5, …, 59, and temporal frequencies elsewhere. Thus the first 60 slots follow a T/H/W pattern and the last four are temporal. “Balanced” describes spreading axes across the spectrum, not assigning exactly one third of channels to each. For ordinary text, all three coordinate streams are equal, so the axis choice does not introduce three different text positions.2

Correct metadata matters. The pinned processor calculates timestamps from sampled frame indices and source frame rate; if frame rate is missing, it warns and defaults to 24. Pre sampled frames without accurate metadata can therefore acquire misleading timestamps. Its video processor defaults to two samples per second, with a frame count bound of 4 to 768, subject to available frames. Video resizing uses a total spatiotemporal pixel budget, not an independent maximum for every frame.4

Training: architecture and curriculum together

Pretraining stage Updated components and sequence budget
S0: alignment, 67B tokens Merger layers only; vision and language backbones frozen; 8,192 positions.
S1: multimodal pretraining, about 1T All components; 8,192 positions; visual and text only data.
S2: long context, about 1T All components; 32,768 positions; more video, agent and long text content.
S3: long context adaptation, 100B All components; 262,144 positions; long videos and documents emphasized.

The report describes a move from per sample loss to a square root normalized per token loss to balance text and multimodal contributions. It does not give a complete operational formula or all optimizer settings in the inspected passage, so this page does not invent a weighting implementation.1

Data engineering is equally important. The authors recaption web images, augment sparse visual clusters, combine synthetic and pseudo labeled document parsing, and synthesize grounded STEM examples. Long document data preserve page order and include questions requiring multiple pages and layout elements. These procedures shape what the model can learn; architectural changes alone do not explain the final model’s gains.1

Document and agent data: the output contract matters

Document supervision is not generic captioning. A layout model establishes regions and reading order; Qwen2.5-VL-72B recognizes those regions; the outputs are assembled into position-aware parsing targets. Synthetic HTML provides precise labels, while real-document pseudo-labels are quality filtered. QwenVL-HTML localizes individual elements; QwenVL-Markdown localizes images and tables and represents tables in LaTeX. Long-document training combines page images and parsing targets in order, plus questions that require evidence across pages and layout types.6

Grounding uses coordinates normalized to 0–1000, rather than raw source pixels. Recovering pixel coordinates requires the relevant image dimensions; that convention must not be confused with rotary position IDs. The 3D corpus uses a unified virtual-camera frame and structured 9-DoF boxes, not direct physical measurements produced by the model.6

General function-call trajectories can contain synthesized tool responses: the report explicitly says the functions need not be executable. GUI trajectories and online-search data are separate sources. Valid-looking synthetic calls are therefore not proof of success in a real tool environment.6

Post-training and its feedback

Standard Instruct SFT and Thinking cold start are different branches. The Thinking data use an approximately 1:1 vision-language to text-only ratio; the standard SFT set uses a different 2:1 ratio. The reported three-stage post-training recipe is a family-level account, not evidence that every checkpoint used an identical teacher or rollout budget.7

The family separates Instruct and Thinking outputs through their SFT data. Standard SFT starts with one epoch at 32K context, followed by a second epoch mixing 256K and 32K inputs. Thinking cold start data include long reasoning traces; some visual mathematics examples are removed if a text only model can solve them without seeing the image.1

Distillation then uses text only data to adapt the language backbone. Its off policy phase learns teacher generated answers; its on policy phase learns on student generated sequences with KL divergence between teacher and student token distributions, computed from their logits. This is not simply another name for SFT on fixed teacher answers.1

The reported reasoning RL uses SAPO, task specific rule or executable verification, approximately 30K queries, and 16 candidate responses for difficulty filtering. Filtering first removes queries with no successful flagship-model response, then excludes queries above a 90% pass rate for the model being trained. These are different difficulty checks. The report uses task format prompts rather than a separate format reward for this reasoning phase, plus a response-language penalty.7 General RL combines rule based and model based rewards, including Qwen2.5-VL-72B-Instruct or Qwen3 judges. The family also describes tool integrated training with answer, intermediate reasoning and tool usage rewards. Exact data and optimization details for every selected checkpoint remain incomplete; the family recipe is not a fully reproducible 8B training run.1

Why rewards need behavioral checks

The authors observed agents reducing tool use to one call to exploit answer and reasoning rewards. They added a reward relative to an offline estimated appropriate tool count. This is a concrete example of a reward proxy shaping behavior, not evidence that the resulting policy always uses tools safely or optimally. Likewise, judge feedback can assess open responses but inherit the judge’s biases and failure modes.1

Inference and evaluation boundaries

The selected Instruct artifact generates text, not a Thinking artifact whose reasoning mode can simply be switched on. Use its chat template, visual processor and timestamps together. Cache backed generation follows the mixed prefill; the supplementary visual features are injected at visual positions, not newly recomputed for each ordinary generated text token.32

Evidence Selected result and condition
8B Instruct, report Table 4 MMMU 69.6; DocVQA test 96.1; Video-MME without subtitles 71.4. Different benchmark metrics and inputs, not a pooled accuracy.
8B Thinking, same table MMMU 74.1, but DocVQA 95.3. Reasoning output does not improve every task and is a separate checkpoint.
DeepStack ablation, Table 12 Average 74.7 to 76.0; InfoVQA 71.9 to 74.2; DocVQA 89.5 to 91.1; TextVQA 80.6 to 80.5.

The DeepStack ablation uses an internal 15B-A2B model pretrained on 200B tokens, with no post-training, evaluated on validation sets. It isolates a design in that experiment; it does not measure the released 8B model’s gain. 1

The vision-encoder comparison is also separate: report Table 11 pairs each encoder with the same 1.7B language model and 1.5T training tokens. For example, RLWDQA rises from 58.7 to 66.1, while some zero-shot image benchmarks fall. Neither this experiment nor the DeepStack ablation establishes the isolated contribution of interleaved positions or timestamps in the released 8B checkpoint.8

Long video needle retrieval uses 235B-A22B-Instruct, one frame per second and dynamically controlled frame resolution. Its reported 1M context result uses YaRN extension beyond native training context. A salient inserted frame retrieval task is not proof of complete understanding of two hours of video, and cannot be transferred to this 8B checkpoint.1

Limitations and reusable lessons

The checkpoint card and report identify Apache 2.0 release terms, supported by the official repository license. That grant does not provide the unreleased data mixtures, all filtering models, or a license for every source document. No weights or benchmarks were executed in this study.39

The core lesson is keep token length, representation depth and feedback quality separate. DeepStack enriches existing visual positions. Interleaved positions and timestamps improve the representation of geometry and time. Training data and rewards teach how to use those signals. Larger context still costs memory and prefill work, and reliable applications still need evidence checks and task specific evaluation.

Continue with InternVL 3 for a contrasting tiling, mixed preference and multimodal position recipe, or revisit Transformer foundations for residual connections, rotary positions and grouped query attention.

References

  1. Qwen Team. Qwen3-VL Technical Report, 2025, sections 2 to 5. Architecture and training inspected in HTML; evaluation Tables 4 and 12 and appendix conditions checked in the versioned PDF. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18

  2. Hugging Face contributors. Pinned Qwen3-VL model implementation, Transformers 4.57.1. Vision mergers, feature extraction, residual injection, interleaved RoPE, placeholder checks, coordinates and cache path inspected; no model execution. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14

  3. Qwen Team. Qwen3-VL-8B-Instruct artifact, configuration, processor settings and card inspected October 10, 2026. Selected model has 16 pixel patches, 27 vision blocks and DeepStack indices 8, 16, 24. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8

  4. Hugging Face contributors. Qwen3-VL multimodal processor, video processor and image processor, pinned to 8cb5963. Artifact pixel bounds override class defaults. ↩ ↩2 ↩3 ↩4 ↩5 ↩6

  5. Qwen Team. Qwen3-VL Technical Report, arXiv v1, Figure 1, p. 3. Artwork retained; surrounding prose and caption omitted. Original copyright retained; source and reuse record. ↩

  6. Qwen Team. Qwen3-VL data construction, report sections 3.2.3–3.2.9, pp. 6–9. Document formats, normalized coordinates, 3D labels and synthesized function responses checked. ↩ ↩2 ↩3

  7. Qwen Team. Qwen3-VL post-training, sections 4.2–4.5, pp. 10–13. Distinct SFT mixtures, difficulty filters, reward construction and tool training checked. ↩ ↩2

  8. Qwen Team. Qwen3-VL ablations, Tables 11–12, p. 24. Encoder comparison and internal 15B-A2B DeepStack experiment are not released-8B causal attribution. ↩

  9. Qwen Team. Official Apache 2.0 license, inspected October 10, 2026; selected card and report independently identify the checkpoint release terms. ↩

Navigation

Type to search…

↑↓ navigate↵ selectEsc close