Skip to content

Position and Long Context

Separate matching geometry, trained context, visible evidence and useful recall.

Updated View as Markdown

Position tells a model where evidence belongs. A context limit tells you how much input a system accepts. Neither tells you how reliably the model uses that input. Keep representation, capacity, visibility, and usefulness separate when reading a long context claim.

Position changes matching geometry

The original Transformer adds sinusoidal position vectors to embeddings. RoPE instead rotates query and key feature pairs before their dot product. It exposes relative displacement through matching geometry rather than adding another vector to the residual stream.12

For one feature pair, with column vectors q and k:

R(a)=[cos⁡a−sin⁡asin⁡acos⁡a](1)R(a)=\begin{bmatrix}\cos a&-\sin a\\\sin a&\cos a\end{bmatrix} \tag{1}

At positions m and n, the rotated dot product becomes:

(R(mθ)q)⊤R(nθ)k=q⊤R((n−m)θ)k(2)\bigl(R(m\theta)q\bigr)^\top R(n\theta)k=q^\top R\bigl((n-m)\theta\bigr)k \tag{2}

θ is the frequency for this pair. Equation (2) follows from the orthogonality of rotation. Real heads contain many pairs with different frequencies; content still determines the score.2

Mechanism mapShift both positions, preserve their separation

Positions 1 and 2

Identical unit content vectors rotate to 45° and 90°. Their separation is 45°.

Positions 2 and 3

The same vectors rotate to 90° and 135°. Their separation remains 45°.

Toy frequency θ = π/4. This illustrates one feature pair, not measured attention weights or complete head geometry.

Figure 1. Absolute angles change; the relative offset stays the same. Original toy geometry for identical unit vectors, informed by RoFormer.2

Figure 1 does not mean a complete model is invariant to every input shift. Masks, boundaries, content, and other position handling remain part of its computation. Nor does a rotation formula establish reliable extrapolation.

Four questions behind a long context claim

Question What to inspect What it does not establish
Representation Position IDs, frequencies, scaling, image and time coordinates Reliable performance at an unseen length
Capacity Trained length, configured limit, backend, memory and processor Every accepted position is used well
Visibility Dense, local, compressed or selected history A long input means every layer directly reads all of it
Usefulness Retrieval, reasoning, ordering and failures across positions A single needle test proves every long document task

This is an interpretation framework used here, not a performance table. Compare it with Attention and memory, which explains how visibility and storage change.

Extend the learned positional regime

YaRN adjusts RoPE frequencies by their wavelength regimes and applies attention scaling. It pairs that adaptation with additional training, rather than establish a universal rule that changing one configuration value is enough.3 DeepSeek-V2 and Qwen2.5 1M show why trained length, extension technique, accepted length, and evaluation length must be recorded separately.

Qwen2.5-1M also uses Dual Chunk Attention: it remaps within-chunk, neighboring and distant positional relationships to ranges seen during training. This is not KV compression or simply dropping distant tokens. Remapping can make distinct long distances look similar; useful recall still needs evaluation.4

Let depth carry information beyond a local window

In Mistral’s sliding attention, a layer reads nearby positions, while representations can carry information farther through stacked layers.5 A theoretical path does not guarantee that the necessary detail survives it. Local visibility and successful distant recall are different claims.

Images and video need coordinates, not just indices

“Two dimensional” does not always mean image height and width. Original GLM uses learned coordinates for a missing span’s location and progress inside that span. It is a task-specific positional design, not multimodal RoPE.6

An image has height and width. A video adds time, and sampling rate changes which frames become visible. Qwen2.5-VL aligns temporal rotary positions with elapsed time; Qwen3-VL adds explicit timestamp text and interleaves rotary frequencies across temporal and spatial coordinates.78

Frame index is not elapsed time
Visual evidence path
  1. Sampled framesFPS and resolution choose what remains visible.
  2. Position and timeRepresent spatial coordinates and the time of each frame group.
  3. Language contextVisual tokens and any timestamp text share the model budget.

Removing a frame removes its direct evidence. Better coordinates cannot reconstruct an unseen event.

Figure 2. Sampling and time representation solve different problems. Original pipeline abstraction from the named Qwen reports.78

Figure 2 is a conceptual pipeline, not a claim that preprocessing alone guarantees temporal grounding. For resizing, patches, crops and merging, continue with Multimodal integration.

Test useful reach, not only accepted length

Vary evidence location, distractor count, and dependency distance independently. Include several facts that must be combined, conflicting evidence, and questions whose answer changes when the relevant evidence changes. For video, vary FPS and event timing separately. Report accuracy alongside state and latency.

These are proposed checks, not results measured here. Selected mechanisms rechecked October 10, 2026. Gemma 3 supplies useful local/global and position ablations; the model studies retain the conditions behind each published context claim.

References

  1. Ashish Vaswani et al. Attention Is All You Need, 2017, section 3.5. ↩

  2. Jianlin Su et al. RoFormer, 2023 revision of the 2021 preprint, sections 3.1 and 3.2. Equations (1) and (2) use equivalent local notation. ↩ ↩2 ↩3

  3. Bowen Peng et al. YaRN: Efficient Context Window Extension of Large Language Models, 2023, section 3 and training experiments. ↩

  4. Qwen Team. Qwen2.5-1M Technical Report, 2025, section 5.1: Dual Chunk Attention remaps positional relationships; the separate sparse runtime is described in section 5.2. ↩

  5. Albert Q. Jiang et al. Mistral 7B, 2023, section 2: sliding attention and reach through depth. ↩

  6. Zhengxiao Du et al. GLM, 2022 revision, section 2.2.1: two learned positional coordinates for blank infilling. ↩

  7. Qwen Team. Qwen2.5-VL Technical Report, 2025, sections 2.1.2 and 2.1.3. ↩ ↩2

  8. Qwen Team. Qwen3-VL Technical Report, 2025, sections 2.1 and 2.3. Timestamp text and Interleaved MRoPE are distinct mechanisms. ↩ ↩2

Navigation

Type to search…

↑↓ navigate↵ selectEsc close