A million-token model is not created by changing one context-length setting. Qwen2.5-1M combines progressive long-context training, positional extrapolation and an inference system that avoids performing every attention operation densely.1
Highlights
- Train to 256K, extrapolate beyond it. Length extension and Dual Chunk Attention play different roles; the reported million-token capability is not a million-token training curriculum.23
- Sparse prefill must respect chunking. Attention patterns are selected from the current chunk’s available queries, not from future queries at the end of the entire prompt.4
- Sparse attention does not eliminate memory. Chunking reduces temporary activation pressure, while the model still retains a long KV history.5
This is a useful systems study: model adaptation, attention approximation, cache size and kernel/backend support must agree before a headline context length becomes practical.
Study scope: report version 1, the 7B Instruct-1M artifact and its documented custom vLLM inference path. Matched configuration and sparse settings inspected; the ordinary Transformers Qwen2 path was inspected for the compatibility boundary. No model execution or performance measurement here. Evidence checked October 10, 2026.
Model in numbers
| Dimension | Value and scope |
|---|---|
| Released variants | 7B and 14B Instruct-1M; selected 7B card reports 7.61B parameters.6 |
| Backbone | Selected 7B: 28 layers, width 3,584, 28 query heads and four KV heads of width 128.7 |
| Training length | Progressive extension to 262,144 tokens; total additional training-token exposure is undisclosed.2 |
| Artifact context | Maximum 1,010,000; DCA chunk size 262,144, local window 8,192 and RoPE base 10,000,000.7 |
| Serving memory | Model-card guidance recommends at least 120 GB aggregate VRAM for a 1M-token 7B request in its documented framework. This is a recommendation, not our measured minimum.6 |
| Compute | Full training FLOPs, accelerator hours and exact hardware allocation are undisclosed.1 |
Architecture: the decoder is familiar
The dense causal decoder retains the Qwen2.5 7B architecture: grouped-query attention, RoPE, pre-RMSNorm residual blocks, SwiGLU FFNs and an untied output head. The main changes concern long-context learning and attention/position handling, not a new vision encoder or expert network. Inputs are text; document retrieval and other tools are outside the decoder.71
The ordinary Transformers 4.57.1 Qwen2 attention implementation does not implement the artifact’s custom DCA behavior. Reading the same weights with a larger maximum-length field is not evidence that the custom positional and sparse-attention path is active. The model card explicitly documents a Qwen vLLM branch for ultra-long inference.86
Dual Chunk Attention: change distance, not history
RoPE normally rotates queries and keys according to their positions, so an attention pair depends on their relative distance. Distances far beyond training can become unfamiliar. DCA remaps the positional relationships to ranges the model has learned while preserving useful local continuity.3
Figure 1. DCA remaps positional distances without dropping the past. Original report Figure 2.9
In Figure 1, intra-chunk attention follows ordinary local positions. Successive-chunk treatment preserves a short neighborhood across the chunk boundary, avoiding an artificial huge jump between adjacent tokens. Distant chunks use bounded/repeated positional patterns. The selected artifact supplies a 262,144-token chunk and an 8,192-token local window.37
Two different historical keys can receive similar remapped positional relationships. DCA is therefore an extrapolation compromise, not exact preservation of all million-token absolute distances. It also does not compress the KV cache: a remapped key still occupies a retained history position.3
Sparse attention and chunked prefill
MInference’s Vertical-Slash patterns combine important key columns with diagonal bands. A small set of query rows estimates which positions matter, and offline profiling assigns head-specific pattern budgets. Dense scoring for all query/key pairs is replaced by selected patterns during the expensive prefill.4
The naive selection procedure examines queries near the end of the complete input. A chunked runtime cannot use those future queries while processing an earlier chunk. Qwen2.5-1M instead selects patterns using the last 64 queries of each 32K prefill chunk. This is a systems constraint on information availability, not just a smaller batch.4
Figure 2. Each prefill chunk selects from queries it can already see. Original report Figure 4.10
Figure 2 explains why the original pattern selector must be adapted. DCA also changes apparent positional relationships. The selection step uses continuous relative positions so distant chunks are not confused by repeated DCA patterns; the final attention computation still uses DCA positions. Selector coordinates and model attention coordinates are different contracts.4
Calibrate retained attention mass
Some heads lose too much useful mass under fixed sparse budgets at very long lengths. The report calibrates those budgets using a million-token sample. For the same query and scores, the retained-mass ratio can be written as:4
Here LSE is log-sum-exp over selected or all valid keys. This quantity measures retained unnormalized attention mass, not answer accuracy. Raising a head’s budget can improve approximation while increasing compute. Good mass retention still requires downstream evaluation.
Training and alignment
The long-context continuation inherits earlier 4K and 32K stages, then extends through 65,536, 131,072 and 262,144. The RoPE base increases to 5M and then 10M in the later stages. Long-extension data uses a 75% maximum-length and 25% shorter mixture, rather than applying a single all-long exposure.2
Natural long material includes web, books, arXiv and code repositories. Synthetic tasks include fill-in-the-middle, keyword or position retrieval and reordering. These provide different dependencies rather than only asking the model to find one inserted sentence.2
SFT first preserves general behavior on short data and then mixes short and 262K examples. Synthetic long-document responses use segment reading and Qwen-Agent/RAG-assisted generation. Offline DPO uses shorter preference pairs, up to 8,192 tokens. The report does not describe an online GRPO stage for this specific extension, so the general Qwen2.5 recipe should not be copied over wholesale.2
Memory and the serving contract
For one million input tokens, batch one and unquantized BF16 KV, a local cache derivation is:7
This excludes model weights, output tokens, temporary buffers and backend overhead. It explains why a 7B model can need far more memory than its weights alone suggest. Chunked prefill reduces temporary activations; it does not remove those 53.4 GiB of idealized KV payload. Quantization or alternative cache formats change the assumption, not the arithmetic.5
The documented framework uses sparse attention, DCA and length extrapolation together. Its historical requirements include Ampere/Hopper GPUs, CUDA 12.1 or 12.3 and a custom vLLM branch. These are inspected release instructions, not a guarantee of compatibility with every current vLLM version. Backend changes should be tested rather than inferred from a successful model load.6
Evaluation and speed: keep conditions attached
The report’s 7B Instruct-1M scores improve RULER at 128K from ordinary Qwen2.5 with DCA+YaRN at 55.1 to 84.4. LV-Eval at 256K improves from 36.9 to 42.7 against the ordinary model with DCA. LongBench-Chat increases from 7.32 to 8.08. These tests cover different skills and are not all million-token measurements.11
Short-task changes are mixed: MMLU-Pro goes from 56.3 to 54.3, MATH from 75.5 to 72.9, while HumanEval rises from 84.8 to 86.0. The extension does not prove zero short-context regression.11
The authors report 3.2–6.7× time-to-first-token speedups at 1M across their tested configurations, using batch one on H20/A100 systems, 7B tensor-parallel four and 14B tensor-parallel eight. This is a system result, not a portable architectural constant. A cited 14B/H20 latency cannot be presented as the selected 7B checkpoint’s latency. Some production optimizations involve BladeLLM; the open custom path should not be assumed to include every measured optimization.5
Limits and lessons
Needle-in-a-haystack retrieval is a useful stress test but does not establish reliable synthesis over an entire million-token document. The report shows substantial degradation for unrefined MInference beyond roughly 400K; calibration helps, but sparse approximation remains a tradeoff. Complete training exposure, compute and production-backend internals remain undisclosed.41
The durable lesson is that trained length, positional support, sparse approximation, cache memory and runtime kernels are separate dimensions. The dense Qwen2.5 study provides the baseline; this extension shows how a familiar decoder becomes a long-context system.
References
-
Qwen Team. Qwen2.5-1M Technical Report, version 1, submitted January 26, 2025. The study uses this exact edition. ↩ ↩2 ↩3 ↩4
-
Qwen Team. Report, §§3–4: progressive pretraining, SFT and DPO. ↩ ↩2 ↩3 ↩4 ↩5
-
Qwen Team. Report, §5.1 and Figure 2: Dual Chunk Attention and positional extrapolation. ↩ ↩2 ↩3 ↩4
-
Qwen Team. Report, §5.2, Figures 3–6: MInference, chunked selection, continuous selector positions and head-budget refinement. ↩ ↩2 ↩3 ↩4 ↩5 ↩6
-
Qwen Team. Report, §§5.2–5.3 and §6.3: chunking, deployment system and speed experiments. ↩ ↩2 ↩3
-
Qwen Team. Selected model card: custom branch, requirements and memory guidance. Selected weights are Apache-2.0 licensed; this is not a report-artwork grant. ↩ ↩2 ↩3 ↩4
-
Qwen Team. Selected configuration and matched sparse-attention configuration, inspected October 10, 2026. ↩ ↩2 ↩3 ↩4 ↩5
-
Hugging Face. Qwen2 implementation, Transformers 4.57.1. Read inspection establishes that this generic attention path does not implement the custom DCA configuration; no runtime test performed. ↩
-
Qwen Team. Report version 1, Figure 2, PDF p. 5. Original-figure provenance and reuse assessment. ↩
-
Qwen Team. Report version 1, Figure 4, PDF p. 7. Original copyright retained; reproduction is a bounded analytical excerpt, not relicensed artwork. Asset notice. ↩
-
Qwen Team. Report, §6, Tables 4–6 and long-context evaluation plots. Results and prompt/length conditions are author-reported. ↩ ↩2

