Architectural evolution is not a ladder of increasingly superior models. It is a set of choices about what information to preserve, where to move it, and how much computation to spend. A useful comparison names the operation that changes and the budget it affects.
A map of the design space
Across tokens
Attention exchanges evidence between positions. Cache and connectivity determine what can be read.
Inside a token
Feature transformations and expert routing allocate computation while returning to the residual width.
Across depth
Residual paths determine how earlier representations reach later computation.
From pixels
An encoder and interface turn visual evidence into representations usable by the language model.
Coordinates affect matching; training teaches these routes what to carry. This map abstracts over the exact model stack.
Figure 1. Information across tokens, features, depth and modalities. Original synthesis of the cited mechanisms, not a literal stack shared by every model.1234
Figure 1 provides four views of information flow. Position changes the geometry of matching; training teaches the routes what to carry. An encoder such as BERT, an encoder and decoder such as T5, and a causal language decoder have different visibility and output contracts. Do not silently assign the decoder schematic to every model.56
The original GLM adds another instructive contract: observed context is bidirectional, while missing spans are generated autoregressively. Its mask, target order and positional representation must be read together. A family name is not proof that later checkpoints retain that exact computation.7
Follow a mechanism into a model
| Design question | Mechanisms to compare | Instructive studies |
|---|---|---|
| What can be read and retained? | Exact execution, KV sharing, latent compression, selected reads, recurrence | Llama 3.1, DeepSeek-V2, Mamba, GLM-5 |
| What does position mean? | RoPE, context extension, local and global reach, visual coordinates | Mistral 7B, Qwen2.5 1M, Qwen2.5-VL |
| Where does computation go? | Gated features, selected experts, shared experts, balancing | Mixtral, DeepSeek-V3, Qwen3 |
| How does depth carry information? | Residual bypasses, multiple streams, depth attention, gated reads and writes | GPT-2, DeepSeek-V4, Kimi K3, Qwen3.8-Next |
| How does visual evidence enter? | Contrastive encoders, queries, projection, feature merging, depth injection | CLIP, BLIP-2, LLaVA, Qwen3-VL |
| What would make a conclusion convincing? | Matched budgets, ablations, negative results, implementation checks | Gemma 3, MiniMax M3, GLM-4.5 |
This is an editorial reading map. The linked studies define exact model versions and evidence boundaries; the table is not a claim that every family uses the same mechanism.
Separate four kinds of change
| Layer of explanation | Example | Do not confuse it with |
|---|---|---|
| Mathematical computation | GQA shares KV heads; MLA compresses KV content.82 | A faster kernel for an unchanged operator |
| Learned behavior | Reasoning RL changes a policy on a pretrained backbone.9 | A new attention block |
| Execution | FlashAttention tiles exact attention and reduces memory traffic.10 | Removal of dense pairwise arithmetic |
| Application system | Tools, retrieval and context management around a model | Capabilities supplied by weights alone |
The last row is a system boundary used here, not a specific product claim. Model reports can change several rows at once. That makes a stronger final model possible, but makes attribution to one ingredient harder.
Read any change in five steps
- Name the problem and baseline.
- Locate the operation, route, or stored representation that changes.
- Identify what stays the same.
- Compare quality, active computation, retained state, total weights, and communication.
- Inspect the experiment and implementation before generalizing.
For the original computation and tensor shapes, return to Foundations. For chronology, use the release map. For testable next questions, use Emerging directions.
Selected mechanisms rechecked October 10, 2026. These pages synthesize the available studies; they do not establish a universal winning architecture.
References
-
Ashish Vaswani et al. Attention Is All You Need, 2017, section 3. ↩
-
DeepSeek-AI. DeepSeek-V2, 2024, section 2. ↩ ↩2
-
Qwen Team. Qwen3-VL Technical Report, 2025, section 2. ↩
-
Colin Raffel et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, 2020, architecture and objectives. ↩
-
Zhengxiao Du et al. GLM: General Language Model Pretraining with Autoregressive Blank Infilling, 2022 revision, sections 2.1 and 2.2. ↩
-
DeepSeek-AI. DeepSeek-R1, 2025, section 2. ↩
-
Tri Dao et al. FlashAttention, 2022, section 3. ↩