Skip to content

Model Deep Dives

A living map of language and vision models, and a starting point for deeper study.

Updated View as Markdown

From the original Transformer to today’s language and vision models, this is a living map of the releases worth understanding. Start with the milestones, then choose a model to explore in depth.

The selection covers 56 releases and research milestones, checked through October 9, 2026. Selection favors established model families and influential research milestones. It is a reading map, not a complete catalog or a popularity ranking.

To compare ideas across releases, use Architectural Evolution. Its mechanism pages connect memory, position, expert routing, residual paths and visual interfaces back to these studies. Read a deep dive for the model’s full training recipe, implementation choices and evidence.

Major releases at a glance

Bold names mark selected landmarks in architecture, training, or adoption. The 22 highlights are an editorial reading guide, not a ranking.

Year Selected models and milestones
2017 Transformer1
2018 GPT2 · BERT3
2019 GPT-24 · T55
2020 GPT-36
2021 CLIP7 · Switch Transformer8 · GLM9
2022 InstructGPT10 · Chinchilla11 · PaLM12 · Flamingo13 · BLOOM14 · ChatGPT15
2023 BLIP-216 · LLaMA17 · GPT-418 · ChatGLM-6B19 · LLaVA20 · Llama 221 · Mistral 7B22 · Mamba23 · Gemini 1.024 · Mixtral 8x7B25 · InternVL26
2024 Gemini 1.5 Pro27 · Claude 328 · DeepSeek-V229 · GPT-4o30 · Llama 3.131 · Qwen2-VL32 · Qwen2.533 · OpenAI o134 · DeepSeek-V335
2025 DeepSeek-R135 · Gemma 336 · Gemini 2.5 Pro37 · Qwen338 · Claude 439 · Grok 440 · Kimi K241 · GLM-4.542 · GPT-543 · Qwen3-VL44
2026 Qwen3.545 · GLM-546 · Claude Mythos Preview47 · DeepSeek-V435 · Kimi K348 · Qwen3.845 · Claude Fable 5.149 · GPT-6 Astra50 · DeepSeek-V4.1-Flash51 · Claude Opus 5.552 · Gemini 4 Argon53

Dates identify the event in the release log: a paper, an announcement, a preview, or a weights release. They do not all mean general availability. Transformer is the architecture anchor; CLIP is a contrastive vision and text model; Mamba is a selective state space architecture, not a Transformer. ChatGPT marks the 2022 research preview.172315

OpenAI o1 records the full model’s December 5, 2024 release in ChatGPT, not the September 12 o1-preview release. Claude Mythos is represented by its April 7, 2026 restricted preview; it is not a general availability event.345447

Model evolution

Read both bands in Figure 1 from left to right. Year placement shows chronology, not lineage. The release log records exact dates and events.

Figure 1. Selected model milestones, 2017 to 2026. AI generated from the cited release list.

Release log: exact dates and event types
Date Model Event recorded
2017-06-12 Transformer1 Architecture paper
2018-06-11 GPT2 Announcement
2018-10-11 BERT3 Paper
2019-02-14 GPT-24 Staged release
2019-10-23 T55 Paper
2020-05-28 GPT-36 Paper
2021-01-05 CLIP7 Announcement
2021-01-11 Switch Transformer8 Paper
2021-03-18 GLM9 Paper
2022-01-27 InstructGPT10 Alignment announcement
2022-03-29 Chinchilla11 Paper
2022-04-05 PaLM12 Paper
2022-04-29 Flamingo13 Paper
2022-07-12 BLOOM14 Weights release
2022-11-30 ChatGPT15 Research preview
2023-01-30 BLIP-216 Paper
2023-02-27 LLaMA17 Paper
2023-03-14 GPT-418 Announcement
2023-03-14 ChatGLM-6B19 Weights release
2023-04-17 LLaVA20 Paper
2023-07-18 Llama 221 Family announcement
2023-09-27 Mistral 7B22 Weights release
2023-12-01 Mamba23 Paper
2023-12-06 Gemini 1.024 Family announcement
2023-12-11 Mixtral 8x7B25 Release article
2023-12-21 InternVL26 Paper
2024-02-15 Gemini 1.5 Pro27 Limited preview
2024-03-04 Claude 328 Family announcement
2024-05-07 DeepSeek-V229 Paper
2024-05-13 GPT-4o30 Announcement
2024-07-23 Llama 3.131 Weights release
2024-08-29 Qwen2-VL32 Announcement
2024-09-19 Qwen2.533 Family announcement
2024-12-05 OpenAI o134 ChatGPT release
2024-12-26 DeepSeek-V335 API release
2025-01-20 DeepSeek-R135 API release
2025-03-12 Gemma 336 Family announcement
2025-03-25 Gemini 2.5 Pro37 Experimental release
2025-04-29 Qwen338 Family announcement
2025-05-22 Claude 439 Family announcement
2025-07-09 Grok 440 Announcement
2025-07-11 Kimi K241 Weights release
2025-07-28 GLM-4.542 Announcement
2025-08-07 GPT-543 Announcement
2025-09-23 Qwen3-VL44 First family weights
2026-02-16 Qwen3.545 First family weights
2026-02-17 GLM-546 Paper
2026-04-07 Claude Mythos Preview47 Restricted preview
2026-04-24 DeepSeek-V435 API preview
2026-07-27 Kimi K348 Weights and report
2026-08-12 Qwen3.845 First family weights
2026-09-01 Claude Fable 5.149 Model release
2026-09-03 GPT-6 Astra50 Initial rollout
2026-09-10 DeepSeek-V4.1-Flash51 API release
2026-09-22 Claude Opus 5.552 Announcement
2026-09-30 Gemini 4 Argon53 Restricted rollout

Some families arrived in stages. For example, GPT-2 initially released smaller models, Gemini 2.5 Pro began as an experimental release, and Gemini 4 Argon initially rolled out to trusted cyber defenders.43753 Qwen2-VL uses the announcement’s August 29 date; its repository records August 30 for the initial smaller model release.3244

Choose a deep dive

Studies of models with public weights follow the Model Deep Dive guide. It connects technical reports to named artifacts and distinguishes reported findings from inspected implementation and measured results.

The first collection contains 41 studies, selected for distinct technical lessons rather than one page per checkpoint. Each starts with the original report and Hugging Face links, then highlights and scale. Architecture, training, inference and evaluation connect to the inspected sources. Original LLaMA is a historical study with restricted release terms; public weights elsewhere also have different licenses and access rules. Each page states its own verification boundary. These are source studies, not locally reproduced model benchmarks.

Browse related generations under GPT, Llama, Mistral, DeepSeek, Qwen, GLM, Kimi, Gemma and MiniMax in the sidebar. Single studies stay directly accessible. The reading paths below connect models by the ideas they teach, across family boundaries.

Language foundations

Study What to understand
GPT-2 Causal prediction, residual paths and generation state
BERT Bidirectional representations and masked prediction
T5 One text-to-text interface, encoder-decoder computation
LLaMA A practical dense-decoder recipe and its disclosure limits
Llama 3 / 3.1 Scaling the recipe, data and long-context extension
Mistral 7B Sliding-window attention and grouped-query memory
GLM / GLM-130B Autoregressive blank filling and stable bilingual scaling
GLM-4 Distinguishing a released 9B model from a wider API family
Qwen2.5 Data scaling, staged alignment and explicit generation scope
Gemma Compact dense models and a practical open weight recipe
Gemma 2 Distillation, attention patterns and training efficiency

Experts and reasoning

Study What to understand
Mixtral 8x7B Sparse expert routing and total versus active size
DeepSeek-V2 Latent attention and fine-grained experts
DeepSeek-V3 Load balancing, FP8 training and multi-token prediction
DeepSeek-R1 Reasoning RL, data curation and distillation
Qwen3 Thinking and direct-answer modes in one generation
GLM-5 Sparse history selection and asynchronous agent learning
Kimi K2 Large-scale experts, Muon and agent data
Kimi K3 Mixing across sequence, depth and expert width
DeepSeek-V4 Sequence compression and constrained residual mixing
DeepSeek-V4.1-Flash Causal encoding, shared caches and approximate replay
gpt-oss Quantized experts, mixed attention and the Harmony agent interface
GLM-4.5 Expert capacity, agent tasks and staged reasoning alignment
MiniMax M1 CISPO and numerical consistency in long reasoning RL
MiniMax M2 Series Full attention, verifiable environments and asynchronous agent RL

Memory architectures

Study What to understand
Mamba Input-dependent state updates instead of an attention cache
Kimi Linear Corrective recurrent memory alongside global lookup
Qwen2.5 1M Long context training and sparse serving versus dense checkpoints
Qwen3.8 Flash Next Sparse experts and recurrent attention in the released backbone
MiniMax 01 Lightning attention, exact history and the long context curriculum

Vision and language

Study What to understand
CLIP Contrastive alignment, not visual text generation
BLIP-2 Learned queries connecting frozen encoders and language models
LLaVA / 1.5 Visual instruction tuning and a simpler connector
Qwen2-VL Dynamic resolution and multimodal position
Qwen3-VL Deep visual injection, position and staged training
Gemma 3 Compact image views and local/global attention
InternVL 3 Native multimodal pretraining and a changing visual budget
Qwen2.5-VL Resolution, video time and document grounding
GLM-4V Shared vision training ideas across dense and expert backbones
Gemma 4 Native multimodal interfaces, model variants and prediction modules
MiniMax M3 / MSA Learned block selection and the native VLM release boundary

This is a curated technical collection, not a page for every checkpoint update. Qwen3 Next, Qwen3.5 and Gemma 3n have useful official technical documentation, but this expansion did not verify a matching standalone generation report. See Qwen3 Next, Qwen3.5 and Gemma 3n for those documentation boundaries. A backbone paper does not establish a later model’s entire vision or alignment recipe.

Pick a reading path

Use Transformer Foundations for the shared mechanisms and Architectural Evolution to compare design choices. This overview will grow as new releases add useful contrasts.

How we read a model

Follow the input from its processor through the model to the output. Compare the report with the released configuration and source code, then examine the training disclosure and evaluation conditions. Keep missing details visible: a model name does not establish a checkpoint, a complete recipe, or a universal performance ranking.

References

  1. Vaswani et al. Attention Is All You Need. 2017. ↩ ↩2 ↩3

  2. OpenAI. Improving language understanding with unsupervised learning. 2018. ↩ ↩2

  3. Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2018. ↩ ↩2

  4. OpenAI. Better language models and their implications. 2019. ↩ ↩2 ↩3

  5. Raffel et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. 2019. ↩ ↩2

  6. Brown et al. Language Models are Few-Shot Learners. 2020. ↩ ↩2

  7. OpenAI. CLIP: Connecting text and images. 2021. ↩ ↩2 ↩3

  8. Fedus et al. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. 2021. ↩ ↩2

  9. Du et al. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. 2021. ↩ ↩2

  10. OpenAI. Aligning language models to follow instructions. 2022. ↩ ↩2

  11. Hoffmann et al. Training Compute-Optimal Large Language Models. 2022. ↩ ↩2

  12. Chowdhery et al. PaLM: Scaling Language Modeling with Pathways. 2022. ↩ ↩2

  13. Alayrac et al. Flamingo: a Visual Language Model for Few-Shot Learning. 2022. ↩ ↩2

  14. BigScience / Hugging Face. Introducing The World’s Largest Open Multilingual Language Model: BLOOM. 2022. ↩ ↩2

  15. OpenAI. Introducing ChatGPT. 2022. ↩ ↩2 ↩3

  16. Li et al. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. 2023. ↩ ↩2

  17. Touvron et al. LLaMA: Open and Efficient Foundation Language Models. 2023. ↩ ↩2

  18. OpenAI. GPT-4. 2023. ↩ ↩2

  19. Team GLM. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. 2024, Section 1 and Table 1 record the 2023 release. ↩ ↩2

  20. Liu et al. Visual Instruction Tuning. 2023. ↩ ↩2

  21. Meta. Meta and Microsoft Introduce the Next Generation of Llama. 2023. ↩ ↩2

  22. Mistral AI. Mistral 7B. 2023. ↩ ↩2

  23. Gu and Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. 2023. ↩ ↩2 ↩3

  24. Google / Google DeepMind. Introducing Gemini: our largest and most capable AI model. 2023. ↩ ↩2

  25. Mistral AI. Mixtral of experts. 2023. ↩ ↩2

  26. Chen et al. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. 2023. ↩ ↩2

  27. Google / Google DeepMind. Our next-generation model: Gemini 1.5. 2024. ↩ ↩2

  28. Anthropic. Introducing the next generation of Claude. 2024. ↩ ↩2

  29. DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. 2024. ↩ ↩2

  30. OpenAI. Hello GPT-4o. 2024. ↩ ↩2

  31. Meta. Introducing Llama 3.1: Our most capable models to date. 2024. ↩ ↩2

  32. Qwen Team. Qwen2-VL: To See the World More Clearly. 2024. ↩ ↩2 ↩3

  33. Qwen Team. Qwen2.5: A Party of Foundation Models!. 2024. ↩ ↩2

  34. OpenAI. Introducing ChatGPT Pro. 2024. ↩ ↩2 ↩3

  35. DeepSeek. Change Log. Dated release notes, inspected October 9, 2026. ↩ ↩2 ↩3 ↩4 ↩5 ↩6

  36. Google DeepMind. Introducing Gemma 3: The most capable model you can run on a single GPU or TPU. 2025. ↩ ↩2

  37. Google DeepMind. Gemini 2.5: Our most intelligent AI model. 2025. ↩ ↩2 ↩3

  38. Qwen Team. Qwen3: Think Deeper, Act Faster. 2025. ↩ ↩2

  39. Anthropic. Introducing Claude 4. 2025. ↩ ↩2

  40. xAI. Grok 4. 2025. ↩ ↩2

  41. Moonshot AI. Kimi K2: Open Agentic Intelligence and dated research index. 2025. ↩ ↩2

  42. Z.ai. Z.ai Releases GLM-4.5. 2025. ↩ ↩2

  43. OpenAI. Introducing GPT-5. 2025. ↩ ↩2

  44. Qwen Team. Qwen3-VL repository. Dated release notes, inspected October 9, 2026. ↩ ↩2 ↩3

  45. Qwen Team. Qwen3.5, Qwen3.6 and Qwen3.8 release notes. Dated release notes, inspected October 9, 2026. ↩ ↩2 ↩3 ↩4

  46. GLM-5 Team. GLM-5: from Vibe Coding to Agentic Engineering. 2026. ↩ ↩2

  47. Anthropic. Project Glasswing. 2026, April 7 launch and early access description. ↩ ↩2 ↩3

  48. Moonshot AI. Kimi K3 Open Day: weights, technical report and infrastructure. 2026. ↩ ↩2

  49. Anthropic. Claude Fable 5.1 and Claude Fable and Mythos 5.1. 2026. ↩ ↩2

  50. OpenAI. Safety overview: GPT-6 Astra. 2026. ↩ ↩2

  51. DeepSeek. Introducing DeepSeek V4.1 Flash: smarter, faster, more efficient. 2026. ↩ ↩2

  52. Anthropic. Introducing Claude Opus 5.5. 2026. ↩ ↩2

  53. Google DeepMind. Gemini 4 Argon: our next era of frontier intelligence. 2026. ↩ ↩2 ↩3

  54. OpenAI. Introducing OpenAI o1-preview. 2024. ↩

Navigation

Type to search…

↑↓ navigate↵ selectEsc close