The first Gemma generation is a useful baseline for understanding what sits behind a familiar parameter label. It combines a conventional causal decoder with a large vocabulary, a gated feed-forward path and size-specific attention choices. Its practical lesson is that parameter count, attention width and cached state are different budgets.12
Highlights
- Two sizes, different attention. Original 2B uses one KV head; 7B uses sixteen, matching its sixteen query heads. Do not transfer the small model’s cache saving to the larger one.2
- A model name is not an exact weight count. The report separately counts embeddings and decoder parameters. Together they exceed the rounded 2B and 7B labels.1
- An instruct checkpoint is a distinct training product. Pretraining, supervised instruction tuning and human-preference optimization have different targets and formatting contracts.31
Scope: original text-only 7B pretrained architecture, with 2B as a concrete attention contrast. Report v4 also includes revised Gemma 1.1 preference results; those are not silently assigned to the original instruction checkpoint. Report, official configuration and targeted Transformers v4.57.1 code inspected October 10, 2026; no weights executed. The gated checkpoint configuration was not accessed.1324
Model in numbers
Representative checkpoint: google/gemma-7b, pretrained.
| Dimension | Value and scope |
|---|---|
| Weight count | 7.751B non-embedding plus 0.787B tied embedding parameters; 8.538B total, derived by addition from report Table 21 |
| Pretraining exposure | 6T tokens; primarily English text, mathematics and code; not a released corpus1 |
| Training hardware | 4,096 TPUv5e chips for 7B; duration and total FLOPs undisclosed1 |
| Context | 8,192 trained tokens; not a claim of useful extrapolation1 |
| Concrete decoder | 28 layers; residual width 3,072; sixteen query and KV heads of width 256; 256,128 vocabulary entries2 |
| Access and terms | Gated public weights under Gemma Terms; reference inference software is separately Apache 2.0352 |
Architecture: follow the dimensions, not the family name
Text becomes token IDs, then scaled embeddings. Each layer reads earlier positions through causal attention and transforms each position through a GELU-gated FFN. Residual additions keep a shared 3,072-wide stream. A final normalization and tied vocabulary head predict the next token.4
Figure 1. Original architecture trace from the inspected reference configuration and implementation. The inset separates query heads from stored KV heads.24
In Figure 1, the attention projection is 4,096 wide, not 3,072: sixteen heads times 256. The output projection maps it back to the residual width. Many introductions assume that head count times head width must equal model width; this released design is a counterexample. That equality is a design convention, not a Transformer law.24
The inspected RMSNorm evaluates the normalization in float32 and applies a one-centered learned scale before casting back. Input embeddings are multiplied by the square root of the residual width. These details matter when translating weights or implementing a numerical comparison; a generic pre-norm decoder is not automatically weight-compatible.4
Read the gated FFN width carefully
The FFN has separate gate and value projections, multiplies their outputs, and projects back. A useful expression is:4
Here x is a residual vector and both expanded branches have width 24,576 in Google’s 7B reference configuration. The product is elementwise; it is not attention over other positions. The gate can amplify, suppress or change the sign of intermediate features before the down projection mixes them.24
The report’s Table 1 lists a feed-forward hidden dimension of 49,152, twice the reference branch width. For this code trace, 24,576 means one branch. Doubling it when constructing both gate and up matrices would incorrectly inflate the model. The paired widths explain the numerical correspondence, but the report does not explicitly define that counting convention; the official configuration establishes the executable branch size.12
MQA changes state, not the number of questions
Original 2B has eight query heads but only one KV head. Each query head can ask a different question of the same stored key/value representations. In 7B, all sixteen query heads have distinct KV heads. For an unquantized cache, a basic storage estimate is:24
B is batch size, L layers, T cached positions, H_KV KV heads, d_h head width and b bytes per element. The factor two counts keys and values. This counts state only, not weights, activations or allocator overhead.
For B = 1, T = 8,192 and two-byte elements, the derived 7B cache is 3.5 GiB. Original 2B’s eighteen layers and one KV head give 144 MiB under the same assumptions. These are arithmetic storage budgets, not measured resident memory or latency. Model-weight quantization alone does not guarantee either cache is quantized.2
Training: language, dialogue, preferences
The public model card identifies web text, code and mathematics as the data domains. It describes filtering and benchmark decontamination rather than releasing a complete data manifest. A corpus description is not a reproducible dataset.3
| Stage | What the target teaches | Disclosure boundary |
|---|---|---|
| Pretraining | Next-token language prediction on the text mixture | Token budgets and infrastructure disclosed; exact mixture and full optimizer schedule not released1 |
| Supervised tuning | Responses to synthetic and human-created English prompts | Filtering, attribution and refusal goals described; complete example set absent1 |
| Human-preference optimization | Prefer outputs judged helpful and safe | Report names a Bradley-Terry reward model and an unspecified novel RL algorithm; do not relabel it PPO or GRPO1 |
The distinction is useful for reproduction. Matching the decoder gives a weight-compatible forward path. Matching the assistant additionally requires the tokenizer, dialogue formatting, data and alignment process. An architecture diagram cannot substitute for those missing ingredients.
Inference: the prompt is part of the contract
The base checkpoint predicts text continuations. For an instruction-tuned checkpoint, use its tokenizer’s conversation template, including role and turn delimiters; the original report warns that an unformatted prompt is out of distribution for that training.13
Prefill processes the prompt and stores each layer’s keys and values. Decode adds new token states and advances positions without recomputing the whole prefix. RoPE is applied to queries and keys; the cached history and causal mask must agree about positions. The selected implementation uses the cache object’s update path, while generation and sampling sit outside the core decoder.4
A model can therefore load successfully yet behave badly if the application changes its template, stop tokens or padding convention. This is an interface failure, not evidence that the architecture is incapable of conversation. No serving throughput was measured here.
Evidence: compare tasks, not a composite headline
Figure 2. Grouped benchmark averages, not an architecture ablation. Reproduced from Gemma Team, report v4 Figure 1, p. 2; figure crop, CC BY 4.0.1
Read Figure 2 by category. Averaging different benchmarks hides prompting and metric differences; it cannot identify whether vocabulary, FFN width or training data caused a gain. The report’s Table 6 also mixes rerun and previously published baselines, and its later preference table describes Gemma 1.1. Those boundaries matter more than a single “better than 7B peers” summary.1
The model card likewise cautions about bias, inaccurate answers and task-dependent performance. A favorable benchmark does not remove the need to test the exact checkpoint in the intended application.3
Limitations and lessons
Training data and alignment are not fully reproducible from public artifacts. The gated checkpoint metadata was not inspected, so public reference configuration and runtime code are identified separately. Report v4 contains more than one instruction-release generation. The model is text-only; multimodal capabilities in later Gemma generations are not inherited backwards.13
Keep three counts separate: rounded family label, total weights and per-request KV state. Trace actual dimensions: attention width need not equal residual width. Treat the template as a learned interface: assistant formatting is not decorative.
Continue with Gemma 2 for distillation and hybrid attention, and Gemma 3 for a compact visual interface. Shared attention and FFN intuition is in Transformer Foundations.
References
-
Gemma Team, Google DeepMind. Gemma: Open Models Based on Gemini Research and Technology, 2024, report v4. Architecture Tables 1–2, training and instruction tuning, evaluation Figure 1 and Tables 5–7. Paper artwork CC BY 4.0; copyright Google DeepMind. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16
-
Google. Gemma PyTorch reference configuration, inspected October 10, 2026. GemmaConfig, get_config_for_7b and get_config_for_2b. Apache 2.0 software, not the model-weight license. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11
-
Google. Gemma 7B model card, original pretrained checkpoint; inputs, training, evaluation and gated access inspected October 10, 2026. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7
-
Hugging Face. Gemma implementation, Transformers v4.57.1. GemmaRMSNorm, GemmaMLP, GemmaAttention, GemmaDecoderLayer and GemmaModel; forward and cache paths inspected, not executed. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9
-
Google. Gemma Terms of Use, model and redistribution terms inspected October 10, 2026. These apply separately from software and paper-artwork terms. ↩
