Skip to content

GPT-2

Understand task transfer, a released causal decoder, parameter counts, and generation cache ownership.

Updated View as Markdown

GPT-2 turns a text prefix into predictions for the next token. Its research question was more ambitious: could training on varied text also teach a model to perform tasks described in that text, without a separate supervised model for each task?1

Highlights

  • Tasks can live in the prefix. The report studies transfer through textual context without updating the model for each benchmark.1
  • A compact architectural baseline. One causal decoder combines learned positions, pre normalization, GELU FFNs, and a tied vocabulary head.2
  • Generation is more than the network. The released model produces logits and incremental KV tensors; the caller owns sampling and cache accumulation.23

For a learner, GPT-2 makes the boundary between representation, prediction, and generation unusually easy to inspect. This is its value in this collection, not a claim that a 2019 model is a competitive assistant today.

Study scope: the report, pinned small checkpoint configuration and tokenizer, and original model and sampling code were inspected. No checkpoint was run. Evidence checked October 10, 2026.

Model in numbers

Representative artifact: openai-community/gpt2, the released small base model.45

Dimension Value and scope
Parameters Approximately 124M in the released card; 124,439,808 from the architecture count below. The report labels its smallest model 117M.412
Decoder 12 layers, width 768, 12 attention heads, head width 64.5
Input Vocabulary of 50,257 tokens; 1,024 learned positions.5
Data Slightly over 8 million cleaned WebText documents, 40 GB of text. This is corpus size, not total token exposure.1
Training budget Total processed tokens, duration, and compute for this small checkpoint are not disclosed in the inspected report. Do not substitute the largest model’s hardware allocation.
Post training This artifact is a base language model, not an instruction or preference trained assistant.4

What changed

The central shift is the task interface. Instead of attaching a supervised head for every task, the report asks whether an ordinary continuation objective can learn useful input and output patterns from diverse text. A document followed by a question, or an article followed by a summary cue, changes what continuation is likely. This is evidence of task transfer, not proof of reliable instruction following.1

The architecture remains recognizably a Transformer decoder. The report moves LayerNorm before each residual branch, adds a final LayerNorm, and describes depth dependent residual initialization. Larger capacity and broader data are part of the experiment; the results do not isolate normalization as the cause of downstream improvements.1

Model identity

Item Studied source
Report Radford et al., 2019, Language Models are Unsupervised Multitask Learners, sections 2 to 41
Released artifact Small GPT-2, pinned card, configuration, and tokenizer metadata456
Code trace OpenAI repository, model.py and sample.py23
Openness Original repository code uses MIT; the hosted artifact card declares MIT. WebText is not released as a browsable training dataset.74

The parameter labels need care: the report’s Table 2 and the released card use different counts. The concrete dimensions here describe the pinned small artifact, not a silent substitution into every historical result.

Architecture

The input contract is a batch of token IDs. Token embeddings and learned position embeddings are added. Twelve residual blocks mix the permitted text history and transform each token’s features. Final normalization and the transposed token embedding matrix produce vocabulary logits.2

A language decoder and its residual block
  1. Token IDs
  2. Token + learned position embeddings
  3. Residual block × L
  4. Final LayerNorm
  5. Vocabulary logits · tied token embedding weights
  6. Softmax + next token selection outside the model
One GPT-2 block with two residual bypassesInput x passes through LayerNorm and causal multi-head attention; its output is added to the unnormalized input. The resulting u passes through LayerNorm and a GELU FFN; that output is added to u. Each bypass avoids both normalization and the branch computation.Input xLayerNormCausal attentionMHA + projection+xuLayerNormGELU FFND → 4D → D+uBlock output

Figure 1. Causal decoder and residual paths. Original schematic based on the pinned released code.2

The objective factorizes a sequence into next token conditionals:1

p(x1:T)=∏t=1Tp(xt∣x<t)(1)p(x_{1:T})=\prod_{t=1}^{T}p(x_t\mid x_{\lt t}) \tag{1}

For a block input x, attention and FFN branches read normalized features, but each residual addition retains the unnormalized stream:2

u=x+Attention⁡(LN⁡1(x))h=u+FFN⁡(LN⁡2(u))(2)\begin{aligned} u &= x+\operatorname{Attention}(\operatorname{LN}_1(x)) \\ h &= u+\operatorname{FFN}(\operatorname{LN}_2(u)) \end{aligned} \tag{2}

Causal attention and learned positions

Each head has width 768 / 12 = 64. The code forms queries, keys, and values, scales their dot products, masks disallowed future positions, and normalizes with softmax. During cached decoding, the mask is aligned to the end of the combined history, not naively to a square current token matrix.2

Learned position rows are indexed using the past length plus the current offset. The 1,024 row position table is an architectural boundary of this artifact; a larger API input allowance would not establish that these learned positions support longer context.52

Dense FFNs and a tied output head

The FFN expands 768 features to 3,072, applies the released GELU approximation, and projects back. Attention mixes positions; this FFN transforms the features at each position. Its two affine projections include biases. The final head reuses token embedding weights rather than storing a second vocabulary matrix.2

Derive the released small model’s parameter count

Let V = 50,257, C = 1,024, D = 768, and L = 12. Counting the embedding tables, attention and FFN weights and biases, two LayerNorms per block, and the final LayerNorm gives this local derivation from the inspected code:

P=VD+CD+L(12D2+13D)+2D=124,439,808(3)P=VD+CD+L(12D^2+13D)+2D=124{,}439{,}808 \tag{3}

The tied vocabulary head contributes no additional V × D matrix. This explains the released architecture count; it does not establish the reason for the paper’s earlier 117M label.251

Initialization: the report is not a training implementation

The report scales residual-layer initialization by the inverse square root of the number of residual layers. That is an explicit depth-aware training choice, separate from moving normalization. The released TensorFlow inference definition uses the same default 0.02 initializer for both residual projections; it restores learned checkpoint weights and does not document the original training initialization. Reinitializing that file is therefore not a faithful reproduction of the report’s training recipe.12

Training

WebText selected linked pages through Reddit voting, then extraction, deduplication, and cleaning. The reported corpus excluded Wikipedia and links created after December 2017. Removing one source does not guarantee the absence of benchmark overlap.1

Training predicts subsequent tokens using text itself as supervision. The report gives a batch size of 512 and says learning rates were selected against a 5% held out WebText split. It does not provide a complete training implementation, total token budget, or all optimizer settings for reproducing the released small model.1

There is no chat template, SFT, or RL stage to infer from the presence of a question and answer prompt. Those prompts change conditioning, not the checkpoint’s training history.14

Inference and cache ownership

The inspected tokenizer metadata uses byte level encoding and decoding with 50,257 vocabulary entries. Tokenization settings remain part of the interface: byte coverage does not mean that every language received comparable training.61

Byte-level BPE is not one token per byte. UTF-8 bytes supply a 256-symbol base, then learned merges compact frequent byte sequences. Before merging, the tokenizer separates character categories, with special treatment of spaces, to avoid spending many vocabulary entries on punctuation variants of the same word. This balances open-string coverage with shorter sequences; byte coverage alone says nothing about equal multilingual competence.1

The original model function returns the current call’s new keys and values. It constructs present before concatenating past for attention. In sample_sequence, the caller concatenates these new tensors onto the retained history, applies temperature and truncation to the last position’s logits, and selects the next token.23

The model reads history. The caller grows it.
One model call
Current tokensT positions
Decoder attentionReads past + current KV
Vocabulary logitsNot a selected token
Returned “present”Only this call's T new KV positions, across all layers
Caller: sample and retain state
Past KV · N positions
…
New KV · T positions
…
Retained KV · N + T
…
Next call: sampled token + retained KV as “past”

Illustrative position bars, not exact token counts. Sampling and cache concatenation are outside the original model function.

Figure 2. Sampling and cache growth belong to the caller. Original diagram from the released model and sampling functions.23

Figure 2 separates the network’s returned tensors from the state retained for the next call. The model uses the full history for attention, but returns only the new KV positions. The caller appends those positions and samples a token.

For batch B, current length T, and retained history N:

Tensor Small checkpoint shape
Current token stream B × T × 768
Vocabulary logits B × T × 50,257
New KV tensors B × 12 × 2 × 12 × T × 64
Accumulated KV history B × 12 × 2 × 12 × N × 64

These shapes follow the pinned code. Caching avoids recomputing past projections, but attention still reads retained keys and values. No latency or memory benchmark was performed here.23

Evaluation and limits of transfer

The report’s task results are not one universal score, and many headline results describe its largest model rather than this small artifact.

Question Author reported evidence and conditions
Does scale improve language modeling? Table 3 gives LAMBADA perplexity 35.13 for the paper’s 117M row and 8.63 for its 1,542M row. Tokenization and evaluation handling matter.1
Can a task cue alter behavior? Table 4 reports CNN/Daily Mail ROUGE average 21.40 with a summary cue versus 15.03 without it, using the largest model and the section 3.6 generation setup.1
Does fluent continuation imply a useful system? The report’s discussion describes summarization as rudimentary and notes uneven transfer across tasks. These are limitations of the same study, not a later replication.1

The scale comparison changes capacity, not just one architectural mechanism. The summary cue comparison is a more targeted prompt intervention, but neither establishes modern assistant reliability. Section 4 investigates training and test overlap; it does not certify every example as uncontaminated.1

The original model card warns about factual unreliability and biases inherited from text. Public weights and readable code are useful for study, but do not make the model appropriate for every deployment.8

Read the objective, network, and generation policy as three connected but different contracts. Check parameter definitions and cache ownership in the artifacts, rather than infer them from the model name.

Continue with Transformer Foundations for the shared attention and FFN mathematics, or attention and memory for the distinction between cache storage and execution cost.

Open questions

The inspected report and release do not explain the earlier 117M label or provide a complete small checkpoint training budget. A reproduction study would need additional training records, not an assumed modern recipe.

References

  1. Alec Radford et al., Language Models are Unsupervised Multitask Learners, 2019. Sections 2.1 to 2.3, 3, 4, and 6; Tables 2 to 4. Inspected October 10, 2026. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18 ↩19

  2. OpenAI, src/model.py. Functions default_hparams, attn, mlp, block, past_shape, positions_for, and model. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15

  3. OpenAI, src/sample.py, the same commit. Functions top_k_logits, top_p_logits, and sample_sequence; inspected, not executed. ↩ ↩2 ↩3 ↩4 ↩5

  4. OpenAI Community, small GPT-2 model card, the same release. Model description, training data, and MIT metadata; written by Hugging Face from OpenAI’s original material. ↩ ↩2 ↩3 ↩4 ↩5 ↩6

  5. OpenAI Community, small GPT-2 configuration. ↩ ↩2 ↩3 ↩4 ↩5 ↩6

  6. OpenAI Community, GPT-2 tokenizer metadata, the same release. ByteLevel pre tokenizer and decoder, vocabulary, and added tokens. ↩ ↩2

  7. OpenAI, GPT-2 repository license, MIT, copyright 2019 OpenAI. ↩

  8. OpenAI, GPT-2 model card, original pinned repository. Out of scope uses and bias limitations. All linked artifacts inspected October 10, 2026; no weight download or checkpoint execution. ↩

Navigation

Type to search…

↑↓ navigate↵ selectEsc close