Skip to content

Transformer Foundations

Build the intuition and technical foundations to understand the Transformer, modern LLMs and VLMs, and their evolving architectures.

Updated View as Markdown

This page builds a foundational understanding of the Transformer: a starting point for studying modern large language models (LLMs), vision language models (VLMs), and the architectures that come next.

Start with a simple idea: turn one sequence into another, as Figure 1 illustrates. A sentence becomes a translation, a document becomes a summary, and a question becomes an answer. This sequence to sequence formulation brings many language tasks under one framework.1 It can also reach beyond text: an image can become a sequence describing the objects it contains and where they are.2 The original Transformer was designed for translation, but the broader idea is a useful starting point: represent an input and learn to produce the desired output.3

Doing this well requires context. “Bank” means something different beside “river” than beside “account.” A Transformer builds numerical representations of its input, then refines them through repeated layers. Each position draws on relevant information from the context it is allowed to see.3

The design also suits large scale training. Unlike recurrent models, which process a sequence through dependent steps, a Transformer can process many positions together using matrix operations that run efficiently on GPUs.3 Researchers can grow the model through more layers and wider representations; studies have found predictable improvements in text prediction as model size, data, and compute increase within tested ranges.4 This is not unlimited: comparing all pairs of positions is costly for long contexts, and ordinary text generation still proceeds one token at a time.3

We follow the original architecture, connect it to Harvard NLP’s readable implementation, and then look at modern LLMs and VLMs. Each component is introduced when it becomes useful.

Figure 1. Different tasks, a shared sequence formulation. AI generated illustration informed by Transformer, T5, and Pix2Seq.312

Reading path What to understand
Architecture and data flow Two streams, encoder memory, decoder states, vocabulary scores
Attention and masking Q, K, V, scaling, softmax, visibility, head shapes, KV sharing
Components and full model Residuals, normalization, FFN, embeddings, position, independent layers
Training and inference Shifted labels, loss, optimization, SFT, decoding, caching
From Transformer to modern LLMs Which contracts persist and which design choices change

Architecture and data flow

Read the original architecture

Figure 2. Original Transformer architecture. Reproduced from Figure 1 of Vaswani et al. (2017, v7), with the source’s scholarly reproduction permission.3

The encoder creates contextual representations of a source sequence. The decoder reads a target prefix and that encoder memory to predict the next target token. Its masked self-attention prevents access to later target positions; cross-attention can read all unpadded source positions. The paper draws post-normalization: residual addition comes before LayerNorm.3

Follow the top level contract

Encoder and decoder data flow
Two token paths connected by encoder memorySource IDs of shape B by S pass through embeddings and position encoding, then N encoder layers, to produce memory Z of shape B by S by D. Shifted target IDs of shape B by T pass through their own embeddings and position encoding, then N decoder layers, producing hidden states of shape B by T by D. The separate dashed connection starts at memory Z and enters the decoder, where every cross attention layer reads that memory. The generator applies a linear vocabulary projection and log-softmax, producing scores of shape B by T by V. The source length S and target length T can differ. The diagram follows the Harvard implementation.SOURCE PATHTARGET PATHcross attentionreads ZSource IDsB × SEmbed +positionB × S × DEncoder × Nself attention+ FFNB × S × DMemory ZB × S × DTarget IDsshifted prefixB × TEmbed +positionB × T × DDecoder × Nmasked self+ cross + FFNB × T × DGeneratorlinear +log-softmaxScoresB × T × VTwo token paths connected by encoder memorySource IDs of shape B by S pass through embeddings and position encoding, then N encoder layers, to produce memory Z of shape B by S by D. Shifted target IDs of shape B by T pass through their own embeddings and position encoding, then N decoder layers, producing hidden states of shape B by T by D. The separate dashed connection starts at memory Z and enters the decoder, where every cross attention layer reads that memory. The generator applies a linear vocabulary projection and log-softmax, producing scores of shape B by T by V. The source length S and target length T can differ. The diagram follows the Harvard implementation.SOURCE PATHTARGET PATHZSource IDsB × SEmbed +positionB × S × DEncoder × Nself attention+ FFNB × S × DMemory ZB × S × DTarget IDsshifted prefixB × TEmbed +positionB × T × DDecoder × Nmasked self+ cross + FFNB × T × DGeneratorlinear +log-softmaxScoresB × T × V

Memory Z connects the two paths. Every decoder cross attention layer reads it; S and T need not match.

Figure 3. Two streams connected by encoder memory. Original diagram based on the Transformer architecture and Harvard implementation.35

Let B be batch size, S source length, T target input length, D model width, and V target vocabulary size. The two sequence lengths can differ: cross-attention uses T queries against S keys and values.

Value Shape Meaning
Source IDs B × S Source token indices
Source embedding B × S × D Token features plus position
Encoder memory Z B × S × D Contextual source features
Shifted target IDs B × T Target prefix at each position
Decoder hidden states B × T × D Contextual target features
Vocabulary logits or log probabilities B × T × V Next token scores per target position

This shape ledger is a local derivation from the module interfaces.5 The encoder preserves S; the decoder preserves T; neither requires source length to equal target length.

About the code examples

These are teaching excerpts, not a standalone runnable library. They assume PyTorch and the helper modules from the pinned Harvard tutorial. Differences between the original paper and the tutorial are explained where they matter. Adapted code is MIT licensed, copyright 2018 Alexander Rush. The full license notice is retained in public/licenses/annotated-transformer-MIT.txt.56

# Adapted from Harvard NLP, commit debc9fd, MIT license.
class EncoderDecoder(nn.Module):
    def __init__(self, encoder, decoder, src_embed, tgt_embed, generator):
        super().__init__()
        self.encoder, self.decoder = encoder, decoder
        self.src_embed, self.tgt_embed = src_embed, tgt_embed
        self.generator = generator

    def forward(self, src, tgt, src_mask, tgt_mask):
        memory = self.encode(src, src_mask)
        return self.decode(memory, src_mask, tgt, tgt_mask)

    def encode(self, src, src_mask):
        return self.encoder(self.src_embed(src), src_mask)

    def decode(self, memory, src_mask, tgt, tgt_mask):
        return self.decoder(self.tgt_embed(tgt), memory, src_mask, tgt_mask)

The top level forward returns decoder features, not probabilities. The generator is a separate vocabulary projection followed by log-softmax:

# Adapted from the same pinned Harvard implementation.
class Generator(nn.Module):
    def __init__(self, d_model, vocab):
        super().__init__()
        self.proj = nn.Linear(d_model, vocab)

    def forward(self, x):
        return F.log_softmax(self.proj(x), dim=-1)

D is a representation width; V is a number of candidate tokens. A modern loss API may consume raw logits and apply the stable log-softmax internally. For example, PyTorch CrossEntropyLoss expects unnormalized logits and applies log-softmax internally. Check the API contract rather than assuming every loss expects the tutorial’s output.7 The tutorial’s generator returns log probabilities for its KL loss.5

The separate generator also makes weight tying an explicit choice. The original paper shares embedding and output projection weights; the pinned tutorial factory instantiates separate modules rather than establishing that sharing. A shared matrix needs compatible vocabulary and width contracts.35

Attention and masking

One head from scores to content

Queries encode matching features for destination positions. Keys encode matching features for source positions. Values carry the content to combine. “What I seek,” “how I can be found,” and “what I carry” are useful interpretations, not fixed semantic roles assigned to individual neurons.

Attention⁡(Q,K,V)=softmax⁡(QK⊤dk)V(1)\operatorname{Attention}(Q,K,V) =\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V \tag{1}

Here dkd_k is key width, and softmax normalizes over key positions. This is the paper’s original scaled dot product attention formula.3

When masking is needed, forbidden scores are set to negative infinity before softmax. Implementations can replace logits or use a fused causal kernel rather than materializing an additive mask.38

One head: from scores to content
Queries and keys determine weights; values supply contentQueries Q and keys K form scaled dot product scores. Optional masking sets forbidden scores to negative infinity. Softmax over key positions produces weights A. The separate value input V enters only at the last step, where A times V produces a weighted mixture of value vectors. The numbered steps run from compare to mask to normalize to mix values.Q and KV · content1 CompareQKᵀ / √dₖmatch + scale2 Maskblocked → −∞only if needed3 Normalizesoftmax → Aweights over keys4 Mix valuesA Vweighted contentQueries and keys determine weights; values supply contentQueries Q and keys K form scaled dot product scores. Optional masking sets forbidden scores to negative infinity. Softmax over key positions produces weights A. The separate value input V enters only at the last step, where A times V produces a weighted mixture of value vectors. The numbered steps run from compare to mask to normalize to mix values.Q and KV · content1 CompareQKᵀ / √dₖmatch + scale2 Maskblocked → −∞only if needed3 Normalizesoftmax → Aweights over keys4 Mix valuesA Vweighted content
Causal visibility

✓ visible · × blocked
Visibility is not an attention weight.

Follow query 2
StageKey 1Key 2Key 3
Scores214
Masked21−∞
Weights0.7310.2690

Output ≈ 0.731 v₁ + 0.269 v₂

Key 3 has the highest score, but is in the future.
Masking gives it zero weight. These numbers are illustrative.

The diagonal is visible. Padding can exclude additional keys.

Figure 4. Scores become weights, then weighted content. Original diagram based on the paper and Harvard implementation; example numbers are calculated here.35

Queries and keys determine routing weights; values supply the content to mix.3 The grid shows permitted connections, not attention weights. For the illustrated query 2, scaled scores [2, 1, 4] become [2, 1, −∞], giving weights about [0.731, 0.269, 0]. The output is therefore about 0.731 times the first value vector plus 0.269 times the second. This row is an illustrative calculation, not a measured model result.

For one head, Tq queries and Tk key/value positions give:

Tensor Shape Interpretation
Q B × Tq × dₖ Destination matching features
K B × Tk × dₖ Source matching features
V B × Tk × dᵥ Source content
QKᵀ and attention weights B × Tq × Tk One routing distribution per query
Output B × Tq × dᵥ Weighted source content

K and V must share their sequence length because the weight assigned to key j multiplies value j. Q can have a different length. The contraction is over key positions, leaving one output for every query. These shapes follow directly from the attention equation.3

Why divide by the square root

Under the illustrative assumption that components of q and k are independent, zero mean, and unit variance:

q⊤k=∑i=1dkqiki,Var⁡(q⊤k)=dk(2)q^\top k=\sum_{i=1}^{d_k}q_i k_i, \qquad \operatorname{Var}(q^\top k)=d_k \tag{2}

The standard deviation grows as dk\sqrt{d_k}. Dividing by that factor keeps the scale near one under these assumptions and helps avoid saturating softmax early. This is a variance argument, not a claim that learned features always obey those assumptions.3

# Adapted from Harvard NLP's attention function, MIT license.
def attention(query, key, value, mask=None, dropout=None):
    scores = query @ key.transpose(-2, -1)
    scores = scores / math.sqrt(query.size(-1))
    if mask is not None:
        # This teaching variant uses -inf instead of the tutorial's -1e9.
        scores = scores.masked_fill(~mask.bool(), float('-inf'))
    weights = scores.softmax(dim=-1)
    if dropout is not None:
        weights = dropout(weights)
    return weights @ value, weights

The order is score, scale, mask, normalize, optional training dropout, and value mixing. Softmax produces normalized rows before dropout; a realized row after dropout need not sum to one. Every scored row needs at least one visible key. An all −∞ row has no meaningful softmax distribution. The tutorial’s finite sentinel does not fix that semantic error, and −1e9 is not safe for every dtype. This excerpt is for understanding, not a substitute for a tested production kernel.5

Padding and causality are different restrictions

In Figure 4, multiplying scores by the Boolean mask instead would give [2, 1, 0], which still assigns positive probability to the blocked cell. Masking weights afterwards requires explicit renormalization. This numerical example is a local calculation.

Mask What it excludes Shape before head broadcast
Source padding Padded source keys B × 1 × S
Target padding Padded target keys B × 1 × T
Causal visibility Keys later than the query 1 × T × T
Combined target visibility Padding or future keys B × T × T
# Adapted from Harvard NLP's subsequent_mask, MIT license.
def subsequent_mask(length):
    future = torch.triu(torch.ones(1, length, length), diagonal=1).bool()
    return ~future  # True means visible in this page's teaching code.

Decoder row ii can see keys j≤ij \le i. This diagonal is valid because the input is shifted relative to the label. Source padding is reused in encoder self-attention and decoder cross-attention. Mask polarity differs across APIs: shape compatibility alone does not establish correct semantics.58

Multi-head attention

Each head uses learned Q, K, and V projections. Its attention weights can differ from those of another head. The outputs are concatenated and projected back to model width. Concatenation is not averaging, and projecting into several spaces is not simply copying the same attention computation.3

head⁡i=Attention⁡(QWiQ,KWiK,VWiV)MultiHead⁡(Q,K,V)=Concat⁡(head⁡1,…,head⁡H)WO(3)\begin{aligned} \operatorname{head}_i &=\operatorname{Attention}(QW_i^Q,KW_i^K,VW_i^V) \\ \operatorname{MultiHead}(Q,K,V) &=\operatorname{Concat}(\operatorname{head}_1,\ldots,\operatorname{head}_H)W^O \end{aligned} \tag{3}
Different heads, different attention weights
Query statesB × Tq × DQuery projection → Q
Key / value statesB × Tk × DKey / value projections → K, V

Self-attention uses the same states for both inputs. Cross-attention can use different lengths, Tq and Tk.

Head 1Q₁ · K₁ · V₁A₁ V₁
Head 2Q₂ · K₂ · V₂A₂ V₂
Head HQₕ · Kₕ · VₕAₕ Vₕ

Aᵢ contains Tq × Tk weights per batch item. Each head returns B × Tq × dᵥ.

Concatenate, then mix with WᴼB × Tq × (H · dᵥ) → B × Tq × D

Concatenate, not average. In the original equal-width setup, dₖ = dᵥ = D/H. The cards are illustrative, not a checkpoint.

Figure 5. Parallel heads, concatenated outputs. Original diagram based on the paper and Harvard implementation; head cards are illustrative.35

Query states have shape B × Tq × D; key and value states have shape B × Tk × D. Each head forms Tq × Tk attention weights per batch item and returns B × Tq × dv. Concatenation and the output projection restore width D. In self-attention both inputs are the same states; in cross-attention their sequence lengths can differ.35

For HH heads in the original equal width setup, dk=dv=D/Hd_k=d_v=D/H:

Learned projections    B × T × D
Reshape                B × T × H × d
Transpose              B × H × T × d
Attention scores       B × H × Tq × Tk
Head outputs           B × H × Tq × d
Transpose back         B × Tq × H × d
Concatenate            B × Tq × D
Output projection      B × Tq × D
# Adapted shape excerpt from Harvard's MultiHeadedAttention, MIT license.
if mask is not None:
    mask = mask.unsqueeze(1)  # Share visibility across the head axis.
query, key, value = [
    linear(x).view(batch_size, -1, num_heads, head_width).transpose(1, 2)
    for linear, x in zip(qkv_linears, (query, key, value))
]
head_output, weights = attention(query, key, value, mask=mask)
output = head_output.transpose(1, 2).contiguous().view(
    batch_size, -1, num_heads * head_width
)
output = output_linear(output)

transpose changes strides. The tutorial requests contiguous storage before using view to merge head and feature axes. If using reshape instead, understand when it can copy rather than assume the memory layout stayed unchanged.5

Three attention patterns

Location Query input Key and value input Visibility
Encoder self-attention Source states Same source states Unpadded source
Decoder self-attention Target states Same target states Unpadded causal prefix
Cross-attention Decoder states Encoder memory Unpadded source

The function is the same; tensor origin and visibility change. This is why cross-attention can return T outputs while reading S memory positions.3

MHA, GQA, and MQA

These variants can retain the same query head count Hq. What changes is the number of distinct key and value heads Hkv.

Figure 6. Query and KV sharing. Reproduced unchanged from Figure 2 of Ainslie et al., GQA (2023), under CC BY 4.0.9

Variant with Hq = 8 Hkv KV sharing Relative cache elements
MHA 8 One KV pair per query head 100%
Illustrated GQA 4 Two query heads per KV pair 50%
MQA 1 All query heads share one KV pair 12.5%

Each query head still produces its own score matrix and weighted output. Q has shape B × Hq × Tq × dₖ, K has B × Hkv × Tk × dₖ, and V has B × Hkv × Tk × dᵥ. A mapping g(h)g(h) assigns each query head its KV head:

head⁡h=softmax⁡(QhKg(h)⊤dk)Vg(h)(4)\operatorname{head}_h =\operatorname{softmax}\left(\frac{Q_h K_{g(h)}^\top}{\sqrt{d_k}}\right)V_{g(h)} \tag{4}

For equal groups, each KV head serves Hq/Hkv query heads. Efficient kernels can reuse K/V without materializing copies for every query head.8 The table’s cache fractions are a local derivation for equal head width and sequence length.9

The GQA paper also converts MHA checkpoints by averaging groups of K/V projection heads and continuing training. That is an uptraining recipe, not a pooling operation in every GQA forward pass. KV sharing primarily targets cache and memory bandwidth; it does not turn full attention into a linear operator.9

Components and full model

The assembly map

A residual path around every sublayer
The unchanged input bypasses LayerNorm, the sublayer, and dropoutInput x follows two branches. One applies LayerNorm, a sublayer, then dropout. The other carries x unchanged. Addition merges them, preserving the model width. This is Harvard's Pre-LN implementation, not the paper's Post-LN order.xunchanged residual xLayerNormSublayerDropout+yBoth branches have width D at the addition.The unchanged input bypasses LayerNorm, the sublayer, and dropoutInput x follows two branches. One applies LayerNorm, a sublayer, then dropout. The other carries x unchanged. Addition merges them, preserving the model width. This is Harvard's Pre-LN implementation, not the paper's Post-LN order.xresidualxLayerNormSublayerDropout+y · width D
One layer · repeat N times
  1. Self-attention
  2. FFN
Each item uses the block above.
Final LayerNorm → Memory Z
One layer · repeat N times
  1. Masked self-attention
  2. Cross-attention · K, V from Z
  3. FFN
Each item uses the block above.
Final LayerNorm → Generator

Repeat with independent parameters. The final norm sits outside the repeat.

Figure 7. Residual bypasses and repeated layers. Original diagram of the pinned Harvard Pre-LN implementation, not the paper’s Post-LN order.53

Each sublayer transforms a normalized branch; its unchanged input bypasses that branch and is added back after dropout. An encoder layer has self-attention and FFN blocks; a decoder layer adds cross-attention. Each stack repeats its layer N times with independent parameters, then applies a final LayerNorm.5

Residual paths and normalization

The Harvard wrapper normalizes the input before the sublayer:

Pre-LN:y=x+Dropout⁡(F(LN⁡(x)))Post-LN:y=LN⁡(x+Dropout⁡(F(x)))(5)\begin{aligned} \text{Pre-LN:}\quad y &=x+\operatorname{Dropout}\bigl(F(\operatorname{LN}(x))\bigr) \\ \text{Post-LN:}\quad y &=\operatorname{LN}\bigl(x+\operatorname{Dropout}(F(x))\bigr) \end{aligned} \tag{5}
# Adapted from Harvard NLP's SublayerConnection, MIT license.
class SublayerConnection(nn.Module):
    def __init__(self, size, dropout):
        super().__init__()
        self.norm = LayerNorm(size)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x, sublayer):
        return x + self.dropout(sublayer(self.norm(x)))

Residual addition requires matching shapes. It carries the input forward while adding a learned update. LayerNorm transforms each token’s feature vector without mixing sequence positions. Normalization placement changes the gradient path; it is not merely interchangeable notation. The tutorial additionally normalizes the output of each completed stack.510

Read the implementation as well as the name. The pinned tutorial’s custom LayerNorm uses x.std and adds epsilon outside the standard deviation. That does not establish numerical equivalence to a framework LayerNorm with its own variance estimator and epsilon convention. For a reproduction, preserve the implementation; for new code, use a tested layer and document the change.5

Encoder and decoder layers

# Explanatory pseudocode adapted from the pinned Harvard layers.
def encoder_layer(x, source_mask):
    x = residual(x, lambda h: self_attention(h, h, h, source_mask))
    return residual(x, feed_forward)

def decoder_layer(x, memory, source_mask, target_mask):
    x = residual(x, lambda h: self_attention(h, h, h, target_mask))
    x = residual(x, lambda h: cross_attention(h, memory, memory, source_mask))
    return residual(x, feed_forward)

The wrapper passes normalized h to the lambda. In cross-attention, only the decoder query stream follows that residual wrapper; keys and values come from the already encoded memory. The self-attention and cross-attention objects have separate learned projection weights.5

The position wise FFN

FFN⁡(x)=ReLU⁡(xW1+b1)W2+b2(6)\operatorname{FFN}(x)=\operatorname{ReLU}(xW_1+b_1)W_2+b_2 \tag{6}

Attention determines which contextual information reaches a position. The FFN then provides a nonlinear feature transformation at that position, using shared weights across tokens. It does not directly mix the sequence axis. The paper’s base configuration expands width 512 → 2048 → 512.3

# Adapted from Harvard's PositionwiseFeedForward, MIT license.
def forward(self, x):
    return self.w_2(self.dropout(self.w_1(x).relu()))

The expansion gives the nonlinear operation more intermediate features while the contraction restores the residual interface. Without an intervening nonlinearity, two linear transformations would collapse into one linear map. That is an algebraic explanation, not proof that every expansion ratio is optimal.

Embeddings and sinusoidal position

Token IDs select learned vectors. The tutorial multiplies them by D\sqrt{D}, adds a sinusoidal position vector, then applies dropout:

PE⁡(p,2i)=sin⁡(p100002i/D)PE⁡(p,2i+1)=cos⁡(p100002i/D)(7)\begin{aligned} \operatorname{PE}(p,2i)&=\sin\left(\frac{p}{10000^{2i/D}}\right) \\ \operatorname{PE}(p,2i+1)&=\cos\left(\frac{p}{10000^{2i/D}}\right) \end{aligned} \tag{7}

Each sine and cosine pair uses a different angular frequency. The position vector has width D so it can be added to token features; a stored 1 × max_length × D buffer broadcasts across the batch.35

Position as a bundle of waves

Exact toy encoding: D = 8. Three of its four sine / cosine pairs are shown.

Sine · solidCosine · dashed
Channels 0, 1 · ω = 1Sine and cosine at angular frequency 1Position ranges from zero to sixty-three. Encoding values range from minus one to plus one. The solid sine starts at zero; the dashed cosine starts at one. Lower frequencies change more slowly. Lines evaluate the analytical functions between integer positions for readability.−10+1016324863Sine and cosine at angular frequency 1Position ranges from zero to sixty-three. Encoding values range from minus one to plus one. The solid sine starts at zero; the dashed cosine starts at one. Lower frequencies change more slowly. Lines evaluate the analytical functions between integer positions for readability.−10+1016324863
Channels 2, 3 · ω = 0.1Sine and cosine at angular frequency 0.1Position ranges from zero to sixty-three. Encoding values range from minus one to plus one. The solid sine starts at zero; the dashed cosine starts at one. Lower frequencies change more slowly. Lines evaluate the analytical functions between integer positions for readability.−10+1016324863Sine and cosine at angular frequency 0.1Position ranges from zero to sixty-three. Encoding values range from minus one to plus one. The solid sine starts at zero; the dashed cosine starts at one. Lower frequencies change more slowly. Lines evaluate the analytical functions between integer positions for readability.−10+1016324863
Channels 4, 5 · ω = 0.01Sine and cosine at angular frequency 0.01Position ranges from zero to sixty-three. Encoding values range from minus one to plus one. The solid sine starts at zero; the dashed cosine starts at one. Lower frequencies change more slowly. Lines evaluate the analytical functions between integer positions for readability.−10+1016324863Sine and cosine at angular frequency 0.01Position ranges from zero to sixty-three. Encoding values range from minus one to plus one. The solid sine starts at zero; the dashed cosine starts at one. Lower frequencies change more slowly. Lines evaluate the analytical functions between integer positions for readability.−10+1016324863

Position p →

Vertical axis: encoding value. Frequencies follow the paper's rule, ωᵢ = 10000^(−2i/D); this is a calculation, not a trained-model measurement.

Figure 8. Position at different frequencies. Calculated from the paper’s sinusoidal rule for a toy width of D = 8.3

Three channel pairs are plotted over positions 0–63, with angular frequencies 1, 0.1, and 0.01. The fourth pair, at 0.001, is omitted. Every pair has a sine channel starting at zero and a cosine channel starting at one; lower frequencies vary more slowly. The lines show the analytical functions between integer positions, not a measured model result.

# Teaching excerpt adapted from Harvard, MIT license; D is even here.
positions = torch.arange(max_length).unsqueeze(1)
frequencies = torch.exp(torch.arange(0, D, 2) * (-math.log(10000.0) / D))
pe = torch.zeros(max_length, D)
pe[:, 0::2] = torch.sin(positions * frequencies)
pe[:, 1::2] = torch.cos(positions * frequencies)
# In an nn.Module, register the fixed tensor as a buffer, not a parameter.
self.register_buffer('pe', pe.unsqueeze(0))
model_input = dropout(embedding(token_ids) * math.sqrt(D) + self.pe[:, :T])

Sine and cosine addition identities make a fixed position offset a linear rotation within each pair. This explains the relative offset motivation; it does not guarantee useful generation beyond the trained length. Ordinary self-attention alone, without position or asymmetric visibility, cannot distinguish arbitrary permutations by order.3

Independent layers and the factory

Deep-copying a module creates independent parameter objects. Calling the same module object N times ties weights. Identical structure and identical initial values are not the same as shared parameters. The tutorial factory builds distinct encoder/decoder layers and distinct self/cross-attention modules:

# Condensed from Harvard's make_model; classes come from that tutorial.
c = copy.deepcopy
attn = MultiHeadedAttention(heads, d_model)
ff = PositionwiseFeedForward(d_model, d_ff, dropout)
position = PositionalEncoding(d_model, dropout)
model = EncoderDecoder(
    Encoder(EncoderLayer(d_model, c(attn), c(ff), dropout), layers),
    Decoder(DecoderLayer(d_model, c(attn), c(attn), c(ff), dropout), layers),
    nn.Sequential(Embeddings(d_model, source_vocab), c(position)),
    nn.Sequential(Embeddings(d_model, target_vocab), c(position)),
    Generator(d_model, target_vocab),
)
for parameter in model.parameters():
    if parameter.dim() > 1:
        nn.init.xavier_uniform_(parameter)

The factory’s Xavier initialization follows construction. Architecture contracts alone do not specify optimization behavior.5

Boundary Invariant
Embedding to stack Final width D
Sublayer to residual add Same input and output shape
Original head split D divisible by H
Memory to cross-attention Width compatible with K/V projections
Decoder to generator Width D
Generator to loss Vocabulary axis V and correctly aligned labels

This boundary checklist is a local synthesis of the paper and tutorial interfaces.35

Training and inference

Teacher forcing and shifted labels

Two ways to predict the next token
Position1234
InputBOSy₁y₂y₃
Labely₁y₂y₃EOS
Scores and known labels meet at the loss, not at generationOne causal decoder and generator pass takes the input row and produces scores at all four positions. Known next-token labels independently enter the loss. Backpropagation and the optimizer update follow. The mask prevents each position from reading future inputs.Causal passdecoder + generatorScoresall positionsLosscompare with labelsKnown label rowScores and known labels meet at the loss, not at generationOne causal decoder and generator pass takes the input row and produces scores at all four positions. Known next-token labels independently enter the loss. Backpropagation and the optimizer update follow. The mask prevents each position from reading future inputs.Causal passdecoder + headScoresall positionsLosscompare labelsKnownlabel row

Loss → backward → optimizer update. Predictions are trained in parallel, but each position sees only its allowed prefix.

An autoregressive feedback loopA prefix containing BOS and the previously chosen token enters the decoder and generator. Choose the next token from its distribution, append that token, then repeat with the updated prefix. Stop at EOS or the length limit. Encoder memory is fixed and not drawn here.Current prefixBOS, ŷ₁Decoder + headnext-token distributionChoose ŷ₂sample or argmaxappend the chosen token; repeatAn autoregressive feedback loopA prefix containing BOS and the previously chosen token enters the decoder and generator. Choose the next token from its distribution, append that token, then repeat with the updated prefix. Stop at EOS or the length limit. Encoder memory is fixed and not drawn here.Current prefixBOS, ŷ₁Decoder + headnext-token distributionChoose ŷ₂sample or argmaxappendtoken

Stop at EOS or the length limit. KV caching saves repeated work; it does not remove the next-token dependency.

Figure 9. Parallel training, autoregressive generation. Original diagram based on Harvard’s Batch and decoding loops; EOS stopping is added here.5

During training, shifted inputs are known in advance and scores at each position are compared with the next known label. Labels enter the loss, not a token selection step. During generation, the selected token is appended to the prefix and fed back for the next step; stop at EOS or the length limit. Cache reuse changes the work performed, not this dependency.51112

Full target       BOS  y₁  y₂  y₃  EOS
Decoder input     BOS  y₁  y₂  y₃
Prediction label  y₁   y₂  y₃  EOS

All correct previous tokens are available at training time. The causal mask prevents a prediction from reading future target inputs. Teacher forcing does not mean that future labels are visible through attention. At inference time the model conditions on its own previous choices instead.5

# Condensed from Harvard's Batch, MIT license.
source_mask = (source != pad_id).unsqueeze(-2)
target_input = target[:, :-1]
target_label = target[:, 1:]
target_mask = (target_input != pad_id).unsqueeze(-2)
target_mask = target_mask & subsequent_mask(target_input.size(-1)).to(target.device)
token_count = (target_label != pad_id).sum()

Padding masks block padded keys; loss handling separately excludes padded prediction targets. Padded query rows can still produce hidden states, which must not accidentally become supervised targets. Normalize the loss by actual targets, not the padded area of the batch.5

Forward pass and optimization

A training step prepares inputs and masks, runs the model and generator, computes token loss, normalizes it, backpropagates, updates parameters at the chosen optimizer boundary, advances the scheduler, and clears gradients. Evaluation disables parameter updates and training dropout. Architecture defines the differentiable function; the loop defines execution policy.5

The paper used Adam with β1=0.9\beta_1=0.9, β2=0.98\beta_2=0.98, and ϵ=10−9\epsilon=10^{-9}, and a warmup followed by inverse square root decay. These are historical recipe choices, not defaults recommended for every current model.3

lr⁡(s)=factor⋅D−1/2min⁡(s−1/2,s w−3/2)(8)\operatorname{lr}(s)=\text{factor}\cdot D^{-1/2} \min\left(s^{-1/2},s\,w^{-3/2}\right) \tag{8}

Here ss is optimizer step and ww is warmup length. Before ww the schedule rises linearly; at ww the branches meet; afterwards it decays as s−1/2s^{-1/2}. The tutorial exposes a multiplicative factor:5

# Adapted from Harvard's rate function, MIT license.
def rate(step, model_size, factor, warmup):
    step = max(step, 1)
    return factor * model_size ** (-0.5) * min(
        step ** (-0.5), step * warmup ** (-1.5)
    )

Label smoothing and loss interpretation

For a simplified vocabulary with no padding class, distribute target mass as:

q(y)=1−ϵ,q(k)=ϵV−1for k≠y(9)q(y)=1-\epsilon,\qquad q(k)=\frac{\epsilon}{V-1}\quad\text{for }k\ne y \tag{9}

The pinned tutorial reserves a padding class, distributes smoothing over V−2V-2 alternative classes, sets padding probability to zero, and removes entire loss rows whose target is padding. It minimizes KL⁡(q∥p)\operatorname{KL}(q\Vert p) using model log probabilities as input.5

Smoothing discourages assigning all probability to the observed label, but does not universally improve every metric. The original paper reports improved accuracy and BLEU while perplexity worsened.3

A useful algebraic distinction: cross entropy equals H(q)+KL⁡(q∥p)H(q)+\operatorname{KL}(q\Vert p). For a nondegenerate smoothed target, its minimum is positive H(q)H(q). KL itself can reach zero when p=qp=q. Do not compare raw smoothed loss to ordinary hard target cross-entropy as if their baselines were the same.

The bridge to VLM supervised fine tuning

The same conditional next token objective applies when context xx includes an image and instruction, and yy is the target answer:

L(θ)=−∑tmtlog⁡pθ(yt∣x,y<t)∑tmt(10)\mathcal{L}(\theta) =-\frac{\sum_t m_t\log p_\theta(y_t\mid x,y_{\lt t})}{\sum_t m_t} \tag{10}

Here y<ty_{\lt t} denotes the preceding answer tokens, and mtm_t selects supervised positions with at least one selected target. This is local notation for token averaged conditional cross-entropy, with the conditional objective and ignore rules described by SFT and loss documentation. It is not a proprietary training recipe.137

An attention mask controls what a position can read. A loss mask controls which predictions contribute gradients. Prompt tokens can remain visible context while their prediction targets are ignored. Completion or assistant only loss depends on data format, chat template, and configuration. Inspect decoded examples and labels rather than relying on a setting’s name.13

With bb examples per device, aa accumulated microbatches, and nn data parallel workers, the nominal effective batch is b⋅a⋅nb\cdot a\cdot n. That count does not prove numerical equivalence to one physical batch.

For microbatch jj with summed token loss SjS_j and supervised token count NjN_j, the true combined token mean is:

L‾tokens=∑jSj∑jNj(11)\overline{\mathcal{L}}_{\text{tokens}}=\frac{\sum_j S_j}{\sum_j N_j} \tag{11}

It is not generally the mean of Sj/NjS_j/N_j. For example, token means 1 and 3 over 10 and 30 tokens give a combined mean 2.5, not 2. Choose whether tokens or examples should receive equal weight, and make distributed reductions and gradient scaling implement that choice. This is a local derivation. Mixed precision scaling, clipping, and optimizer stepping must respect the effective batch boundary.14

Full fine tuning updates the selected model weights. LoRA freezes base weights and learns low rank updates to selected matrices. It reduces trainable parameters and their optimizer state, not the need to run the base model. Rank and target matrices are design choices; fewer trainable parameters do not automatically mean a smaller inference model.15

Evaluate a data or training change

This is a proposed experiment checklist, not a report of executed experiments:

  1. Define the hypothesis, downstream metric, and important workload slices.
  2. Separate training, development, and final test roles. Group pages and synthetic variants of one source document when a random split would leak shared information. Hold out templates or sources if that is the intended generalization test.
  3. Fix the starting checkpoint, evaluation protocol, and declared budget. Equal steps and equal tokens are different comparisons.
  4. For two changes, compare baseline, A, B, and A+B when feasible.
  5. Repeat training when feasible. Resampling evaluation documents does not measure training seed variability.
  6. If only A+B was tested, attribute the result to that combined intervention.

Greedy decoding and the cache

# Condensed from Harvard's greedy_decode, MIT license.
# Assumes one source example. No KV cache is implemented in this excerpt.
memory = model.encode(source, source_mask)
prefix = torch.tensor([[bos_id]], device=source.device)
for _ in range(max_length - 1):
    mask = subsequent_mask(prefix.size(1)).to(source.device)
    hidden = model.decode(memory, source_mask, prefix, mask)
    log_probs = model.generator(hidden[:, -1])
    next_token = log_probs.argmax(dim=-1)
    prefix = torch.cat([prefix, next_token[:, None]], dim=1)
    # Added stopping rule; the pinned tutorial runs to the length limit.
    if next_token.item() == eos_id:
        break

Use evaluation mode and disable gradient recording for generation. The tutorial’s simple loop recomputes the full prefix each step; source memory is computed once. The EOS break above is an explicit teaching addition.5

Incremental causal decoding can instead reuse each layer’s earlier K/V. The new query reads cached history plus its own new K/V. Earlier causal states do not change when a later token is appended, so reuse preserves the model’s causal computation, up to numerical differences. For encoder/decoder models, cross-attention K/V derived from fixed memory can also be reused.12

For LL layers, cached length TcacheT_{\text{cache}}, equal key/value head width dd, and ss bytes per element, a simple uncompressed self attention cache uses:

KV cache bytes≈2LB TcacheHkvd s(12)\text{KV cache bytes}\approx 2LB\,T_{\text{cache}}H_{\text{kv}}d\,s \tag{12}

This is a local element-count derivation. It excludes allocation overhead, padding, sharding, quantization metadata, and any extra cross-attention cache. For unequal K/V widths, replace 2d2d with dk+dvd_k+d_v.

Greedy decoding selects the most probable next token locally. Beam search retains candidate prefixes; temperature rescales logits; top-k truncates to a fixed number of candidates; nucleus sampling retains a probability mass. These are decoding policies, not different trained architectures. Their usefulness depends on the task; greedy choice does not guarantee the best whole sequence.1116

Computational cost

For a dense decoder with LL layers, batch BB, sequence length TT, width DD, and FFN width proportional to DD, a local matrix multiplication count gives:

Full sequence forward:

O(LB(TD2+T2D))(13)\mathcal{O}\bigl(LB(TD^2+T^2D)\bigr) \tag{13}

One cached decoding step at history length TT:

O(LB(D2+TD))(14)\mathcal{O}\bigl(LB(D^2+TD)\bigr) \tag{14}

Projections and the FFN contribute the width squared term; QK⊤QK^\top and weight/value mixing contribute the sequence interaction term. This omits the vocabulary head, whose full sequence cost can be O(BTDV)\mathcal{O}(BTDV), and assumes ordinary dense attention rather than experts or restricted connectivity. A causal triangle changes constants, not quadratic growth. A full sequence forward is not the cost of repeatedly recomputing that prefix during generation.

The attention score tensor contains BHT2BHT^2 elements if materialized. Exact tiled kernels can avoid storing that full matrix while retaining quadratic arithmetic.17

From Transformer to modern LLMs

Design choices evolve along several axes
Axis2017 designLater choices
TopologyEncoder + decoderDecoder-only models
NormalizationPost-LNPre-norm; LayerNorm or RMSNorm
PositionAdded sine / cosine vectorsLearned embeddings or RoPE
FFNReLUGELU or gated activations
AttentionMHA; decoder cross-attentionMHA, GQA, MQA and other variants
Execution is another axis. Exact tiled attention kernels can change how attention is computed without changing the attention operation. Caching, precision and parallelism add separate serving choices.

These choices can coexist. This is a comparison, not a ranking or a recipe that every model follows.

Figure 10. Coexisting design choices, not an inevitable timeline. Original comparison synthesized from the cited architecture and implementation sources.31819202191712

One causal stream instead of two

A decoder language model removes the separate source encoder and its cross-attention path. Prompt, context, and continuation occupy one causal sequence. Attention, FFN, residual width, normalization, position, and a vocabulary head remain recognizable. GPT-2 is a concrete example with learned position embeddings and pre-normalized blocks, not the full list of later choices in the diagram.18

LayerNorm and RMSNorm

LayerNorm centers a feature vector and rescales it using variance. RMSNorm rescales by root mean square without subtracting the mean. This removes a statistical operation while preserving rescaling invariance; it is not the same function as LayerNorm and is not universally better for every model.19

Additive position and RoPE

Sinusoidal position adds a vector to the input residual stream. RoPE instead rotates query and key feature pairs with position dependent angles, making their dot product depend on relative displacement. It changes compatibility geometry, not just a stored table. Reliable long context behavior still needs appropriate training and evaluation.20 Continue with Position and long context.

ReLU, GELU, and gated FFNs

The original FFN uses ReLU; GPT-2 uses GELU. A gated FFN introduces a second input projection and multiplies the branches before projecting back:

SwiGLU⁡(x)=[SiLU⁡(xWg)⊙(xWu)]WdSiLU⁡(z)=z sigmoid⁡(z)(15)\begin{aligned} \operatorname{SwiGLU}(x)&=\bigl[\operatorname{SiLU}(xW_g)\odot(xW_u)\bigr]W_d \\ \operatorname{SiLU}(z)&=z\,\operatorname{sigmoid}(z) \end{aligned} \tag{15}

Biases are omitted in this notation. The gate is a learned multiplicative feature interaction, not a probability constrained to [0,1]. SwiGLU uses three projection matrices, so fair comparisons should account for parameter and compute budgets, not simply reuse the same hidden width. Shazeer’s paper reports gains in its controlled experiments, not a universal guarantee.21 Continue with FFNs and experts.

Different meanings of efficient attention

Change What changes What not to infer
FlashAttention IO schedule and score materialization Full attention becomes linear arithmetic
MQA or GQA Distinct KV projection heads and cache Query head count must shrink
Sliding window Allowed direct connections Every token still directly reads all history
MLA Joint compressed KV representation It is just fewer ordinary KV heads

FlashAttention preserves the dense attention operator up to numerical differences. Sliding window attention changes connectivity; Mistral describes how information can still travel across layers. DeepSeek-V2’s MLA introduces low rank KV compression. These mechanisms address different costs and should not be bundled into one unexplained claim of efficiency. The comparison table is a local synthesis of these mechanisms and the GQA design.1792223 Continue with Attention and memory.

Dense and routed experts

An MoE layer routes each token to selected FFN experts rather than activating all experts. Switch Transformer uses one expert per token with routing weights, capacity handling, and load balancing. The residual output still has width D. More total parameters do not imply that all parameters are active for every token, nor that device communication becomes free.24 Sparse expert activation is not the same as sparse attention.24

Information across depth

Attention and FFNs describe computation within a layer. Residual paths describe how representations reach later layers. Start with the original single-stream residual, then compare multiple streams, attention across earlier layers and gated reads and writes in Depth and residual paths. That companion page supplies the equations, source reports and their limits.

Read a model as a set of contracts

For any model report or implementation, record:

  1. Stack: encoder/decoder, prefix, or causal decoder?
  2. Normalization: which statistic, and where in the residual path?
  3. Position: additive, rotary, learned, or biased?
  4. Attention: full, local, latent, linear, or hybrid?
  5. Heads: query count, KV count, and head widths?
  6. FFN: activation, intermediate width, dense or routed?
  7. Context: trained length, served length, and any extension method?
  8. Runtime: cache, kernel, precision, and parallelism?
  9. Depth: one residual stream, multiple streams, or selective depth reads?
  10. Evidence: exact checkpoint, pinned code, and supporting ablations?

This is a local reading checklist. It helps separate the mathematical model from the strategy used to execute it. For VLMs, add how image features enter the model and how their positions and token budget are defined. Continue with Multimodal integration and Model Deep Dives.

Check your understanding

Shape and information checks
  • Why can S differ from T? Cross-attention maps T queries over S keys/values.
  • What does the decoder return? D-wide features, before the V-wide generator.
  • Which axis does attention softmax normalize? Key positions, for each query.
  • Does a causal mask make dense attention linear? No; the dense interaction matrix remains quadratic in length.
  • Does deep-copying layers share parameters? No; reusing the same object would.
  • Can ignored prompt labels remain visible context? Yes; loss and attention masks serve different purposes.13
  • Does KV caching alter causal dependencies? No; it reuses earlier computation.12
  • Does an MoE layer activate every expert? No; the router chooses a subset.24
Optional implementation checks
  1. Assert the source, memory, target, head, residual, and vocabulary shapes. Deliberately swap sequence axes and predict the first failing operation.
  2. Test self-attention, cross-attention with different lengths, padding, and causality. Changing hidden K/V positions should not change visible outputs when dropout is disabled.
  3. Verify that two cloned layers have distinct parameter storage. Check every residual addition for equal shapes.
  4. Print shifted inputs, labels, and one combined mask row. Compare cached and uncached generation logits within numerical tolerance.
  5. Fill the reading checklist for one model, cite each source section, and mark whether an efficiency choice changes arithmetic, traffic, cache, or active parameters.

These are learning exercises, not tests reported as completed on this page.

Sources and attribution

The primary implementation reference is Harvard NLP’s Annotated Transformer at commit debc9fd.5 Read the original tutorial for its full runnable implementation and experiments. Adapted excerpts retain Alexander Rush’s MIT license.6 Paper figures retain their own reproduction terms, not Latent Light’s MIT grant. Numbered references connect the explanations to their supporting sources.

References

  1. Colin Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR 2020, arXiv version 4. A shared text input and text output formulation covers translation, summarization, question answering, classification, and other language tasks. ↩ ↩2

  2. Ting Chen et al., Pix2seq: A Language Modeling Framework for Object Detection, ICLR 2022, arXiv version 2. Sections 2.1–2.2 describe serializing object labels and bounding box coordinates into tokens and generating them conditioned on an image. ↩ ↩2

  3. Ashish Vaswani et al., Attention Is All You Need, NeurIPS 2017, arXiv version 7. Sections 3.1–3.5 define the architecture, attention, FFN, embeddings, and position; sections 5.3–5.4 describe optimization and smoothing. Figure 1 is reproduced under the source’s stated scholarly or journalistic reproduction permission. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18 ↩19 ↩20 ↩21 ↩22 ↩23 ↩24 ↩25 ↩26 ↩27 ↩28

  4. Jared Kaplan et al., Scaling Laws for Neural Language Models, 2020, arXiv version 1. The authors report empirical relationships between language modeling loss, model size, dataset size, and training compute within the studied ranges. These findings do not guarantee improvement on every downstream task. ↩

  5. Harvard NLP, The Annotated Transformer, commit debc9fd747bb2123160a98046ad1c2d4da44a567. Inspected interfaces include EncoderDecoder and Generator (lines 230–269), residuals and stacks (292–438), attention and heads (441–628), FFN and position (677–786), make_model (822–848), Batch and training (907–1024), LabelSmoothing (1148–1176), and greedy_decode (1313–1328). Teaching changes are marked beside the excerpts. ↩ ↩2 ↩3 ↩4 ↩5 ↩6 ↩7 ↩8 ↩9 ↩10 ↩11 ↩12 ↩13 ↩14 ↩15 ↩16 ↩17 ↩18 ↩19 ↩20 ↩21 ↩22 ↩23 ↩24 ↩25 ↩26 ↩27 ↩28

  6. Alexander Rush, Annotated Transformer MIT license, copyright 2018, same pinned commit. The complete notice is retained with the adapted excerpts. ↩ ↩2

  7. PyTorch, CrossEntropyLoss, version 2.14. Logit inputs, LogSoftmax equivalence, ignored labels, and loss reduction. ↩ ↩2

  8. PyTorch, Scaled dot product attention, version 2.14. Mask polarity, causal handling, backend selection, GQA, and numerical differences between kernels. ↩ ↩2 ↩3

  9. Joshua Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, EMNLP 2023. Section 2 describes KV sharing and uptraining; Figure 2 compares MHA, GQA, and MQA. The inspected arXiv HTML version 3 provides the same mechanism. Figure 2 is reproduced unchanged under CC BY 4.0. ↩ ↩2 ↩3 ↩4 ↩5

  10. Ruibin Xiong et al., On Layer Normalization in the Transformer Architecture, ICML 2020. Analysis of normalization placement and gradient behavior. ↩

  11. Hugging Face, Transformers GenerationConfig, main documentation inspected October 3, 2026. Greedy decoding, beams, temperature, top-k, and top-p parameters. ↩ ↩2

  12. Hugging Face, Transformers cache strategies, main documentation inspected October 3, 2026. KV reuse and the encoder/decoder cache section. Main documentation can change; check the installed version when implementing. ↩ ↩2 ↩3 ↩4

  13. Hugging Face, TRL SFT Trainer: completion and assistant message loss, inspected October 3, 2026. Configuration and templates determine which tokens are supervised; behavior must be checked against the installed version. ↩ ↩2 ↩3

  14. PyTorch, Automatic mixed precision examples: gradient accumulation, version 2.14. Scaling, clipping, and optimizer boundaries for an effective batch. ↩

  15. Edward J. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021, ICLR 2022. Frozen base weights and trainable low rank adaptation matrices. ↩

  16. Ari Holtzman et al., The Curious Case of Neural Text Degeneration, ICLR 2020. Nucleus sampling and comparison with other decoding policies. ↩

  17. Tri Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, NeurIPS 2022, arXiv version 2. Sections 2–3 distinguish attention arithmetic from memory traffic and materialization. ↩ ↩2 ↩3

  18. Alec Radford et al., OpenAI, Language Models are Unsupervised Multitask Learners, 2019, section 2.3. GPT-2 architecture and normalization changes. ↩ ↩2

  19. Biao Zhang and Rico Sennrich, Root Mean Square Layer Normalization, NeurIPS 2019. Rescaling without mean centering. ↩ ↩2

  20. Jianlin Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding, 2021, arXiv version 5, section 3. Position dependent rotations and relative displacement in query/key dot products. ↩ ↩2

  21. Noam Shazeer, GLU Variants Improve Transformer, 2020, arXiv version 1. Gated FFN definitions and controlled comparisons. ↩ ↩2

  22. Albert Q. Jiang et al., Mistral 7B, 2023, arXiv version 1, section 2. Sliding window connectivity and information propagation across layers. ↩

  23. DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model, 2024, arXiv version 5, section 2.1. Multi-head latent attention and joint KV compression. ↩

  24. William Fedus, Barret Zoph, and Noam Shazeer, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, JMLR 2022, arXiv version 3, sections 2–3. Expert routing, capacity, and balancing. ↩ ↩2 ↩3

Navigation

Type to search…

↑↓ navigate↵ selectEsc close