---
title: "FFNs and Experts"
description: "Understand feature gates, sparse routing and the gap between active compute and stored capacity."
---

> Documentation Index
> Fetch the complete documentation index at: https://latentlit.com/llms.txt
> Use this file to discover all available pages before exploring further.

# FFNs and Experts

import FocusCanvas from '../../../../components/FocusCanvas.astro';
import ArchitectureAtlas from '../../../../components/architectures/ArchitectureAtlas.astro';
import SourceFigure from '../../../../components/architectures/SourceFigure.astro';

Attention moves information between positions. A feed forward network, or
FFN, transforms the features at each position. A **feature gate** changes
that transformation; an **expert router** chooses which transformations run.
The two can coexist, but they are not the same mechanism.[^transformer][^glu][^mixtral]

## A feature gate modulates, rather than selects

SwiGLU makes two projections of the same input and multiplies their outputs
before a final projection. Omitting biases:[^glu]

$$
\operatorname{SwiGLU}(x)=\bigl(\operatorname{SiLU}(xW_g)\odot xW_u\bigr)W_d,\qquad \operatorname{SiLU}(a)=a\,\sigma(a)
\tag{1}
$$

x is a row vector of width D. W<sub>g</sub> and W<sub>u</sub> map D to hidden
width F; W<sub>d</sub> maps F back to D. The symbol ⊙ means elementwise
multiplication, and σ is the logistic sigmoid.

<a id="figure-1"></a>

**One token, two feature branches**

**SwiGLU**: The same input x enters Wg and Wu. SiLU transforms the gate branch; elementwise multiplication combines both branches; Wd returns the result to width D.

Every branch is computed. The gate is not a probability and does not choose an expert.

**Figure 1.** Two computed branches, one multiplicative transformation. Independent schematic of the bias omitted SwiGLU FFN.[^glu]

Both branches in [Figure 1](#figure-1) run for every token. SiLU is smooth,
can be negative, and is unbounded above. The branch is therefore not a
probability in the range zero to one. Its learned multiplication can modulate
features without being an expert selection rule.[^glu]

### Compare gates at a comparable budget

A conventional two projection FFN has about 2DF<sub>dense</sub> weights;
a gated FFN has about 3DF<sub>gate</sub>. Ignoring biases, equalizing those
counts gives a local calculation:

$$
F_{\mathrm{gate}}\approx\frac{2}{3}F_{\mathrm{dense}}
\tag{2}
$$

The GLU paper uses this width adjustment. Its controlled T5 results support
gated variants in that setting, not a guarantee that the same activation
always wins at arbitrary width or training budget.[^glu]

## A router chooses separate FFNs

An MoE sublayer holds several experts, each with separate weights. A router
scores the current token representation and dispatches it to selected experts.
The weighted result returns to the usual residual width. Routing can change
with token and layer; an expert ID is not automatically a fixed subject label.[^mixtral]

<a id="figure-2"></a>

![Mixtral selects two parallel experts for a token.](/figures/model-studies/mixtral/expert-routing.png)

**Figure 2.** Selected experts run in parallel, not in a chain. Reproduced from Jiang et al., Mixtral Figure 1; unchanged figure crop, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).[^mixtral]

**Image description:** The Mixtral router selects two of eight experts, which process the same token in parallel before their weighted outputs are combined.

In Mixtral, each expert is itself a SwiGLU FFN. Thus the router in
[Figure 2](#figure-2) selects entire transformations, while the gate in
[Figure 1](#figure-1) operates inside each selected transformation.[^mixtral]

For token x, let S(x) be the selected set and g its router logits. A compact
form of Mixtral's top two normalized mixture is:[^mixtral]

$$
y=\sum_{i\in S(x)}\frac{e^{g_i}}{\sum_{j\in S(x)}e^{g_j}}E_i(x),\qquad |S(x)|=2
\tag{3}
$$

E<sub>i</sub> is expert i. The weights in this equation sum to one over the
selected set. Do not assume every other MoE normalizes or selects identically.

## Routing designs differ in meaningful ways

| Design | What changes | Concrete study |
| --- | --- | --- |
| Top two from eight experts | A simple sparse replacement for the dense FFN | [Mixtral](/architectures/models/mixtral/)[^mixtral] |
| Shared plus routed experts | Shared computation runs alongside a selected subset | [DeepSeek-V2](/architectures/models/deepseek-v2/)[^v2] |
| Bias controlled selection | Balancing biases affect selection, not the output mixing weights | [DeepSeek-V3](/architectures/models/deepseek-v3/)[^v3] |
| Routed experts without shared experts | Different family members have dense or sparse capacity | [Qwen3](/architectures/models/qwen3/)[^qwen] |

This is a mechanism comparison, not a quality ranking. V3's “auxiliary loss free”
balancing refers to its main batch balancing method; a small sequence balancing
loss remains.[^v3] Likewise, GLM-4.5's separately trained reasoning, agentic and
chat specialists are policies, not its routed FFN experts.[^glm]

For additional released designs, compare [gpt-oss](/architectures/models/gpt-oss/)
and the [MiniMax family](/architectures/models/minimax-01/).
Keep each study's exact version and counting convention.

## Sparse computation still needs a system

Selected expert weights reduce active computation relative to evaluating every
expert. Total weights still need storage somewhere. Dispatch, gathering,
communication, load imbalance, precision, and batch size affect whether the
design is efficient in practice.

Compare total parameters, active parameters, actual FLOPs, weight residency,
communication, and latency separately. “Active parameters” is a useful
summary, not a complete FLOP or VRAM measurement. Nor is a sparse FFN the
same thing as sparse attention.

Selected mechanisms rechecked October 10, 2026. No routing or serving
experiments were run here. For the surrounding residual block, continue with
[Depth and residual paths](/architectures/evolution/depth-and-residual-paths/).

[^transformer]: Ashish Vaswani et al. [Attention Is All You Need](https://arxiv.org/html/1706.03762v7), 2017, section 3.3.
[^glu]: Noam Shazeer. [GLU Variants Improve Transformer](https://arxiv.org/html/2002.05202v1), 2020, equations 5 and 6 and the matched width experiments. Equation (2) is a local parameter count.
[^mixtral]: Albert Q. Jiang et al. [Mixtral of Experts](https://arxiv.org/html/2401.04088v1), 2024, section 2, Figure 1 and routing analysis. Equation (3) restates its selected softmax mixture.
[^v2]: DeepSeek-AI. [DeepSeek-V2](https://arxiv.org/html/2405.04434v1), 2024, section 2.2.
[^v3]: DeepSeek-AI. [DeepSeek-V3 Technical Report](https://arxiv.org/html/2412.19437v1), 2024, section 2.1.2.
[^qwen]: Qwen Team. [Qwen3 Technical Report](https://arxiv.org/html/2505.09388v1), 2025, architecture: dense and expert variants.
[^glm]: Zhipu AI. [GLM-4.5](https://arxiv.org/html/2508.06471v1), 2025, post training specialists and unified ARC policy.

Source: https://latentlit.com/architectures/evolution/ffns-and-experts/index.mdx
