Mixture of Experts: why a model with 671 billion parameters uses only 37

In the model cards of modern open-source models you come across a strange pair of numbers: DeepSeek-V3 has “671 billion parameters, 37 billion active”, gpt-oss-120b “117 billion, 5.1 billion active”. How can a model not use most of its parameters — and what does it have them for, then? The answer is the Mixture of Experts (MoE) architecture, today the de facto standard for large models. In this article we break down how an MoE layer works, how the router decides which expert gets which token, why experts don’t specialise the way you would expect — and above all what sparsity means in practice: why an MoE model needs memory according to its total parameters but runs at the speed of the active ones, and how to exploit that when self-hosting.

What total and active parameters mean

Let’s start with what that pair of numbers says exactly. Total parameters are all the weights the model has stored — what you download from Hugging Face and what has to live in memory. Active parameters are those that actually take part in computing one specific token. For a classic (dense) model both numbers are the same: Llama 3.1 70B pushes every token through all 70 billion weights. For an MoE model they differ, and dramatically — DeepSeek-V3 uses only ~5.5% of its weights for each token, gpt-oss-120b as little as ~4.4%.

Important: which part of the weights is used is decided at runtime, separately for every token and every layer. So it is not the case that some slice of the model is selected at startup and the rest lies idle — while generating a single answer, a large proportion of all experts take turns, just never all at once.

Where MoE changes the Transformer

To understand what exactly is being “thinned out”, we need to know where the weights are. Every Transformer block has two main parts: attention (mixes information between tokens) and the feed-forward network (FFN, processes each token independently — two linear layers with an activation between them). It is little known that in a typical LLM the FFN layers make up roughly two thirds of all parameters; in gpt-oss-120b the MoE/FFN parts account for 114.7 of the total 116.8 billion, i.e. 98% (its attention has only ~1 billion thanks to an economical design).

And this is exactly where MoE steps in: it replaces the single FFN in each block with a set of N independent copies (“experts”) and a small router that selects the k most suitable ones for each token. Attention remains unchanged — which is why MoE saves neither KV cache nor work with the context; it saves computation in the FFN. The outputs of the selected experts are summed, weighted by the router’s scores, and processing continues as in an ordinary Transformer.

The router: how a token chooses experts

The router is almost laughably simple: a single linear layer. It takes the token’s vector (e.g. 4,096 numbers) and multiplies it by an N × 4,096 matrix, where N is the number of experts. The result is N scores; from these the top-k are selected, i.e. the k experts with the highest scores. Their scores are normalised (by softmax or sigmoid gating) and used as weights when summing the outputs. The whole mechanism in pseudo-PyTorch:

import torch
import torch.nn.functional as F

class MoELayer(torch.nn.Module):
    def __init__(self, dim: int, n_experts: int, top_k: int):
        super().__init__()
        self.router = torch.nn.Linear(dim, n_experts, bias=False)
        self.experts = torch.nn.ModuleList(
            [FeedForward(dim) for _ in range(n_experts)]
        )
        self.top_k = top_k

    def forward(self, x):                      # x: [tokeny, dim]
        skore = self.router(x)                 # [tokeny, n_experts]
        vahy, idx = torch.topk(skore, self.top_k, dim=-1)
        vahy = F.softmax(vahy, dim=-1)         # normalizace jen top-k

        vystup = torch.zeros_like(x)
        for i, expert in enumerate(self.experts):
            maska = (idx == i).any(dim=-1)     # ktere tokeny mne chteji
            if maska.any():
                w = vahy[maska][idx[maska] == i].unsqueeze(-1)
                vystup[maska] += w * expert(x[maska])   # pocita se JEN tady
        return vystup

The key detail is in the comment on the penultimate line: an expert computes only over the tokens the router assigned to it. This is where the whole saving comes from. And because the router is part of the model, its weights are learned together with everything else — nobody tells it how to split tokens; the gradient pushes it towards a split that minimises the error.

The problem learned routing brings: load balancing

A learned router has an insidious degenerative loop: an expert that is a hair better at the start of training receives more tokens, so it learns faster, so it receives even more tokens… and you end up with a model where three experts do everything and the rest is dead weight. The classic solution is an auxiliary loss — an added loss function that penalises uneven utilisation (this is how Switch Transformer and Mixtral do it). But it has a price: you are optimising something other than prediction quality. DeepSeek-V3 therefore introduced auxiliary-loss-free balancing: a small bias is added to each expert’s score, dynamically pushed negative for overloaded experts and positive for underused ones — it evens out the traffic without contaminating the gradient. A detail, but it is exactly this type of detail that decides whether MoE training converges.

How many experts to pick and how to score them: top-k, softmax, sigmoid

Two seemingly minor decisions in router design have surprisingly large consequences. The first is the value of k. The original Shazeer design (2017) used top-k with k ≥ 2, arguing that the router needs to compare at least two experts to have something to learn from. Switch Transformer then showed that top-1 works too — and is cheaper in both compute and communication. Mixtral returned to top-2, fine-grained architectures went to top-8. Interesting is the return to top-1 in Llama 4: Maverick sends every token to a shared expert plus a single routed expert out of 128. It makes sense precisely in combination — the shared expert guarantees that every token gets some “basic processing”, so the routed expert can afford to be narrowly specialised and one is enough. The reward is minimal compute per token (17B out of 400B) and half the volume of communication between GPUs compared with top-2.

The second decision is the scoring function. Classically the router normalises scores with softmax — but that ties all experts together: the scores must sum to 1, so when one expert strengthens, it necessarily takes from the others. With 256 experts, moreover, softmax pushes most scores towards zero and the differences between candidates at the top-k boundary are in the noise. DeepSeek-V3 therefore switched to sigmoid: every expert gets an independent score between 0 and 1 (only the selected top-k set is normalised), so experts don’t “compete for a budget” and the gradient doesn’t push towards suppressing competitors. It sounds like cosmetics, but theoretical analyses suggest that sigmoid gating mitigates exactly the winner-take-all dynamics we described under load balancing.

Token choice vs. expert choice: who picks whom

Everything described above is so-called token choice routing: each token independently picks its top-k experts. The whole balancing problem follows from this — tokens don’t coordinate on anything, so they may all pile onto one expert. In 2022 Google proposed an elegant inversion: expert choice. Instead of tokens picking experts, each expert picks the top-c tokens from the batch it wants to process. Load balancing is thereby solved by construction — each expert processes exactly c tokens, no auxiliary loss is needed. An extra bonus: the number of experts per token is variable, so “hard” tokens (which more experts want) get more compute and trivial tokens less. In experiments this accelerated training convergence more than 2×.

So why doesn’t everyone use it? Expert choice has a fundamental flaw for generative models: the expert picks tokens across the whole batch at once, so the decision about one token depends on the other tokens — including those that don’t yet exist in autoregressive generation or belong to other users. When decoding token by token the mechanism collapses (there is nothing to pick top-c from) and during training information leaks from the future. That is why you find expert choice in encoders and research papers, while production LLMs stay with token choice and handle balancing with the auxiliary mechanisms from the previous section.

Shared expert and fine-grained experts

Newer MoE models add two improvements. A shared expert is an expert that receives every token regardless of the router — it captures “common” knowledge (grammar, general patterns), so the routed experts can differentiate. DeepSeek-V3 has 1 shared + 256 routed, Kimi K2 similarly; Qwen3 and gpt-oss do without one. Fine-grained experts then means making 256 small experts instead of 8 large ones and selecting top-8 instead of top-2: by combining small experts the number of possible “line-ups” grows (for DeepSeek it is 256 choose 8 ≈ 10¹⁴ combinations per layer), so the model can compose knowledge much more flexibly.

What experts do (not) specialise in

Intuition says one expert will be “for medicine” and another “for Python”. Analyses — ever since the Mixtral paper — show something else: specialisation is mostly syntactic and token-based, not topical. An expert specialises in punctuation, numbers, indentation in code, English verbs — because routing is decided per token according to its vector, not according to the document’s topic. It is a good reminder that MoE is not a “committee of doctors” but a statistical trick to increase capacity: the name Mixture of Experts dates from 1991 and is rather a marketing relic. Incidentally — the original paper Adaptive Mixtures of Local Experts is also signed by Geoffrey Hinton, whom you know from our article on seven personalities of AI; the idea waited over 25 years for its time before Shazeer et al. (2017) revived it inside neural networks and Mixtral (2023) popularised it in open source.

Training MoE: why it is more fragile than a dense model

So far we have described MoE from the perspective of a finished model. Training, however, is a chapter of its own — and most of the “odd” constructions we have seen (auxiliary loss, DeepSeek’s bias, shared expert) exist precisely because of training pathologies. Let’s go through the main ones, because they explain why MoE long remained the domain of a few labs with giant budgets.

Capacity factor: when tokens don’t fit

GPUs are bad at computing with tensors whose shape isn’t known in advance. But the router distributes tokens dynamically — one time an expert gets 50 tokens, the next time 500. Classic implementations (from GShard to Switch Transformer) solve this by giving each expert a fixed buffer:

# kapacita experta = prumer na experta * rezerva
expert_capacity = (tokens_per_batch / num_experts) * capacity_factor
# capacity_factor byva 1.0 az 1.5

And here comes the unpleasant surprise: what about tokens that don’t fit into an overloaded expert’s buffer? They are dropped. Literally — the token doesn’t pass through the expert and continues to the next layer only via the residual connection, unprocessed (a technique known as token dropping). Switch Transformer states that with well-functioning load balancing under 1% of tokens are dropped, which training survives. A higher capacity factor limits dropping, but you pay for it with memory and wasted compute: buffers are padded to a fixed size even when half empty.

That this trade-off is unnecessary was shown only by MegaBlocks (2022): it reformulated the MoE layer computation as block-sparse matrix multiplication, where each expert gets exactly as many rows as the router assigned to it. No fixed buffers, no dropping, no padding — so-called dropless MoE — and yet training tens of percent faster, because block-sparse kernels run on modern GPUs almost as efficiently as dense multiplication. Today it is the standard: the Databricks training stack, for example, builds on MegaBlocks, and the same idea is used by the fused MoE kernels in vLLM, which we’ll get to shortly. The capacity factor hasn’t disappeared from view, though — you’ll meet it in the configuration files of many frameworks and older models, so now you know what it means.

Router z-loss: a battle with numerics

The second pathology is more treacherous. While scaling MoE, the Google team ran into training that ran fine for hundreds of thousands of steps and then suddenly collapsed into NaN. They described the culprit in the ST-MoE paper: the router logits gradually grow (nothing constrains them — softmax is shift-invariant, so the model has no reason to keep them small), and because training is in bfloat16, large values in the softmax exponential cause rounding errors. And the router is the only place in the Transformer where a tiny numerical error discretely flips a decision — the token goes to a different expert, the gradient jumps, the instability cascades. The solution is again almost laughably simple, the so-called router z-loss: the square of the log-sum-exp of the logits is added to the total loss, which keeps the logits near zero and the softmax in a numerically safe zone. Since then z-loss has been a standard part of MoE recipes.

Why MoE tolerates fine-tuning poorly

Switch Transformer and ST-MoE after it already document that MoE models behave worse than dense models when fine-tuned on small datasets: they have an enormous number of parameters (capacity) relative to the amount of fine-tuning data, so they easily overfit — ST-MoE describes cases where a sparse model reached 100% accuracy on the training set and yet lagged behind the dense baseline on validation. The router itself is fragile too: the distribution of tokens in fine-tuning data differs from pretraining, balancing goes astray and expert specialisation can fall apart. Recommended antidotes from the literature: markedly higher dropout inside the experts (Switch uses 0.4 in experts vs. 0.1 elsewhere), a smaller learning rate, or fine-tuning only the non-expert parameters. For practice this yields advice we’ll repeat in the conclusion: if you plan your own fine-tuning, reach for a dense model, or for LoRA adapters that leave the expert (and router) weights alone.

Upcycling: MoE from a finished dense model

Training MoE from scratch is expensive and unstable. There is a shortcut — sparse upcycling (Komatsuzaki et al., 2022): take a trained dense model, copy its FFN layer N times as the initial state of the experts, add a randomly initialised router and continue training. The experts start identical and differentiate only as the router starts sending them different tokens. You save most of the pretraining cost — Qwen1.5-MoE (explicitly upcycled from the dense Qwen-1.8B to 14.3B total / 2.7B active parameters) reports ~25% of the training cost compared with a comparable dense 7B model. With Mixtral 8x7B the community noticed that the architecture exactly copies Mistral 7B and the expert weights were strikingly similar — Mistral never confirmed it, but upcycling from Mistral 7B is the generally accepted hypothesis. The limit of this shortcut is described by later research: upcycled experts stay in the vicinity of a common starting point, so the full diversity of experts trained from scratch is not reached — with a large enough budget, training from scratch wins. That is why neither DeepSeek-V3 nor Kimi K2 use upcycling.

Why vendors use MoE: the economics of scaling

From scaling laws we know that model quality grows with the number of parameters and with the amount of compute. MoE decouples these two quantities: you can increase capacity (total parameters, i.e. how much the model “knows”) without compute per token (active parameters, i.e. how long and at what cost it computes) growing proportionally. DeepSeek-V3 was trained in ~2.8 million GPU hours — a fraction of what a dense model of comparable quality would cost. And at inference you pay (in time and money) for 37B of compute even though the model “knows” 671B. That is why practically all large models use MoE today — and why API providers can offer giant models at the prices of small ones.

Note the trend over time: from 8 large experts with top-2 (Mixtral) to hundreds of small ones with top-4 to top-8 and extreme sparsity — Kimi K2 activates 3.2% of the model, gpt-oss-120b 4.4%.

How many dense parameters an MoE actually corresponds to

The natural question is: is gpt-oss-120b a “120” or a “5”? The academic answer was given by Clark et al. (DeepMind, 2022) in their work on scaling laws for routed models: MoE quality can be described by a power law in two independent variables (compute per token and number of experts) and from it one can derive the Effective Parameter Count — the size of a dense model that would reach the same loss. Two findings are worth remembering. First, the benefit of additional experts falls off logarithmically — doubling the number of experts brings nowhere near what doubling the active parameters does; that is why nobody builds a model with 10,000 experts. Second, the advantage of routing shrinks as the base model grows, although the extrapolation of exactly where it vanishes is, by the authors’ own admission, very sensitive to the measured coefficients — and practice so far shows that at today’s sizes MoE still pays off handsomely.

The community uses a simpler empirical rule: the geometric mean of total and active parameters, i.e. √(total × active). For gpt-oss-120b this gives √(117 × 5.1) ≈ 24B — so expect roughly the quality of a dense ~24B model at the compute cost of a 5B model. It is a heuristic as old as Mixtral, no law of nature; what is interesting, though, is that it was independently supported by newer empirical fine-tuning research: Qwen3-30B-A3B behaves during fine-tuning like a dense model of ~8.5B parameters, which is almost exactly the geometric mean (10.1B) and far from both extremes. For a quick estimate when choosing a model it is the best single-number approximation we have — just don’t forget that the real ranking of models is always decided by benchmarks on your task.

VRAM, self-hosting and distributed serving

And now practically, following on from our article on hosting open-source LLMs. For self-hosting two rules apply that pull in opposite directions:

• Memory is governed by total parameters. The router may pick any expert for the next token, so all weights must be ready. Qwen3-235B-A22B in 4-bit quantisation takes ~125 GB, DeepSeek-V3 almost 400 GB — regardless of how many parameters are “active”.
• Speed is governed by active parameters. Generation is limited by how many weights have to be read and multiplied per token. Qwen3-235B-A22B therefore generates at roughly the speed of a 22B model, gpt-oss-120b even at the speed of a 5B model — an order of magnitude faster than their size would suggest.

From this follows on what hardware MoE makes sense: wherever there is relatively plenty of memory but little compute. Typically a Mac with unified memory (an M-chip with 128 GB RAM handles Qwen3-235B-A22B, which on GPUs would require two H100s — and generates briskly because it reads only 22B per token) or a server with lots of RAM and a single GPU. Conversely, a pure GPU deployment with small VRAM is the least favourable for MoE: you pay for expensive memory for weights that are asleep most of the time.

How data centres handle it: expert parallelism and all-to-all

When a model doesn’t fit on one GPU, the dense world reaches for tensor parallelism (TP): each weight matrix is sliced among GPUs and after each layer the partial results are merged with an all-reduce operation.

With MoE this works too, but it is wasteful — you would slice into eight pieces even experts that don’t compute at all for a given token, and the small matrices of fine-grained experts become inefficient when sliced. More natural is expert parallelism (EP), first deployed in GShard: don’t divide experts among GPUs in pieces, but as whole units. With 256 experts and 8 GPUs each card has its own 32 experts; after the router’s decision the token is physically sent to the card where its expert lives, computed there and the result is sent back.

The price is a new communication pattern: all-to-all. Every GPU holds tokens for the experts of all other GPUs, so in every MoE layer all cards exchange data with all others — twice (dispatching tokens there, combining outputs back). This is an order of magnitude more demanding on the network than all-reduce, and in large deployments the all-to-all latency is the dominant cost; that is why DeepSpeed-MoE introduced hierarchical all-to-all (first within a node over NVLink, then between nodes) and why DeepSeek wrote its own communication library DeepEP for running V3. And there is one more consequence we already know: load balancing. The auxiliary loss from the first half of the article was not just about “dead experts” — with expert parallelism, unbalanced traffic means one GPU computes while seven wait. Balancing experts is therefore also a systems problem, not just a statistical one.

If you serve MoE via vLLM, you get most of this ready-made. MoE layers are computed by so-called fused MoE kernels (Triton): instead of a loop over experts, as in our pseudo-example, tokens are grouped by their assigned expert and all expert multiplications run in a single kernel launch over a “stacked” weight matrix — conceptually the same trick as the MegaBlocks block-sparse formulation. On a single node experts are by default just replicated within TP; real expert parallelism is enabled with the --enable-expert-parallel switch, and for multi-node deployments vLLM also offers a DeepEP backend and EPLB (Expert Parallel Load Balancer), which moves replicas of overloaded experts at runtime. For a single GPU at home you don’t need to deal with any of this — a different game is played there.

A trick for home hardware: experts into RAM

Sparsity enables one more manoeuvre that doesn’t work with dense models: splitting the model between GPU and CPU by role. Keep attention, embeddings and the KV cache (small, used constantly) on the GPU; move the expert weights (huge, each used only occasionally) to system RAM. llama.cpp has a switch directly for this:

# gpt-oss-120b na jedne RTX 4090 (24 GB): experty do RAM, zbytek na GPU
$ llama-server -m gpt-oss-120b-mxfp4.gguf --n-gpu-layers 999 --n-cpu-moe 30

# --cpu-moe        = vsechny experty na CPU
# --n-cpu-moe N    = experty prvnich N vrstev na CPU, zbytek na GPU

Why does it work? Per token only the weights of the top-k experts are read from RAM (for gpt-oss-120b ~3 GB instead of 60), which slower RAM manages in reasonable time — and the computationally heavy rest runs on the GPU. The community routinely runs 120B models this way on a single consumer card at around 20–30 tokens/s. With a dense 120B model the same machine would generate a handful of tokens per minute. A practical tuning procedure: start with a high --n-cpu-moe (even the number of all layers), watch free VRAM and gradually lower the number — every layer of experts that fits on the GPU adds speed.

The --n-cpu-moe switch is just a convenient shortcut (it was added to llama.cpp only in PR #15077, August 2025) over the more general -ot / --override-tensor mechanism, which places tensors according to a regular expression over their names. The weights of routed experts have names like blk.7.ffn_up_exps.weight in GGUF files, so:

# rucne totez, co --cpu-moe: vsechny expertni tenzory na CPU
$ llama-server -m model.gguf -ngl 999 -ot "\.ffn_(up|down|gate)_exps=CPU"

# experty vrstev 0-19 na CPU, zbytek nechat na GPU
$ llama-server -m model.gguf -ngl 999 -ot "blk\.(1?[0-9])\.ffn_.*_exps=CPU"

One treacherous detail: models with a shared expert have, besides the _exps tensors (routed), also _shexp tensors (shared). The shared expert computes every token, so it belongs on the GPU — a regex targeting only _exps correctly leaves it there, but if VRAM overflows and you start offloading more aggressively, this is exactly the tensor that should go to the CPU last. The same logic applies generally: keep on the GPU everything that is used for every token (attention, embeddings, KV cache, shared expert), move to RAM only what is used rarely.

Prefill vs. decode: sparsity behaves in two different ways

LLM inference has two phases with completely different characteristics. Prefill is processing the entire input prompt; it computes hundreds to thousands of tokens at once and is usually limited by compute, not memory.

Because each token chooses experts independently, with a thousand tokens almost certainly all experts are activated. The weights are thus read in full anyway. The MoE saving in prefill therefore doesn’t come from less memory reading but from saved FLOPs: each token passes through only the top-k experts.

Decode is the subsequent generation one token at a time. It works with small matrices, is limited by memory bandwidth and at batch 1 really only the weights of the top-k experts are read per token. This is exactly what the experts-in-RAM trick rests on: slow memory is troubled only by small reads in decode, while the compute-hungry prefill runs on the GPU.

From this also follows a practical consequence for configurations with --n-cpu-moe: generation speed tends to be surprisingly good, but prefill of long prompts is noticeably slower than with a full GPU run — in prefill you need all experts, and they sit in slow RAM next to a slow CPU. If your workload is “long document, short answer” (RAG, summarisation), expert offload hurts you more than generation-focused tokens/s benchmarks show.

Three warnings before you get excited

• The KV cache doesn’t shrink. MoE thins out only the FFN; attention and its memory for context remain. Long contexts cost the same for MoE as for a dense model of the same dimension (some models address this with other techniques — DeepSeek has MLA, gpt-oss a sliding window on half of its layers).
• Batching changes the maths. The saving “only top-k experts are read” holds for a single request. With a large batch (a server with dozens of concurrent users), traffic for most experts meets in every layer, so almost the whole model is read anyway — the advantage shifts from memory bandwidth to saved FLOPs. For home use (batch 1), though, it is a clear win.
• Active parameters are not quality. A model with 22B active out of 235B total is not “just a 22B model” — its knowledge capacity corresponds more to the whole. But don’t trust it like a dense 235B model either; reality tends to sit at the geometric mean (~72B), as we discussed above. Compare via benchmarks, not parameters.

What to do about it in practice

• Read both numbers. Size memory by total parameters (plus KV cache), estimate speed by active ones. A “235B model” can be operationally cheaper than a 70B dense model.
• For local running on a single GPU, small MoEs are today the best choice (gpt-oss-20b within 16 GB, gpt-oss-120b with --n-cpu-moe on a 24GB card) — the speed of a few-billion-parameter dense model, quality a class higher.
• A Mac with unified memory is ideal for MoE: the large shared memory holds the whole, the low compute demand per token suits the weaker GPU part.
• Quantise the experts boldly — they tolerate 4 bits very well (gpt-oss is even trained in MXFP4); attention and embeddings are more sensitive, keep those at higher precision (most GGUF quantisations do this by themselves).
• On a server with a large batch, expect the memory advantage of sparsity to weaken — size by FLOPs and bandwidth, not by anecdotes from home setups. For vLLM, the configuration from the hosting article applies.
• Fine-tuning MoE is a harder discipline (overfitting of large capacity on small data, breaking the router’s balancing — see the training section) — if you plan fine-tuning, start with a dense model or use LoRA only on the attention layers.
• With expert offload to CPU, think about prefill. Generation will be brisk, but processing a long prompt needs all experts from slow RAM. For RAG and summarisation of long documents measure time-to-first-token, not just tokens/s.

About the author

Zdeněk Vais

Founder · Senior Principal Software Engineer, Oracle

Zdeněk has been programming since high school. Since then he has gone from programmer through team leader to architect, building large financial systems, scientific applications and IoT solutions for aircraft along the way. He currently works as a senior principal software engineer at Oracle, where he builds the cloud for media services.

Meet our team

← Back to the blog