How Mixture of Experts Routing Networks Decouple LLM Parameter Count From Compute Cost

On May 7, 2024, the research team at DeepSeek published a technical report detailing a 236-billion parameter language model that required only a fraction of the expected computing power to run. The model, DeepSeek-V2, achieved this by activating just 21 billion parameters for any given token, leaving over 90 percent of its neural network dormant during each calculation.[1]

This release cemented a structural shift in how frontier artificial intelligence is built. Instead of forcing every piece of data through every artificial neuron—a method known as dense modeling—developers are increasingly relying on a routing mechanism called Mixture of Experts (MoE).[5]

The fundamental constraint of dense models lies in their mathematical rigidity. In a dense architecture, generating a single word requires a matrix multiplication operation involving every single parameter in the model.[5]

As models grow to improve their reasoning capabilities, this dense compute requirement scales linearly. A 400-billion parameter dense model requires roughly ten times the memory bandwidth and processing power per token as a 40-billion parameter model, creating a hard ceiling on how large models can grow before they become too expensive to operate.[5]

Mixture of Experts bypasses this bottleneck by decoupling the model's total knowledge capacity from its active computational cost. It achieves this by replacing the standard, monolithic layers of the neural network with specialized sub-networks.[5]

The mechanism relies on two distinct components: the "experts" and the "router." Inside an MoE layer, the standard feed-forward neural network is divided into multiple independent partitions.[5]

"Mixtral is a sparse mixture-of-experts network," researchers from Mistral AI noted in their January 2024 paper detailing the Mixtral 8x7B model. "It is a decoder-only model where the feedforward block picks from a set of 8 distinct groups of parameters."[2]

The router acts as a high-speed traffic controller. When a token—a mathematical representation of a word or concept—arrives at an MoE layer, the router evaluates it and calculates a probability distribution across all available experts.[5]

It then selects only the top one or two experts best suited to process that specific token, ignoring the rest. The token is processed by those specific experts, and their outputs are combined and passed to the next layer.[5]

For example, if the input token relates to Python code, the router might send it to Expert 3 and Expert 7. If the next token relates to French grammar, it might be routed to Expert 1 and Expert 5.[5]

Because only a subset of experts is activated, the computational cost remains low. Mixtral 8x7B contains 47 billion total parameters, but because it only routes tokens to two of its eight experts at any time, it runs at the speed and active cost of a 13-billion parameter model.[2]

This sparsity is what allows models to scale efficiently. The total parameter count—which dictates how much knowledge the model can store—can be expanded massively, while the active parameter count—which dictates the inference speed—remains constrained.[5]

The foundational research for routing data to specialized sub-networks dates back to 1991, but its application to modern Large Language Models was popularized by Google's Switch Transformer paper, published on January 11, 2021.[3]

"In this work we propose the Switch Transformer, an architecture that decouples compute from parameters," the Google researchers wrote, demonstrating that a trillion-parameter model could be trained efficiently by routing tokens to just one of 2,048 experts.[3]

However, implementing MoE introduces significant engineering challenges, primarily around memory management. While the active compute is low, the entire model must still reside in the GPU's Random Access Memory (VRAM).[5]

A 236-billion parameter model like DeepSeek-V2 still requires hundreds of gigabytes of VRAM just to hold the dormant experts, meaning it cannot run on a single consumer-grade graphics card despite its low active compute requirements.[1]

Furthermore, routers can develop a "representation collapse" during training. This occurs when the router learns to send all tokens to the same few experts, leaving the rest of the network untrained and useless.[5]

To prevent this bottleneck, engineers introduce an auxiliary load-balancing loss during the training phase. This mathematical penalty forces the router to distribute tokens relatively evenly across all available experts, ensuring the entire network learns.[2]

The exact specialization of these experts is rarely human-readable. An expert does not explicitly become the "math expert" or the "history expert" in a way a human programmer would design.[5]

Instead, they specialize in abstract, high-dimensional vector patterns. One expert might activate for punctuation and conjunctions, while another activates for specific syntactic structures across multiple languages.[5]

The trend toward extreme sparsity is accelerating. On March 27, 2024, Databricks released DBRX, a 132-billion parameter model that activates 36 billion parameters per token, utilizing a highly granular architecture of 16 experts where four are chosen per token.[4]

By comparing the architectures of Mixtral, DBRX, and DeepSeek-V2, a clear trajectory emerges: as total model size increases, the percentage of active parameters drops sharply. Future frontier models will likely feature trillions of parameters divided into thousands of highly granular experts, activating less than one percent of their network per token to maintain inference speeds.[5]