Terabyte Embedding Tables and Compute-Bound Neural Layers Force Meta's DLRM to Split Training

On May 31, 2019, Maxim Naumov and a team of researchers at Facebook AI released an open-source framework called the Deep Learning Recommendation Model (DLRM). It was not marketed as a breakthrough in artificial general intelligence. Instead, it offered a production-grounded blueprint of how a platform serving billions of users actually ranks its feeds.[1]

The architecture laid bare a fundamental hardware contradiction in modern recommendation systems. DLRM handles two entirely different types of data simultaneously. Dense numerical features, like a user's age or the time of day, pass through traditional neural networks.[1][2]

Sparse categorical features, such as a specific user ID or a unique page URL, map to learned vectors in massive databases called embedding tables. These two sides of the model place opposing demands on the silicon that runs them.[2]

The dense neural layers, known as multi-layer perceptrons (MLPs), require intense computational power but consume very little memory. The sparse embedding tables do the exact opposite. They require almost no computation but demand staggering amounts of memory capacity.[3]

In industrial deployments, these embedding tables routinely hold billions of entries. A 2025 Meta research paper noted that a 3.2 trillion parameter model stores over 98 percent of its weights in these tables. That footprint requires more than five terabytes of memory, far exceeding the 80-gigabyte capacity of a modern NVIDIA H100 GPU.[3]

To train a model that physically cannot fit on a single chip, engineers must split the workload across multiple processors. But the standard method for distributing deep learning models breaks down when applied to terabyte-scale tables. DLRM forced the industry to adopt a hybrid approach, splitting memory and compute along entirely different axes.[1][3]

The memory wall of sparse features

Deep learning frameworks like PyTorch and TensorFlow were originally built to optimize convolutional networks for image recognition or transformers for natural language. Those architectures are compute-bound. They rely on dense matrix multiplications where the same relatively small set of weights is applied to massive batches of data.[2]

Recommendation systems operate under a different physical reality. When a user logs into a social network, the system must predict their interaction with millions of potential posts. The inputs are overwhelmingly categorical, consisting of discrete user IDs, post IDs, device types, and location tags.[2]

Neural networks cannot process raw categorical IDs directly. Instead, the system assigns a unique, high-dimensional vector to every single category. These vectors are stored in embedding tables, and as the model trains, it adjusts the numbers to capture the latent relationships between users and content.[1][2]

Because a platform like Meta tracks billions of unique users and trillions of distinct posts, the embedding tables grow to monstrous proportions. "Embeddings contribute the majority of the parameters, with several tables each requiring in excess of multiple GBs of memory," Naumov's team wrote in the original 2019 DLRM paper.[1]

This creates a severe memory bottleneck during training. While the dense MLP layers might contain only a few million parameters, the embedding tables scale into the trillions. The silicon industry has optimized GPUs for raw floating-point operations, but memory capacity has scaled much more slowly, leaving recommendation models starved for space.

Why data parallelism fails for embeddings

When a traditional neural network outgrows a single GPU, engineers typically use data parallelism. In this setup, the entire model is copied onto every available GPU in the cluster. The training data is then chopped into smaller batches, and each GPU processes a different batch simultaneously.[3]

Data parallelism works flawlessly for the dense MLP layers in DLRM. Because the MLPs are small, replicating them across 64 or 128 GPUs consumes a negligible fraction of the cluster's total memory. This allows the system to scale its computational throughput linearly as more chips are added.[3]

But applying data parallelism to the embedding tables is physically impossible. You cannot copy a five-terabyte embedding table onto an 80-gigabyte GPU. Even if the table were small enough to fit, replicating it across every device would waste the cluster's most precious resource, which is high-bandwidth memory.

Instead, DLRM applies model parallelism to the sparse features. The giant embedding tables are sharded, or sliced, into smaller pieces. Each GPU in the cluster holds only a fraction of the total table, meaning a 16-GPU cluster might host exactly one embedding table per chip.[1]

This solves the memory constraint, but it creates a severe logistical problem. During a training step, a single batch of data might require lookups from all 16 embedding tables. Because the tables are physically scattered across different chips, the GPUs must constantly pause their calculations to request vectors from one another.

The butterfly shuffle communication bridge

The collision of model parallelism for embeddings and data parallelism for MLPs creates a structural fault line in the middle of the DLRM architecture. The bottom half of the network is distributed by memory, while the top half is distributed by data. Bridging that gap requires a massive, synchronized data exchange.[1][3]

Once the GPUs have retrieved the necessary embedding vectors from their local shards, they must share them. Every GPU needs the complete set of vectors for its specific batch of data before it can pass those vectors into the dense MLP layers. This requires an exhaustive all-to-all communication pattern.[3]

In PyTorch and NVIDIA's collective communication libraries, this exchange is often called a butterfly shuffle. Every chip in the cluster simultaneously transmits data to every other chip. It is a chaotic, bandwidth-heavy operation that stresses the physical interconnects, like NVLink or InfiniBand, tying the server racks together.[3]

"Combined model and data parallelism is a unique requirement of DLRM as a result of its architecture and large model sizes," the Meta researchers noted. Because standard frameworks did not support this hybrid natively in 2019, the team had to write custom implementations to force the hardware to execute the shuffle efficiently.[1]

The interaction layer immediately follows this shuffle. DLRM computes the dot product between all pairs of embedding vectors and the processed dense features. As the original paper explained, the model "specifically interacts embeddings in a structured way that mimics factorization machines to significantly reduce the dimensionality of the model."[1]

Scaling limits and the reality of production

While the hybrid parallelism strategy solved the immediate bottleneck in 2019, the relentless growth of recommendation models has exposed new limits. As companies push toward tens of trillions of parameters, simply sharding tables across more GPUs triggers diminishing returns. The network itself becomes the primary constraint.

When a cluster scales to thousands of GPUs, the all-to-all butterfly shuffle becomes a crippling liability. The time spent transmitting data across the network begins to eclipse the time spent actually computing the neural network updates. The system becomes communication-bound, leaving expensive silicon sitting idle while waiting for vectors.

To keep the hardware fed, engineers have introduced aggressive optimizations. A 2025 Meta paper proposed a two-dimensional sparse parallelism approach, applying data parallelism on top of model parallelism to restrict the shuffle to smaller sub-groups of GPUs. Other researchers have deployed 4-bit integer quantization to compress the vectors before transmission.

Despite the marketing surrounding generative AI, recommendation systems remain the financial engine of the consumer internet. The DLRM architecture proves that serving personalized feeds is not just a machine learning challenge. It is fundamentally a distributed systems problem, governed by the hard physical limits of memory capacity and network bandwidth.[4]