Single-Token Autoregressive Decoding Operates at One FLOP per Byte, Trapping Accelerator Tensor Cores Far Below the Roofline Ridge Point

The binding constraint for any processor to reach its advertised speed is that data must arrive at the compute units at least as fast as those cores can multiply it. For the current generation of large language models generating text one word at a time, this condition catastrophically fails.[3]

The architecture of autoregressive decoding forces the system to read the entire model from memory for every single step forward. Because each new word depends on the sequence that came before it, the generation process cannot be parallelized across time.[3]

The result is a structural arithmetic bottleneck that leaves the world's most advanced artificial intelligence accelerators sitting almost entirely idle. When an Nvidia H100 GPU generates a single token, its specialized Tensor Cores spend more than 99 percent of their cycles waiting for data to arrive.[3]

This phenomenon is known as the memory wall. It dictates the economics of artificial intelligence deployment, governs the design of next-generation hardware, and explains why generating a response is fundamentally slower than reading a prompt.[3]

The arithmetic of a single token

To understand the bottleneck, one must trace the exact mathematical operations required to generate a single token. A language model is essentially a massive collection of weight matrices.[3]

In a standard 7-billion-parameter model stored in 16-bit floating-point precision, those weights occupy roughly 14 gigabytes of space. During inference, the GPU must load every single one of those parameters from its High Bandwidth Memory into its on-chip static RAM.[2][3]

For each parameter loaded, the processor performs exactly two mathematical operations: one multiplication and one addition. This ratio defines the workload's arithmetic intensity.[1][3]

Because the system loads two bytes of data to perform two floating-point operations, the arithmetic intensity of single-token decoding is exactly one FLOP per byte. This 1:1 ratio is the inescapable mathematical reality of batch-size-1 autoregressive generation.[3]

The roofline model and the ridge point

Hardware architects visualize this relationship using the roofline model, a framework that plots a processor's performance limits based on arithmetic intensity. The model forms a literal roof shape with a sloped ceiling and a flat top.[1]

The sloped section represents workloads limited by memory bandwidth, where performance scales linearly with data delivery. The flat horizontal line represents the absolute physical limit of the compute cores, where the processor is performing math as fast as its silicon allows.[1]

The intersection of these two lines is called the ridge point. It represents the exact arithmetic intensity required to keep the compute cores fully fed with data and escape the memory bottleneck.[1]

On an Nvidia H100 SXM, the theoretical peak compute for 16-bit operations is 989 teraflops. Its memory subsystem can deliver data at a maximum rate of 3.35 terabytes per second.[3]

Dividing the compute capacity by the memory bandwidth yields a ridge point of 295 FLOPs per byte. To fully utilize an H100, an algorithm must perform 295 operations for every byte it pulls from main memory.[3]

The utilization collapse

Single-token decoding operates at just one FLOP per byte, placing it 295 times below the H100's ridge point. The workload is pinned to the extreme far left of the roofline plot, deep in the memory-bound regime.[1][3]

The physical consequence of this ratio is a massive utilization collapse. The GPU loads weights at 3.35 terabytes per second, but because it does almost nothing with each byte, the compute units finish their work instantly.[3]

At a batch size of one, the peak compute utilization of an H100 is less than 0.4 percent. The processor is effectively acting as a highly expensive memory controller, with its 989-teraflop Tensor Cores starved of work.[3]

This explains why a 14-gigabyte model takes roughly seven milliseconds just to load its weights across a two-terabyte-per-second memory bus, while the actual computation requires only a fraction of a millisecond. The remaining time is pure latency.[3]

The prefill asymmetry

This memory-bound reality only applies to the decoding phase. When a user first submits a prompt, the model processes all the input tokens simultaneously in a phase called the prefill.[3]

During prefill, the GPU still loads the 14 gigabytes of weights once. However, it multiplies those same weights against every token in the prompt at the same time.[3]

If a user submits a 2,048-token prompt, the arithmetic intensity jumps to roughly 2,048 FLOPs per byte. This pushes the workload far past the 295-FLOP ridge point and onto the flat roof of the performance chart.[1][3]

In this compute-bound regime, the Tensor Cores run at maximum capacity, and memory bandwidth is no longer the limiting factor. This structural asymmetry explains why processing a 1,000-word prompt takes a fraction of a second, while generating a 1,000-word response takes several seconds.[3]

How batch size shifts the bottleneck

The primary lever engineers use to increase arithmetic intensity during generation is batching. By processing multiple user requests concurrently, the server can load the model weights once and apply them to several different output tokens simultaneously.[3]

Increasing the batch size from one to 32 raises the arithmetic intensity to 32 FLOPs per byte. While this is still well below the H100's ridge point, it effectively multiplies the throughput of the system by a factor of 32 without requiring additional memory reads.[3]

However, batching introduces its own constraints. Every concurrent sequence requires its own Key-Value cache to store the context of previously generated tokens.[3]

As the batch size and context length grow, the cache consumes an increasing share of the GPU's memory capacity and bandwidth. Researchers have found that at large batch sizes, DRAM bandwidth saturation remains the primary bottleneck, stalling over half of the attention kernel cycles.[3]

Hardware and software interventions

Because raw compute power cannot solve a memory bandwidth problem, hardware manufacturers have shifted their focus to memory speed. The Nvidia H200 maintains the exact same 989-teraflop compute capacity as the H100, but upgrades the memory to the newer HBM3e standard.[2][3]

This new memory standard delivers 4.8 terabytes per second of bandwidth, a 43 percent increase over the H100. For memory-bound decoding workloads, this translates directly into a 43 percent increase in tokens generated per second.[2][3]

On the software side, quantization offers a mathematical workaround. By compressing 16-bit weights into 8-bit or 4-bit integers, engineers halve or quarter the amount of data that must cross the memory bus.[3]

An 8-bit model requires only half the memory bandwidth to load, effectively doubling the generation speed. While this introduces minor precision losses, it is currently the most effective software intervention for escaping the memory wall.[3]

Alternative architectures, such as speculative decoding, attempt to bypass the bottleneck entirely. These systems use a tiny, fast draft model to guess several tokens ahead, then use the large model's idle compute capacity to verify the entire batch in a single parallel step.[3]

The fundamental physics of data movement dictate that memory bandwidth improves much more slowly than compute density. Over the past decade, artificial intelligence accelerators have seen exponential leaps in teraflops, while memory speeds have scaled linearly.[1][3]

Until a paradigm shift in hardware architecture allows memory and compute to physically merge, the single-token decoding process will remain trapped below the ridge point. The industry's most powerful processors will continue to spend their time waiting for data to arrive.[3]