How PagedAttention Partitions GPU Memory to Solve the LLM Context Bottleneck

The inference scheduler sitting between a large language model and its hardware decides exactly how much memory to allocate to a user's prompt. It can partition the GPU's video RAM into rigid, contiguous chunks, or it can divide it into dynamic, non-contiguous pages. It makes this choice the millisecond a new request arrives, determining whether the server can handle ten concurrent users or a hundred.

When a model like Llama 3 or GPT-4 generates a response, it does not recalculate the entire conversation history for every new word. Instead, it stores the mathematical representations of past words in a temporary storage area known as the Key-Value (KV) cache.

As the model generates more text, this cache grows linearly. In 2023, researchers at UC Berkeley identified that the KV cache was consuming up to 30% of total GPU memory during inference, creating a severe bottleneck that limited concurrent user capacity.[1]

The traditional method for managing this memory was highly inefficient. Schedulers would allocate a large, contiguous block of memory for the maximum possible length of a conversation. If a user only asked a short question, the rest of that allocated memory sat empty and unusable by other requests.

This phenomenon, known as internal memory fragmentation, wasted massive amounts of expensive hardware capacity. According to the UC Berkeley team, "existing systems waste 60% to 80% of memory due to fragmentation and over-reservation."[1]

To solve this, developers introduced PagedAttention, an algorithm inspired by how traditional computer operating systems manage RAM. Instead of reserving one massive block, PagedAttention divides the KV cache into small, fixed-size pages.

Each page typically holds the attention keys and values for a specific number of tokens—often 16 or 32. Crucially, these pages do not need to sit next to each other on the physical silicon.[1]

A central block table tracks where each page is located. When the model needs to reference a past token, the scheduler looks up the virtual address in the block table and translates it to the physical location on the GPU.

This mechanism practically eliminates internal fragmentation. Memory is allocated dynamically, one page at a time, only when the model actually generates a new token that requires storage.

The efficiency gains are measurable and immediate. By implementing PagedAttention in the open-source vLLM engine, developers reduced memory waste to under 4%.[1]

Freeing up that wasted memory allows the server to load more concurrent user requests into the same GPU. In benchmark tests on NVIDIA A100 GPUs, vLLM achieved 2x to 4x higher throughput compared to standard Hugging Face Transformers.[1]

NVIDIA has since integrated similar paging techniques into its TensorRT-LLM software stack. In an August 2023 technical breakdown, NVIDIA engineers noted that optimizing the KV cache is critical because "memory bandwidth is the primary bottleneck for LLM inference."[2]

However, the paging mechanism introduces its own computational overhead. The scheduler must constantly update the block table and perform address translations during the highly latency-sensitive generation phase.

What remains unproven is how well PagedAttention scales as context windows expand from 128,000 tokens to over a million. At those extremes, the block table itself becomes a massive data structure that requires optimization.

The next generation of inference engines will have to decide how to compress the cache itself. Until hardware bandwidth catches up to model size, the software scheduler's ability to pack memory blocks tightly dictates the commercial viability of deploying frontier AI.