How Systolic Arrays Pass Data Through AI Chips to Execute Matrix Multiplication

Chip designers face an intractable trade-off when building hardware for artificial intelligence. Proponents of general-purpose compute argue that algorithms evolve too quickly to lock them into silicon; they advocate for architectures that retain enough flexibility to handle whatever mathematical structures researchers invent next. Conversely, domain-specific architecture engineers maintain that flexibility is a waste of power and time. They argue that because deep learning relies almost entirely on one specific operation—multiplying massive grids of numbers together—hardware should abandon programmability in favor of hardcoded, rigid pathways optimized exclusively for that single task.[2][4]

The debate centers on a fundamental limitation of traditional computer design known as the von Neumann bottleneck. In a standard processor, executing a calculation requires fetching a number from main memory, bringing it to the arithmetic logic unit, performing the math, and sending the result back. "The problem is that memory access is slow and consumes a lot of energy," explains H.T. Kung, who co-authored the foundational 1982 paper on systolic architectures. When a neural network requires billions of multiply-accumulate operations, the chip spends more time and power moving data back and forth than it does actually computing.[1]

To solve this, engineers turned to a concept Kung named after the human heart. In a systolic array, data is pumped through a grid of processing elements in a rhythmic, continuous flow. Instead of sending a result back to memory, each element performs a single arithmetic operation and immediately passes the output to its neighbor in the next clock cycle. The data ripples through the silicon grid, being reused multiple times by different elements before the final result exits the array.[1][3]

This theoretical architecture became the foundation of modern AI infrastructure in 2017, when Google revealed the details of its first Tensor Processing Unit. The original TPU featured a massive systolic array containing 65,536 processing elements arranged in a 256-by-256 grid. By keeping the data moving through this grid without returning to dynamic random-access memory, Google achieved a massive leap in efficiency. The company reported that the TPU delivered 15 to 30 times higher performance and 30 to 80 times higher performance-per-watt than contemporary CPUs and GPUs for inference tasks.[2]

The specific way data moves through these grids determines their efficiency. The Google TPU utilizes a weight-stationary data flow. In this setup, the parameters of the neural network—the weights—are loaded into the processing elements and held in place. The input data flows across the grid from left to right, while partial sums cascade from top to bottom. As Telesens researchers detail in their 2018 architectural breakdown, this minimizes the energy spent moving weights, which are often the largest matrices in the calculation.[3][6]

While Google built dedicated application-specific integrated circuits, Nvidia integrated systolic principles directly into its flexible graphics processing units. Nvidia's Hopper architecture, introduced in 2022, relies heavily on fourth-generation Tensor Cores. These cores are essentially miniature systolic arrays embedded within the broader GPU structure. They allow the chip to execute dense matrix multiplication with the efficiency of a systolic array while retaining the surrounding programmable logic needed to handle non-matrix operations.[4]

Other companies have pushed the data flow concept in different directions. Groq, founded by former Google TPU engineers, developed the Language Processing Unit. According to the company's 2025 technical documentation, the LPU abandons the traditional systolic array's reliance on complex control logic and instead uses a deterministic architecture where the exact path and timing of every piece of data is known by the compiler before the program even runs. This approach aims to eliminate the latency spikes that occur when traditional chips have to wait for memory fetches.[5]

The efficiency of systolic arrays is now moving beyond massive data centers. In a March 2026 analysis, robotics researcher Avik De highlighted how systolic arrays are being adapted for edge computing and general robotics. Because these chips can execute complex matrix math on a strict power budget, they allow autonomous systems to run sophisticated perception models locally without relying on a constant connection to cloud servers.[7]

The rigid nature of systolic arrays remains their primary vulnerability. If an AI model requires operations other than dense matrix multiplication—such as the sparse matrices used in Mixture of Experts architectures, where many values are zero—the systolic array wastes clock cycles multiplying by zero. The hardware cannot easily skip these operations because the rhythmic flow of data must be maintained across the entire grid.[3]

Hardware designers are currently attempting to build arrays that can dynamically route around zeros or adapt to different data flows on the fly. The success of these hybrid designs will determine whether the next generation of artificial intelligence relies on increasingly specialized silicon, or if the industry will be forced to invent an entirely new architecture to keep pace with algorithmic shifts.[3]