← Back to list
Hardware & Semiconductor
#시스톨릭 어레이#데이터플로우#행렬곱 가속#AI 반도체#TPU
Last updated · 2026-11-16

Systolic Array

1. Overview

A. Definition

A Systolic Array is a dataflow-based, special-purpose parallel compute structure in which identically structured Processing Elements (PEs) are arranged in a regular grid, and data moves synchronously to neighboring PEs on every clock—much as a heart pushes blood out in beats (systole)—while each PE performs a Multiply-Accumulate (MAC) operation.

The essence of a systolic array is that the flow of data, not control, drives computation. In a von Neumann machine, the processor repeatedly fetches operands from memory and writes results back for each operation, and this memory round-trip becomes the bottleneck for both performance and power. To avoid this bottleneck at its root, a systolic array reads data from memory only once and then passes it through the array so that many PEs reuse the same value. In other words, the design philosophy trades "fewer memory accesses" for "more local, nearest-neighbor communication between compute units," which is why it delivers overwhelming efficiency on regular, repetitive operations—especially matrix multiplication (GEMM) and convolution.

The concept was proposed in 1978–1979 by H.T. Kung of Carnegie Mellon and Charles E. Leiserson of MIT in the context of VLSI design. Their key insight at the time was that "what is expensive on a chip is not the compute units but long wires and memory accesses, so if all communication is restricted to short wires between adjacent cells, a regular structure can be integrated very densely." Decades later this idea was revived as the matrix engine of modern AI accelerators including Google's TPU—a classic example of a computer-architecture textbook topic rising again to the center of industry.

B. Background and Necessity

Three pressures converge behind the renewed attention to systolic arrays. The first is the Memory Wall. Fine-grained semiconductor scaling raised compute speed quickly, but external DRAM bandwidth and latency have not kept pace, so today the real cost of AI computation lies not in multiplication itself but in the data-movement energy of hauling operands from distant memory. A systolic structure, which reuses data loaded once hundreds of times inside the array, dramatically reduces this movement.

The second is the explosion of deep-learning computation. Inference and training of deep neural networks ultimately reduce to repeated large matrix multiplications and convolutions, operations that repeat the same MAC pattern regularly. A workload whose data-access pattern is statically predictable needs no general control logic such as branch prediction or out-of-order execution, so it is a perfect match for a systolic array, where control can be stripped away and compute units and wires laid down regularly.

The third is the limit of power and area efficiency. With Moore's Law slowing and Dennard scaling ended, adding more transistors now also drives up power density, so raising performance effectively requires abandoning generality and specializing circuits for specific operations—a Domain-Specific Architecture (DSA). A systolic array can pack the most MACs into a given area with almost no control overhead, making it a core choice for AI-accelerator designs that aim to maximize operations per watt (TOPS/W).

C. Characteristics

A systolic array's character can be summarized as ① a regular, homogeneous grid of PEs, ② synchronous pipelined data pumping, ③ nearest-neighbor-only local communication, and ④ maximized Operational Intensity through high data reuse. These four axes interlock. The regularity of homogeneous PEs (①) lets data be pushed in like a pipeline (②); restricting all communication to adjacent cells (③) removes long wires and global control, enabling dense integration; and as a result a value loaded once is reused repeatedly (④) as it traverses the array, completing the power-efficiency story.

Conversely, the moment this regularity breaks, all four advantages weaken at once. For small matrices, sparse operations, or irregular access patterns, the array runs half-empty and utilization plummets. Understanding a systolic array therefore amounts to knowing "under which conditions its strengths hold and under which they collapse," and this duality governs both the dataflow choices and the practical design trade-offs discussed below.

2. Structure and Operating Principle

A systolic array's performance depends on how data is streamed into the array to maximize reuse. The structural diagram below shows a typical system configuration in which the host and memory hierarchy cooperate with a 2D PE array.

flowchart LR
  subgraph HOST["Host system"]
    CPU["CPU(control·graph decomposition)"]
    DRAM["External memory(DRAM/HBM)"]
    CPU --- DRAM
  end
  subgraph ACC["Accelerator(on-chip)"]
    WFIFO["Weight FIFO/buffer"]
    ABUF["Activation buffer(SRAM)"]
    ARR["PE array(2D Systolic MAC)"]
    ACC2["Accumulator/partial-sum collection"]
    POST["Activation·normalization·pooling"]
    WFIFO --> ARR
    ABUF --> ARR
    ARR --> ACC2
    ACC2 --> POST
    POST --> ABUF
  end
  DRAM -- "weight stream" --> WFIFO
  DRAM -- "input activation" --> ABUF
  CPU -- "command/descriptor" --> ARR

When the host CPU decomposes the model's compute graph and delegates matrix multiplications and convolutions to the accelerator, weights are supplied through a weight FIFO and input activations through an on-chip SRAM buffer, each feeding a different edge of the PE array. As data passes through the array, partial sums accumulate, and the result leaving the array passes through activation, normalization, and pooling units before returning to the buffer. The decisive point here is the design intent to minimize slow, power-hungry external DRAM access by keeping data loaded once inside the array for as long as possible.

A. Processing Element (PE) and Data Pumping

The PE, the smallest unit composing the array, is an extremely simple circuit made of one multiplier, one adder, and a few registers. On every clock, each PE multiplies the input arriving from the left (or top) by the weight it holds or passes through, then adds it to the partial sum descending from above (or the left) and forwards it to the next PE. This simplicity is precisely the strength of the systolic structure: because the PE is simple, tens of thousands can be densely packed into the same area, and because control is distributed, no global controller or long wires are needed. Google's TPU v1 integrating a 256×256 grid—that is, 65,536 MAC units—onto one chip and performing over sixty thousand multiply-accumulates per cycle at 700 MHz is the representative example of this dense integration.

The term data "pumping" refers to the way data is pushed in one cell at a time, regularly, along the pipeline. An important property is the existence of a load-and-drain interval. For the array's first result to emerge on the opposite side, data must traverse the whole grid, so for an N×N array there is an initial fill latency of about 2N clocks and a similar drain latency at the end. Because of this, when streaming consecutive large matrices the latency is amortized across the throughput and efficiency is maximized, but processing a small matrix as a one-off makes the fill-and-drain overhead relatively large and utilization drops. This is the basis of the practical rule of thumb that "systolic arrays favor large batches and large matrices."

B. Matrix-Multiplication Mapping and Data Flow

Looking at how the matrix product C = A × B is mapped onto a systolic array through a reduced 2×2 example, the weights (B) stay resident in each PE, the inputs (A) flow in row by row with a time skew from the left, and partial sums accumulate downward and exit the bottom. The reason data enters skewed by one cell at a time is to align timing so that each PE meets its correct pair of input and weight on the same clock.

flowchart TB
  A1["A row1 input"] --> P11["PE11(MAC)"]
  A2["A row2 input"] --> P21["PE21(MAC)"]
  P11 --> P12["PE12(MAC)"]
  P21 --> P22["PE22(MAC)"]
  W11["weight b11"] -.resident.-> P11
  W12["weight b12"] -.resident.-> P12
  W21["weight b21"] -.resident.-> P21
  W22["weight b22"] -.resident.-> P22
  P11 --> S1["partial sum downward"]
  P12 --> S2["partial sum downward"]
  P11 --> P21
  P12 --> P22
  P21 --> C1["result c·1"]
  P22 --> C2["result c·2"]

The key gain in this mapping is the data-reuse ratio. One element of a row of A flows rightward and is reused across all PEs in that row, while a weight stays resident in the array and is multiplied repeatedly with every input that flows in. As a result, data of size O(N²) is supplied once to perform close to O(N³) operations, maximizing computation relative to external memory bandwidth (operational intensity). This is precisely the core mechanism by which a systolic array bypasses the memory wall.

3. Dataflow Types

A systolic array's performance and power efficiency depend heavily on the dataflow strategy that decides what is held stationary inside the array and what is streamed through. The data that can be reused are of three kinds—Weight, Input activation, and Partial sum—and depending on which is kept resident inside the PE, the design divides into the four representative types below. The type choice is not a mere implementation detail but a strategic decision about whose data-movement energy to save.

A. Weight Stationary

Weights are preloaded into each PE and input activations are streamed through. Because weights, once loaded, are not reloaded during the entire array computation, this is especially advantageous for inference, where weight reuse is heavy. In inference, the same weight matrix fixed by training repeatedly processes many input batches, so pinning the weights to the array can drive their movement cost essentially to zero.

Google's TPU v1 and AWS Inferentia adopting this approach is a design decision aimed precisely at the economics of such inference workloads. For example, if about sixty thousand weights are loaded into a 256×256 array, the weights are not re-read from memory for the thousands of subsequent inputs flowing in but are reused, greatly easing DRAM-bandwidth pressure.

However, when model parameters exceed the array's capacity, weights must be reloaded block by block, and during loading the array sits idle, unable to compute. Practical designs therefore use a weight FIFO and double buffering to overlap "loading the next block" with "computing the current block," hiding load latency behind computation. Ultimately the success of weight stationary hinges on how far the weight-reload frequency can be lowered and how well loading can be hidden.

B. Output Stationary

Each PE takes charge of one output element (partial sum), continually accumulating it in an internal accumulator while streaming both weights and inputs through. Close to the classic Kung–Leiserson form, it does not move partial sums out of the PE, thereby minimizing partial-sum movement energy and accumulation precision loss.

The reason this approach is advantageous lies in the nature of accumulation. In a long dot product or deep convolution, obtaining a single output value requires summing hundreds to thousands of multiplication results; moving this partial sum to memory or another PE each time accumulates movement energy, and repeated quantization destabilizes precision as well. Output stationary blocks this cost at the source by binding the partial sum inside the PE.

In exchange, weights and inputs must be supplied anew every cycle, so input-side bandwidth demand is relatively large. Output stationary is therefore chosen when on-chip buffer bandwidth is sufficient or when output reuse dominates the compute pattern. The choice between weight stationary and output stationary ultimately reduces to the workload-characteristic question of "is weight movement expensive, or is partial-sum movement expensive?"

C. Input / Row Stationary

This approach either fixes input activations as the reuse axis, or, like the Row Stationary scheme proposed by MIT's Eyeriss, maps an entire row's convolution computation onto one PE to simultaneously and in balance maximize the reuse of weights, inputs, and partial sums.

The insight of row stationary is to go beyond the limitation that "fixing only one kind of data increases the movement of the other two kinds." Because convolution has reuse opportunities in weights, inputs, and partial sums alike, minimizing the total data movement across the PE array and the on-chip memory hierarchy by considering all three axes together is more advantageous from a whole-energy standpoint than optimizing a single axis. Eyeriss reported superior energy efficiency with this strategy compared with fixing only one kind.

The price is greater mapping and scheduling complexity. Which row goes to which PE and how partial sums are gathered becomes intricate, increasing the burden on the compiler and hardware control logic. This is a good example of the essential tension in accelerator design—the trade-off between hardware efficiency and software complexity—and suggests that the dataflow choice is directly tied to the decision to invest in the compiler stack.

D. Type Comparison and Selection Criteria

The differences among the four types ultimately come down to the answer to "which data movement is most expensive in this workload." If weight reuse dominates, as in inference, weight stationary is favorable; if deep accumulation dominates, output stationary; and if the reuse pattern is mixed, row stationary. The table below is a supporting summary; the actual design is decided by profiling the target model's compute distribution.

Dataflow Held stationary Key gain Representative use Weakness
Weight Stationary Weights Minimal weight movement, inference efficiency TPU v1, AWS Inferentia Reload cost for large-model weights
Output Stationary Partial sums Minimal accumulation movement/precision loss Classic systolic, some DSPs Large input bandwidth demand
Input Stationary Input activations Maximized input reuse Certain CNN accelerators Increased weight movement
Row Stationary Row-level composite Balanced reuse of all three Eyeriss Complex mapping/scheduling

4. Comparison with Other Parallel Architectures

A systolic array's identity becomes clear in comparison with GPU and vector (SIMD) architectures. A GPU is a centralized data-supply structure in which thousands of general-purpose cores exchange data through shared memory and register files. Its flexibility is excellent and it handles diverse operations, but because each core frequently accesses registers and memory, its data-movement energy is relatively large and general control logic occupies die area. A systolic array, by contrast, has data flow directly between PEs, omitting the register-file and shared-memory round-trips, so on regular matrix operations it achieves higher MAC density and power efficiency than a GPU at the same area and power.

The fundamental reason for the difference lies in the "data-movement path." In a GPU, operands traverse memory↔registers↔compute units several times, whereas in a systolic array all reuse is finished while the input enters one edge of the array and exits once at the opposite side. This structural difference appears as the gap in power efficiency. That said, this holds only under the premise of "regular operations"; for general workloads with many branches or irregular data dependencies, the GPU's flexibility is overwhelmingly advantageous. The TPU v1 paper (ISCA 2017) indeed reported tens of times the performance/power advantage over contemporary GPUs on inference, but it made clear that this result is confined to the specific regular workload of inference.

Modern AI systems therefore combine the two not as rivals but in a complementary relationship. That is, a heterogeneous-computing structure has become standard, in which CPUs and GPUs handle irregular, control-centric parts while a systolic engine (the MXU/MAC array of TPUs and NPUs) is dedicated to regular large matrix multiplications. A systolic array has established itself not as a "competitor" to general-purpose accelerators but as a "dedicated engine" that peels off and handles the most expensive compute segment.

5. In Depth: Practical Cases and Recent Trends

Systolic arrays have moved beyond an academic concept to the heart of today's AI-semiconductor competition. The most emblematic case is Google's TPU. TPU v1, with a 256×256 weight-stationary systolic array (65,536 MACs), achieved about 92 TOPS at INT8 at a level of about 40 W, dramatically lowering data-center inference cost. Later generations (the TPU v2–v5/v6 line) carry multiple Matrix Multiply Units (MXUs) and, adding BF16 training support and large-scale interconnect (ICI), have scaled to TPU Pods binding thousands of chips, reaching even LLM training. The important point is that the DNA of the core matrix engine is still the systolic structure.

The second axis of the trend is the proliferation of Big Tech's in-house accelerators. AWS loaded systolic-family matrix engines into Inferentia for inference and Trainium for training; Tesla's Dojo for autonomous-driving training, and the MAC arrays of many mobile and edge NPUs (Samsung, Qualcomm, Apple) also effectively adopt systolic or variant dataflows. This reflects an industry-wide strategic shift toward "lowering dependence on general-purpose GPUs and bringing in dedicated silicon tuned to the workload."

The third axis is structural evolution. To ease the low-utilization problem of fixed-size arrays, there is active research into flexible/reconfigurable dataflow, multi-pod designs that split a large array into small tiles to raise utilization even on small matrices, sparsity support that skips the zeros of sparse matrices, and combination with ultra-low-precision numeric formats such as FP8 and INT4. There are also ongoing attempts to graft the systolic concept onto PIM (Processing-in-Memory) and analog matrix computation that place compute units near memory. In short, a 1970s concept is being reinterpreted along the axes of bandwidth, precision, and flexibility to match 2020s AI demand.

6. Considerations and Implications

From a professional-engineer standpoint, a systolic array should be understood not as "the concept itself" but as the design capability to judge its application conditions and trade-offs accurately.

  • Judging workload fit (utilization): A systolic array's benefit is fully realized only on large matrices with high operational intensity. On small, sparse, or irregular workloads utilization plummets, so before adoption one must profile the target model's compute distribution and match the array size and tiling strategy. The misconception that "it is unconditionally advantageous for all AI" must be guarded against.
  • Dataflow–model alignment: As with weight stationary for inference-centric work, output stationary for deep accumulation, and row stationary for composite reuse, matching the dataflow to the dominant reuse axis is what makes power efficiency pay off. Hardware choice must be decided together with the compiler and mapping strategy, and the maturity of the software stack (compilers such as XLA) governs real performance.
  • Precision–accuracy trade-off: Low-precision arithmetic such as INT8, FP8, and INT4 greatly raises throughput and power efficiency but entails model-accuracy loss. One must manage the loss with quantization-aware training (QAT) and calibration, and strike an engineering balance that selects precision within the service-quality criteria (SLA).
  • Heterogeneous-computing integration and dependency: Because a systolic engine operates not alone but as part of a heterogeneous system combined with CPUs and GPUs, bottlenecks across data movement, scheduling, and the memory hierarchy (weight-load latency, activation bandwidth) must be designed together. One must also strategically consider the lock-in risk to a specific vendor's accelerator and compiler, as well as ecosystem maturity.
  • Future outlook and related technologies: While reconfigurable dataflow, sparsity support, ultra-low precision, and PIM grafting push the performance limit upward, HBM, chiplets, and advanced packaging underpin the bandwidth that feeds data to the array. A systolic array, as the common compute foundation of AI semiconductors linked across NPUs, TPUs, on-device AI, and data-center acceleration, is expected to remain a core axis of the power-efficiency competition.

References


In one line: A systolic array is a dataflow structure that pumps data in beats through a grid of homogeneous PEs and processes multiply-accumulate via local communication to maximize data reuse; by choosing strategies such as weight-, output-, and row-stationary it bypasses the memory wall, and it is a computer-architecture classic revived as the power-efficient matrix engine of modern AI accelerators like the TPU and NPU.