IM Engineer Study Notes
← Back to list
AI & Data
#SSM#Mamba#선택적 상태공간#장문맥#Transformer 대안
Last updated · 2026-10-22

State Space Models (SSM) and the Mamba Architecture

1. Overview

A. Definition and Background

A State Space Model (SSM) is a family of deep neural networks that borrows the linear state-space equations of control theory (x'(t)=Ax(t)+Bu(t), y(t)=Cx(t)) for sequence modeling, compressing past information into a fixed-size hidden state and processing an input sequence in linear time (O(n)). Mamba is the landmark 2023 architecture that adds an input-dependent selective mechanism to achieve performance rivaling the Transformer at linear complexity.

The history of sequence modeling is a long struggle over the question "how do we remember the past?" Recurrent neural networks (RNNs) compress the past by updating a hidden state one step at a time, but gradient vanishing makes long-range dependencies hard to learn, and sequential computation cannot be parallelized, so training is slow. The Transformer, introduced in 2017, used self-attention to directly connect every pair of tokens, solving long-range dependency and parallel training at once, and has since become the de facto standard for large language models (LLMs).

The Transformer's self-attention, however, carries a fundamental limitation: compute and memory grow as O(n²) in sequence length n. As documents grow longer — as the context window swells to tens or hundreds of thousands of tokens — cost explodes quadratically, and at inference the KV cache, which stores the keys and values of every past token, keeps growing in proportion to sequence length. For tasks that must handle sequences hundreds of thousands long, such as long genomic sequences, high-resolution audio, or long-form documents, this quadratic cost is fatal.

SSMs and their apex, Mamba, target exactly this point. Instead of attention that "looks at the whole past," if you compress the past into a fixed-size state and carry it along, cost grows only linearly with sequence length and the state size stays constant at inference. In other words, Mamba attempts to combine the efficient inference of RNNs with the parallel training and expressiveness of Transformers in one framework, and it sits at the center of the "post-Transformer" architecture debate.

B. Necessity and Characteristics

SSMs are needed because the sequences modern AI handles are growing rapidly longer. Understanding long codebases, analyzing hundreds-of-pages contracts, processing long-duration speech and sensor streams, and modeling DNA and protein sequences all demand scales of tens of thousands to millions of tokens. In this domain O(n²) attention hits a memory wall, and a model linear in length is demanded as a practical alternative. Linear complexity is more than a speed improvement — it is a qualitative shift that turns problems of previously unmanageable length into "solvable" ones.

This characteristic lets SSMs excel especially on "signal-like" data — audio waveforms, sensor time series, genomic sequences — that is very long, regular, and where local structure matters. Conversely, on data such as code or formulas that must precisely reference distant tokens, the direct lookup of attention may be advantageous. Thus the blanket claim that "SSMs are better/worse than Transformers" does not hold; one must start from the premise that advantage depends on the nature of the data and the demands of the task.

The characteristics of the SSM family can be summarized in three points. First, the dual form. A linear time-invariant (LTI) SSM can parallelize at training time in a convolution form that processes the whole sequence at once, and switch at inference time to a recurrence form that updates the state one step at a time, generating with constant time and constant memory per token. Second, it is strong at long sequences. Initializing the state matrix favorably for long-term memory (HiPPO) preserves information from thousands to tens of thousands of steps earlier in the state. Third, it is highly efficient at inference. With no KV cache, memory stays constant regardless of sequence length, so throughput far exceeds the Transformer for long-context generation.

That said, early SSMs (such as S4) lacked the ability to "selectively remember and forget by content" because of their LTI structure — always using the same A·B·C matrices regardless of input. They were weak at the content-based reasoning by which attention pinpoints and references specific tokens. Mamba's core innovation is the selective mechanism that breaks this constraint by making the matrices input-dependent, and this is the turning point that lifted SSMs from "toy-benchmark winners" to "language-model contenders."

In sum, the SSM family simultaneously targets the three demands of sequence modeling — capturing long dependencies, training parallelism, and low-cost inference — within one mathematical framework. RNNs had inference efficiency but were weak at training parallelism and long-term memory; Transformers gained expressiveness and parallelism but lost inference efficiency. SSMs, and Mamba in particular, revived the old idea of "state compression" with modern initialization (HiPPO), a selective mechanism, and hardware optimization, offering a new balance point among the three demands — and therein lies their historical significance.

2. The Mathematical Structure of State Space Models and Two Computation Modes

The root of the SSM is the continuous state-space equation of control engineering. When an input signal u(t) arrives, the internal state x(t) evolves as x'(t)=Ax(t)+Bu(t), and the output is determined by y(t)=Cx(t). Here A is the state-transition matrix that sets how the state transitions and decays over time, B is the input's effect on the state, and C is the matrix mapping the state to the output. In deep learning this continuous system is discretized and applied to sequences, introducing a step size Δ and obtaining discrete matrices Ā, B̄ via zero-order hold (ZOH) and the like.

In discretization, the step size Δ is the value that sets how finely the continuous system is sampled, corresponding to "time resolution." A small Δ reflects the input finely and tends to hold the past state longer; a large Δ reflects the current input strongly and updates the state fast. As we will see, Mamba implements "selective memory" precisely by dynamically adjusting this Δ by input, and the key difference is that in a fixed SSM this value is frozen as a learned constant.

Discretized, two equations become central: the state update x_k = Ā·x_{k-1} + B̄·u_k, and the output y_k = C·x_k. This recurrence form, like an RNN, updates the state one step at a time, so at inference it produces the next output in constant time and constant memory without reviewing the entire past. Under the LTI assumption (Ā, B̄, C identical at every step), the same computation can be unrolled into a single long convolution kernel K̄, parallelizing the whole sequence at once via FFT at training. This duality — that "the same model can be computed as recurrence or as convolution" — is the secret of SSM efficiency.

The structural diagram below shows the whole picture of how a continuous SSM, through discretization, splits into the recurrence and convolution modes, and how each is used for training and inference.

flowchart TB
  CONT["Continuous SSM(x'=Ax+Bu, y=Cx)"] --> DISC["Discretization(step Delta, ZOH)"]
  DISC --> REC["Recurrence mode(x_k=A_bar x_k-1 + B_bar u_k)"]
  DISC --> CONV["Convolution mode(batched via kernel K_bar)"]
  REC --> INF["Inference: constant time/memory per token"]
  CONV --> TRN["Training: whole-sequence parallel processing"]
  DISC --> HIPPO["HiPPO initialization(preserves long-term memory)"]
  HIPPO --> REC
  HIPPO --> CONV

One caveat is that the recurrence and convolution forms are mathematically fully equivalent. They merely change the computation method with the same parameters, so the result is identical; it is only a matter of which is more efficient for the situation (training or inference). The moment this equivalence breaks is precisely where Mamba's selective mechanism is introduced; from then on, in exchange for giving up the convolution form, a separate efficiency device — the parallel scan — becomes necessary.

The choice of state dimension N is a key hyperparameter of SSM design. N sets "into how many dimensions the past is compressed": a large N holds more past information and raises expressiveness but increases compute and memory cost, while a small N is efficient but loses more information. The decisive difference from the Transformer's KV cache, which grows in proportion (O(n)) to sequence length n, is that the SSM's state memory is fixed at O(N) regardless of n. For example, with N=16, whether processing 100,000 or 1,000,000 tokens, the state memory stays the same.

Unrolling the convolution form a bit more, the output y of an LTI SSM can be expressed as the convolution of input u with a single long kernel K̄=(C·B̄, C·Ā·B̄, C·Ā²·B̄, …). That is, if you unroll "the result of rolling the state one step at a time" ahead of time into a single filter, you can compute the output over the whole sequence at once via FFT. This is the core mathematical device that let S4 parallelize training even on sequences tens of thousands long. The problem is that computing this kernel K̄ stably and efficiently from powers of Ā is numerically tricky, which S4 solved by restructuring the matrix into a diagonalizable form.

The decisive role here belongs to the theory of HiPPO (High-order Polynomial Projection Operator). Setting A at random lets the state quickly forget the past, but HiPPO designs A to optimally approximate the past signal onto an orthogonal polynomial basis, compressing and preserving information even thousands of steps earlier in the state. S4 (Structured State Space Sequence, 2021) made this HiPPO matrix efficient with a diagonal-plus-low-rank structure, far outperforming contemporary Transformers on long-range dependency benchmarks like the 16K-length Long Range Arena and proving the potential of the SSM family. In particular, S4 became the first to meaningfully solve Path-X (a 16K-length task of flattening image pixels into 1-D) that no model had cracked, empirically showing that "state compression alone can capture very-long-term dependencies."

The practical implications of this structure are clear. Train fast with convolution that maximally exploits GPU parallelism, and process service-stage token generation cheaply with recurrence that carries only the state. For example, when training a 100,000-token document with the same model you use parallel convolution, and when generating an actual response one token at a time you update only a fixed-size state, so memory does not explode even with long contexts. This is an SSM-only strength that the Transformer structurally finds hard to possess.

3. From S4 to Mamba — The Principle of Selective State Spaces (Selective SSM)

S4's limitation was the linear time-invariant (LTI) structure itself. Using the same A·B·C·Δ at every step, it compresses the past "in exactly the same way" regardless of input content. This is ideal for signal processing but unsuited to tasks like language that must judge "whether the word that just arrived matters or can be ignored." For instance, you must let unnecessary particles and whitespace flow by while remembering key named entities for a long time, yet fixed matrices make such content-based selection impossible. This very weakness is why S4 could not surpass the Transformer in language modeling.

Mamba (2023, Gu & Dao) solves this with the selective mechanism (Selective SSM, commonly S6). The core idea is to make B, C, and Δ no longer fixed parameters but functions of the current input u_k. That is, at each step it looks at the input and dynamically decides "how much to reflect this information into the state (B)," "how to project the state onto the output (C)," and "how fast to update/forget the state (Δ)." A large Δ focuses on the current input and resets the past; a small Δ ignores the present and retains the state. Thereby Mamba, like attention, secures the ability to selectively remember important tokens.

The intuition of the selective mechanism put into words is this. Δ (the step size) is a kind of adaptive forget gate that sets "how sensitively to react to the current input." When a meaningful token (a named entity, a keyword) arrives, the model enlarges Δ to strongly imprint that information into the state; for repeated particles, whitespace, or noise tokens it shrinks Δ, letting them flow by almost without changing the state. The idea resembles the LSTM gate, but it is qualitatively different in that it operates input-dependently across all channels and steps and allows large-scale training via a hardware parallel scan. In other words, Mamba realized a "selectively compressing RNN" at GPU scale.

This selective ability shows up especially dramatically in two synthetic tasks presented in the Mamba paper. One is selective copying, a task of picking only specific tokens out of those mixed in at random intervals and copying them in order. The fixed-matrix S4 fails because it cannot change "what to copy" by input, but input-dependent Mamba solves it almost perfectly by placing only the copy targets into the state and ignoring the rest. The other is the induction heads task, reproducing a previously seen pattern ("B follows A") later — a core capability of in-context learning — which Mamba learns and generalizes even to long contexts. These two tasks are compelling evidence that "selectivity is the key to language-model performance."

The moment the matrices become input-dependent, however, the LTI assumption breaks and they can no longer be unrolled into a single fixed convolution kernel. The means of parallel training is lost. Mamba overcomes this with a hardware-aware parallel scan. It views the recurrence relation as an associative scan (prefix-sum) operation and computes it with a parallel prefix-sum algorithm, while processing the state in fast SRAM rather than slow HBM and fusing kernels (kernel fusion) to minimize memory I/O. The detailed process diagram below shows this flow.

flowchart LR
  IN["input token u_k"] --> PROJ["linear projection"]
  PROJ --> SEL["generate selection params(B_k, C_k, Delta_k = f(u_k))"]
  SEL --> DISC2["per-input discretization(A_bar_k, B_bar_k)"]
  DISC2 --> SCAN["parallel scan(associative scan, fused SRAM kernel)"]
  SCAN --> STATE["state update x_k"]
  STATE --> OUT["output y_k = C_k x_k"]
  PROJ --> GATE["gate branch(SiLU)"]
  GATE --> OUT

Why this hardware optimization is decisive becomes clear when you recall the GPU memory hierarchy. An input-dependent SSM must handle an enlarged state tensor at every step, and writing it all to high-bandwidth memory (HBM) and reading it back makes memory I/O the bottleneck. Mamba's selective-scan kernel scans and dissolves the expanded state inside fast on-chip SRAM without materializing it to HBM — removing the memory bottleneck the same way FlashAttention did for attention. As a result, even at the cost of recomputation, overall throughput rises. That Mamba's efficiency holds only when algorithmic innovation (the selective mechanism) and systems innovation (the fused scan kernel) interlock shows well how modern AI architectures advance through the co-design of mathematics and hardware.

The Mamba block wraps this selective SSM in a gated structure. It projects the input into two branches, passing one through the SSM and modulating the other with an activation (SiLU/Swish) gate before multiplying, replacing the Transformer's attention+FFN block with a single homogeneous block. As a result, Mamba shows language-model performance rivaling or surpassing similar-scale Transformers without using attention at all, with inference throughput several times (about 5×) higher per the authors' report and linear scaling in sequence length. Concretely, results were presented showing it handles genomic sequences on the order of a million tokens without quality degradation, proving practicality in the long-sequence domain.

4. Comparison — RNN · Transformer · SSM (Mamba)

To choose among the three families you must understand the fundamental difference in "how they handle the past." RNNs compress the past into a state but, being sequential, are slow to train and weak at long-range dependency. Transformers directly connect all tokens, with outstanding expressiveness and parallelism, but O(n²) cost and an ever-growing KV cache hold them back. Mamba compresses the state like an RNN while securing expressiveness via the selective mechanism and training speed via the parallel scan, splitting the difference. This difference stems not from simple superiority but from a trade-off among expressiveness, efficiency, and long-term memory.

Category RNN/LSTM Transformer SSM/Mamba
Training complexity O(n), sequential (no parallel) O(n²), parallel O(n), parallel (scan)
Inference memory constant (state) growing (KV cache) constant (state)
Long-range dependency weak (gradient vanishing) strong (direct links) strong (HiPPO·selection)
Content-based reference limited very strong strong (selective)
Long sequences (100k+) unrealistic cost explosion favorable, linear scaling

Indeed Mamba can be seen as a "modernized RNN" in that it carries on the RNN lineage while overcoming two decisive limitations. First, it mitigated the RNN's gradient vanishing with HiPPO-based state design and stable discretization, securing long-range dependency. Second, it removed the RNN's sequential training bottleneck with the parallel scan, enabling Transformer-scale large training. That is, Mamba's novelty lies less in an entirely new idea than in "making the classical idea of state compression work again" with modern mathematical and systems techniques.

The contrast with the Transformer in particular comes from a philosophical difference in how information is accessed. Attention keeps the entire past "losslessly" and looks it up directly when needed, so it is overwhelming at the associative recall of exactly copying and recalling a specific token. Mamba, by contrast, "lossily compresses" the past into a fixed-size state, so it gains the ability to choose what to discard but is relatively weak at "copying an arbitrary token it just saw verbatim." For this reason pure SSMs may lag attention on tasks of precisely extracting a specific fact from a long context (needle-in-a-haystack).

The nature of this lossy compression is clearest understood as the "budget" of state dimension N. Attention's effective memory capacity grows essentially without bound in proportion to context length, but the SSM's memory must fit within a fixed budget — the N-dimensional state. The longer the context, the more information must be squeezed into the same size, so the model must more aggressively judge "what is important." The selective mechanism is precisely the device that solved this judgment by learning, but the fact that information is fundamentally compressed into a finite budget does not change. Thus on tasks where arbitrary precise recall is essential, enlarging N or mixing in attention is unavoidable as a complement.

The practical implications are clear. Mamba's linear efficiency is advantageous for overall-trend, summarization, and streaming processing of very long sequences, while attention is still strong for precise fact recall and complex reasoning. Therefore real-world design converges not on "one or the other" but on a hybrid trend. If a few attention layers handle precise recall and many Mamba layers efficiently process long contexts, you can secure quality and cost at once. Indeed, designs mixing attention and Mamba layers at roughly a 1:7 ratio have been reported.

Seen in numbers, the cost contrast becomes even clearer. When context doubles, the Transformer's attention compute rises about 4×, but Mamba's increases only about 2×. At the inference stage too, the Transformer's KV cache starts to occupy memory on par with the model weights past about 32K tokens, whereas Mamba keeps state memory constant and accommodates far longer contexts on the same hardware. That said, this efficiency stands on the price of "lossily compressing the past," so determining upfront whether a task is one where some information may be discarded must be the starting point of design.

5. Deep Dive — Recent Trends and Hybrid Architectures

Research after Mamba unfolded in two directions. The first is theoretical unification. In 2024, Mamba-2 presented the theory of State Space Duality (SSD), revealing that the selective SSM and attention are in fact two expressions of the same mathematical structure. Through this it recast the computation into matrix-multiply-centric operations while greatly enlarging the state dimension, raising GPU efficiency, and Mamba-2 achieved faster training with a larger state. SSD is significant in that it opened a path to design and optimize the two paradigms within one framework, beyond the "attention vs. SSM" dichotomy.

The practical value of this unification lies in letting SSMs absorb the assets of the existing Transformer ecosystem. Tensor-core optimizations and distributed-training techniques developed for attention become largely reusable for SSMs from the SSD perspective, which accelerates the maturation of the SSM ecosystem. Conversely, on the attention side too, borrowing the SSM's linear-kernel ideas to raise long-context efficiency is a cross-transfusion taking place, so the boundary between the two camps grows ever blurrier.

The second is hybrid practicalization. AI21's Jamba (2024) combined Mamba layers, Transformer attention layers, and a mixture of experts (MoE) to unveil a commercial-grade model that efficiently handles a 256K-token long context within a single-GPU memory budget. In the numbers of this flagship case, at the same context length it reduced KV-cache memory to a fraction of a pure Transformer's while raising throughput. NVIDIA and other institutions have also produced results showing that Mamba-attention hybrids deliver longer contexts and faster inference at quality on par with pure Transformers, establishing hybrids as a leading direction for long-context LLMs.

The reason hybrids work is that the weaknesses of the two mechanisms are complementary. Pure SSMs are weak at associative recall — exactly copying and recalling a specific token in a long context — and pure attention explodes in cost for long contexts. If you place only a few attention layers (around 1/8 of the whole) to handle "the moments precise lookup is essential," and fill the rest with Mamba layers to perform most context processing at linear cost, you can keep Transformer-grade quality while greatly lowering the cost of long contexts. Adding MoE on top enlarges parameter count while suppressing per-token active compute, raising efficiency a further notch. This design of dividing roles — "attention for precise recall, SSM for efficient long-term compression" — is settling in as the realistic solution for long-context LLMs.

Application areas are also widening. In genomics, SSMs are applied to DNA foundation models processing hundreds of thousands to millions of bases; in audio/speech, to real-time processing of long waveforms; in vision, to Vision Mamba (Vim) that views an image as a sequence of patches; and in time series/sensor streams, to long-term monitoring. What they share is that all are tasks where "sequences are long, streaming is strong, and constant-memory inference matters." For example, flattening a high-resolution medical image into a patch sequence yields tens of thousands of tokens, so attention cost surges; Vision Mamba processes this at linear complexity and has reported memory-efficiency and throughput advantages on long patch sequences. Industrial tasks like real-time anomaly detection on equipment sensor streams — where "an endlessly lengthening sequence must flow by at constant memory" — are also a domain where the SSM's constant-memory inference naturally fits.

The evaluation of SSMs does not end with a single metric. One must include tasks that measure precise recall of specific facts within a long context, like needle-in-a-haystack, and quantitatively grasp "what quality was given up for efficiency." The expected exam directions from a professional-engineer perspective are ① the contrast of the Transformer's O(n²) limit with the SSM's linear complexity, ② the LTI limit of S4 and the principle of Mamba's selective mechanism, ③ a trade-off comparison of RNN·Transformer·Mamba, and ④ the background of hybrid (Jamba·SSD) design and long-context strategy. An answer is effectively developed by weaving the diagram, complexity, and cases around the essence of "selectively compressing the past into a fixed state."

6. Considerations and Implications

When applying SSM·Mamba to practice and architecture selection, the professional engineer must consider the following comprehensively.

First, an architecture-selection strategy by task nature. If precise fact recall and complex reasoning are core, raise the attention proportion; if very long context, streaming, and low-latency inference are core, raise the Mamba proportion. For most real-world tasks the optimum is neither pure SSM nor pure Transformer but a hybrid, and a design sense for tuning the attention-layer ratio to required quality and cost budget matters. Reasoning backward from requirements, rather than chasing "the latest architecture," is the professional-engineer attitude. For instance, for in-house knowledge retrieval where average inputs are short and accurate QA matters, an attention-centric design is reasonable, but for a service summarizing long-duration logs and call records in real time, a Mamba-centric design greatly lowers total cost.

Second, trade-offs at the hardware and systems level. Mamba's linear efficiency depends heavily on low-level optimizations such as parallel scan and kernel fusion, so performance hinges on the maturity of specific accelerators and frameworks. While the Transformer has a very mature ecosystem such as FlashAttention, the SSM family's kernels and serving stacks are developing fast but are still relatively shallow. At adoption, one must evaluate the support level of the training/inference stack and operational stability together. In particular, unless you verify upfront that quantization, distributed inference, and serving-framework support for SSMs are as mature as for Transformers, and that custom kernels are validated on the target hardware, the theoretical efficiency may not reproduce in actual operation.

Third, recognizing and complementing the expressiveness limit. Pure SSMs have a structural weakness at associative recall and precise copying, so for tasks that must accurately quote a specific fact from a long document, you must fill the gap by combining a few attention layers or external memory such as retrieval-augmented generation (RAG). A perspective of designing pipeline-level complements, rather than trying to meet every requirement with the architecture alone, is needed.

Fourth, long-context strategy and cost governance. The SSM's constant inference memory can greatly lower the total cost of ownership (TCO) of long-context services, but lengthening without quality verification can lose information to lossy compression. You must benchmark context length, state size, and attention ratio together with quality metrics (long-context recall, summarization accuracy) to strike a balance, and need governance that continuously monitors so cost savings do not lead to quality degradation. Evaluation, too, is insufficient with average performance metrics alone; you must include tasks that measure precise recall of specific facts within long contexts, like needle-in-a-haystack, to quantitatively grasp "what quality was given up for efficiency."

Fifth, outlook and related technologies. As SSD theory erodes the boundary between attention and SSMs, future foundation models are likely to evolve toward freely combining the two mechanisms. In domains that must "process long, continuous sequences at low cost" — such as on-device AI, edge streaming, and long-term agent memory — the importance of SSMs is expected to grow. In particular, on-device and edge environments are memory- and power-constrained, so the Transformer with its growing KV cache is at a disadvantage, while the SSM, which maintains a long context with constant memory, is structurally favorable. Therefore the professional engineer should view the SSM not as a replacement for the Transformer but as a complementary design element, holding a balanced design judgment that blends attention and SSMs appropriately per task.

In sum, SSMs and Mamba are a meaningful alternative that cracked the preconception that "the Transformer is the only answer." Points remain to be complemented in ecosystem maturity and precise-recall ability, but the fundamental advantages of linear complexity and constant memory clearly hold value in an AI trend of ever-lengthening sequences. The professional engineer must hold a balanced design judgment that starts from requirements (context length, recall precision, latency, cost, operating environment) and blends attention and SSMs appropriately, rather than blind faith in or rejection of a specific architecture.

References

In one line: The State Space Model (SSM) compresses the past into a fixed-size state to process sequences linearly (O(n)), obtaining both efficient inference and parallel training through its recurrence·convolution dual form; Mamba, with an input-dependent selective mechanism and a hardware-aware parallel scan, surpasses the Transformer's O(n²) limit to offer an alternative strong at long context and streaming, and today it converges on hybrids (Jamba·SSD) combined with attention.