Modern Transformers are powerful because self-attention lets every token read every earlier token. The cost is also the limitation: for a sequence of length L, full causal attention materializes an interaction pattern that grows as O(L^2). Kimi Linear explores a different tradeoff. Its Kimi Delta Attention (KDA) mechanism keeps a recurrent matrix state, updates that state with a delta rule, and uses a learned gate to control how much memory survives.
Why quadratic attention becomes expensive
For one attention head, let q_t, k_t, and v_t be the query, key, and value at position t. Causal attention computes:
score(t, i) = q_t^T k_i / sqrt(d)
weight(t, i) = softmax(score(t, 1..t))_i
output_t = sum_i weight(t, i) v_iDuring training, the score matrix has roughly L^2 / 2 causal entries per head. FlashAttention reduces memory traffic and avoids storing the full matrix, but it does not change the quadratic interaction count. During autoregressive decoding, a KV cache makes each new step linear in the current context length and the cache grows with the number of tokens.
Linear attention replaces the normalized sum with an associative feature map. If phi(q)^T phi(k) approximates the kernel, the output can be rearranged as:
S_t = S_(t-1) + phi(k_t) v_t^T
z_t = z_(t-1) + phi(k_t)
y_t = (phi(q_t)^T S_t) / (phi(q_t)^T z_t)The sequence can now be processed recurrently with a fixed-size state. The price is representational: the state is a compressed summary, so it cannot preserve every pairwise interaction exactly.
From attention to recurrent state
Delta attention gives the state a more selective update than simple accumulation. Let S_(t-1) be a state matrix that maps keys into values. A plain associative update adds the outer product v_t k_t^T, but it does not explicitly correct what the old state would have predicted for the new key.
The delta rule first computes the memory's prediction:
p_t = S_(t-1) k_tThen it forms a write error:
e_t = v_t - p_tand writes an outer-product correction:
S_t = S_(t-1) + beta_t e_t k_t^Twhere beta_t is a learned or input-dependent write strength. This update has an intuitive interpretation: if the state already maps k_t to the desired value, the error is small and little needs to be written. If the prediction is wrong, the delta update corrects the mapping at that key direction.
Equivalent formulations factor the update as a forgetting term plus a value write, for example:
S_t = S_(t-1) (I - beta_t k_t k_t^T) + beta_t v_t k_t^TThe exact implementation includes normalization and parameterization details, but the core idea is stable: update a bounded memory by correcting its prediction instead of appending an unfiltered token record.
How the Kimi gate changes memory
KDA adds a data-dependent decay or gating mechanism. Denote the gate by alpha_t, constrained to a useful range such as (0, 1). A conceptual update is:
p_t = S_(t-1) k_t
e_t = v_t - p_t
S_t = alpha_t * S_(t-1) + beta_t * e_t k_t^THere alpha_t controls retention and beta_t controls the strength of the correction. A token that begins a new topic can learn to decay stale state. A token inside a stable entity or code block can retain more state. This is more expressive than a fixed exponential decay because the forgetting behavior is conditioned on the current content.
In practice, the gate is produced from the token representation and parameterized for numerical stability. A common design uses a sigmoid-like value for retention and a positive write scale. Implementations also normalize keys, use head-specific dimensions, and apply careful initialization so the recurrent state does not immediately explode or vanish.
At inference time, the important systems property is that the state is updated in place. The model does not need to retain a key-value pair for every previous token for this attention branch. Its memory is approximately fixed in the sequence dimension, although it still scales with the hidden and head dimensions.
Why a hybrid architecture matters
A bounded recurrent memory is not a universal replacement for exact attention. Some tasks need sharp retrieval of a specific earlier token, exact copying, or interactions between distant positions that a compressed state may blur. Kimi Linear therefore uses a hybrid strategy: most layers use efficient linear attention, while a smaller number of full-attention layers preserve high-fidelity global retrieval.
This is a useful architectural pattern. The linear layers handle the majority of token processing with low memory growth; full-attention layers periodically refresh exact cross-token interaction. The hybrid ratio is a quality-throughput knob:
- More linear layers reduce KV-cache memory and improve long-context decoding throughput.
- More full-attention layers improve exact retrieval but increase training and serving cost.
- Layer placement matters because full-attention layers can act as information exchange points for the compressed streams.
Hybrid models also complicate batching. The serving runtime must maintain recurrent states for linear layers and KV caches for the full-attention layers, with separate memory accounting and checkpoint logic.
Systems and implementation tradeoffs
Training
The recurrent form is sequential across time, which can appear less parallel than a standard attention matrix. Efficient training uses chunkwise scans: a chunk can be processed with a compact boundary state, and associative scan kernels parallelize work within the available hardware structure. The state at a chunk boundary becomes the initial condition for the next chunk.
Decoding
For a linear head, each new token requires a state update and a query-state product. The cost is independent of the total sequence length once the fixed state exists. Full-attention layers still require their KV cache, so the total model cost is determined by the hybrid allocation, not by KDA alone.
Numerical stability
Repeated matrix updates can accumulate error. Practical implementations need normalized keys, bounded gates, stable precision choices, and monitoring for state norms. A useful diagnostic is the distribution of ||S_t|| by layer and head across long synthetic sequences. Sudden growth often indicates an unconstrained write scale or poor initialization.
Kernel design
The theoretical state size is only half the story. A fast implementation must keep recurrent state in registers or fast shared memory where possible, fuse the prediction, error, gate, and write operations, and avoid materializing intermediate tensors. Memory layout should support both the scan path used in training and the one-token update path used in decoding.
Limits and open questions
KDA trades exact history for learned memory. It can be excellent when the task rewards compression, recency control, and streaming behavior, but no recurrent state can guarantee lossless storage of an arbitrarily long context at fixed size. Hybrid full-attention layers reduce this risk rather than eliminating it.
Evaluation should therefore go beyond average language-model loss. Test needle retrieval at increasing context lengths, exact copying, code completion, multi-hop evidence, streaming latency, state reset behavior, and degradation after very long sequences. Compare both quality and the actual end-to-end serving profile, including memory bandwidth, batch size, and tail latency.
The deeper lesson is that attention is not one indivisible operation. A model can combine exact associative lookup with learned recurrent memory, assigning each mechanism the part of context handling it is best suited to perform. Kimi Delta Attention is one concrete example of that broader design space.
The future of long-context models may not be bigger KV caches, but better learned memories paired with a small amount of exact attention.