Understanding Kimi Delta Attention.

A notebook is an appealing metaphor for recurrent memory: read a page, revise your notes, move on. But a useful memory must do something harder than collect facts. It must replace an association when new information contradicts it, and then decide which parts of the past are worth keeping.

Kimi Delta Attention (KDA) does both. Its state is a matrix; its writes are error corrections; its forgetting is selective across channels. The mathematical story is already interesting. Then a computational problem appears: each correction depends on the memory produced by the preceding token. How can a GPU process many tokens together without changing the answer?

Our route will take us from associative memory to the delta rule, from decay to structured matrix transitions, and from products of rank-one updates to a triangular system. Along the way we will encounter an idea from numerical linear algebra, the WY representation, originally developed for products of Householder transformations, and see why it is unexpectedly relevant to modern linear attention.

We will work with one attention head, column-vector keys and values, and a key-by-value state matrix. A production KDA layer also includes learned projections, short causal convolutions, normalization, output scaling and gates; those details will return once the core algorithm is clear. This is the same progression used by the Kimi Linear paper, developed here with the tensor orientation used in my PyTorch implementation.

Attention as associative memory

Start with ordinary causal attention. A query at time t compares itself with the earlier keys, turns those comparisons into weights, and uses the weights to combine values.

That is powerful because earlier positions remain separately available. But keeping all those keys and values means the cache grows with sequence length, and computing attention over a long training sequence involves many pairwise interactions.

There is another way to organize the computation. Imagine that, instead of retaining each pair separately, we maintain a matrix that stores a running collection of associations:

Each outer product kvᵀ adds an association. Querying the matrix gives a weighted mixture of stored values, because:

This is an unnormalized linear-attention-style memory, not an algebraic replacement for softmax attention. In particular it omits softmax’s content-dependent normalization. The benefit is structural: the state has dₖ × dᵥ entries, no matter how many tokens have passed.

But that compression immediately creates a problem. If we write v at key k by adding kvᵀ, the new readout at the same key is not necessarily v. It is the old readout plus a contribution from the new write. When keys overlap, old and new associations interfere.

The question becomes much more precise: can we write what the memory is missing rather than simply adding the new value?

Learning to overwrite

Suppose the current state is S, the incoming key is k and its desired value is v. First ask the state what it already predicts:

Then write the error at the key, rather than the full desired value. With write strength β:

Why this particular outer product? One answer comes from a small online-learning problem. Treat the memory as a linear predictor and penalize the error on the current key-value pair:

A single gradient-descent step of size β on this objective gives exactly the delta rule. The intuitive “write the difference” operation is also a rank-one optimization step.

We can go further. If we normalize k so that ‖k‖₂ = 1, query the memory immediately after the update:

This is a remarkably clear result. When β = 1, the new state returns v at the written key. When 0 < β < 1, it moves partway toward v. If the error is already zero, there is nothing to correct.

This is a local guarantee, not complete protection from interference. Writing at another key that is not orthogonal to the first can still change the first association. Fixed-size memory must decide which information to prioritize.

So far we have learned how to correct memory. But we have not learned how to forget.

Learning what to forget

Suppose a conversation changes topic. Some accumulated information should remain important; other information is stale. Corrective writing handles contradictory associations, but it does not explicitly tell the model how long to preserve different parts of its state.

A simple solution would multiply the entire state by one retention factor α before the next write. That is the idea behind a scalar-gated delta rule such as Gated DeltaNet: all key channels decay at the same rate within a head.

KDA makes that choice finer-grained. Instead of one scalar, it uses a vector of retention factors:

Why perform decay before the prediction? Because a correction should respond to the information that actually survived. If forgetting changed the memory’s answer, the delta error must be measured from the decayed state.

This ordering matters algebraically. Expanding the update gives:

The transition (I − βkkᵀ)D is a diagonal-plus-low-rank (DPLR) matrix: a diagonal map plus a rank-one correction. Unlike the scalar case, D generally does not commute with kkᵀ. We cannot simply move it to the other side.

At this point we understand the recurrent algorithm. Each token decays memory, consults it, corrects the prediction, and lets its query read the updated state.

Now comes the obstacle. Every step needs the state produced by the previous one.

The recurrence becomes a problem

For generation, a single token-at-a-time recurrent update is attractive. The state remains fixed-size as the context grows. For training, however, processing an entire sequence through one long chain of dependent state updates leaves much of the GPU’s parallel compute capacity unused.

One possibility is a parallel scan over affine transitions. It is mathematically valid: pairs of “transition plus write” operators can be composed associatively. But for a matrix-valued state, representing all intermediate dense transitions can be expensive in computation and memory. Songlin Yang’s DeltaNet explanation illustrates why that apparently elegant route is not automatically a practical GPU algorithm.

A better compromise is to process small chunks of C tokens. Within a chunk, we reorganize the mathematics into batched matrix operations; between chunks, we carry only the final memory matrix.

The central question becomes:

Can we describe all C corrections using the incoming state, without constructing C full intermediate memory matrices?

There is a structural reason to think we can. Our transition is almost diagonal: its non-diagonal part is only rank one.

From rank-one updates to WY

Before including decay, look at the simpler transition from the delta rule:

This expression resembles the Householder matrices used in numerical linear algebra. A Householder reflector has the form I − 2uuᵀ for a unit vector u. Our delta transition is generally not an orthogonal reflection: its strength β is different. The connection is more specific: both are identity matrices modified by a rank-one outer product.

The WY representation of Bischof and Van Loan showed how products of such structured transformations can be represented compactly rather than expanded into chains of large dense matrices.

Consider just two delta transitions, applied chronologically:

The extra interaction depends on the overlap between the two keys. As we multiply more transitions, those interactions become more complicated. But the crucial insight is that their effect can still be encoded with a modest number of auxiliary vectors and a triangular coefficient system.

For the no-decay case, if K stacks the C keys as rows, the cumulative transition can be represented in a WY-like form I − KᵀTK, with T containing correction coefficients. We will derive the precisely scaled version shortly.

For KDA, channel-wise decay changes the algebra: the cumulative transition is a diagonal product minus a compact structured correction. That is a generalization of the same idea, not a claim that KDA consists of Householder reflectors.

This gives us a guiding principle rather than a black-box algorithm:

Instead of multiplying full transitions, solve for the interactions between the corrections they produce.

The triangular system

This section is the heart of the derivation. We will make the correction dependencies explicit, then turn them into a matrix solve.

Inside one chunk, label the tokens 1, 2, …, C, and call the memory entering the chunk S₀. Define each actual corrective write as a value-space vector z_i:

Let Γ_i = D_iD_i−1···D₁ denote cumulative decay from the start of the chunk. Because the D’s are diagonal, their products are diagonal too. For j < i, let Γ_j→i = D_i···D_j+1, the decay applied to a write at position j before position i reads it. Set Γ_i→i = I.

Unroll the recurrence only as far as token 3:

Each old correction survives through the diagonal decays that follow it. Now substitute those expansions into the definition of z_i.

For the first token, the incoming state is all that matters:

For the second token, the first correction has already modified the state:

The third correction has two earlier corrections to account for. Rather than writing an even longer formula every time, define one base row and one coefficient for each dependency:

Then every correction obeys the same rule:

There is a beautiful additional interpretation. Since a strictly lower-triangular C × C matrix is nilpotent, A^C = 0. Therefore:

Think of A as a weighted directed acyclic graph from earlier corrections to later ones. A single power represents one dependency hop; higher powers collect longer paths. The inverse sums all causal routes through which an earlier correction influences a later one. This is the graph interpretation highlighted in Songlin Yang’s treatment of the UT transform.

We have not removed causality. We have encoded its complete effect in one triangular linear system.

Putting the chunks together

Now we can derive the computational form that connects the theory to the code. Denote

The last line is the practical detail to notice. Instead of multiplying many retention factors, or explicitly computing Γ_iΓ_j⁻¹, we subtract cumulative log-decays and exponentiate the difference. This directly represents the desired interval and avoids an explicit reciprocal of a potentially tiny cumulative product.

Stack the keys and values as rows of matrices K and V. Set L_i,: = (Γ_ik_i)ᵀ. Let T be the solution to (I − A)T = I, and let Β = Diag(β₁,…,β_C) be the diagonal write-strength matrix.

Then define:

Why does this work? The unsolved base corrections are B = Β(V − LS₀). Substitute this into Z = (I − A)⁻¹B and distribute the multiplication. We recover exactly U − RS₀.

Every correction in the chunk is now represented as one row of Z, calculated without iterating through full memory states.

We still need two things: outputs for the tokens and a state to pass to the next chunk.

The token outputs. At position i, the query sees the decayed incoming state plus all corrective writes from positions j ≤ i:

Collect the coefficients into a lower-triangular matrix H, where H_ij = q_iᵀΓ_j→ik_j for j ≤ i and zero otherwise. If the rows of Q̄ are (Γ_iq_i)ᵀ, then the whole chunk’s outputs are:

The outgoing state. Every correction is decayed from its token to the end of the chunk:

These are the entire chunkwise mechanics. One small triangular system gives corrected writes; two matrix products recover outputs and the boundary state.

From mathematics to the GPU

The resulting algorithm has the features we wanted:

• A fixed-size state carries information between chunks.
• A strictly causal triangular matrix captures the dependence among corrections in each chunk.
• Matrix multiplications perform much of the work inside a chunk.
• Channel-wise log-decays preserve KDA’s fine-grained forgetting without building explicit large transition matrices.

It is important not to overstate the parallelism. Chunks still depend on previous chunks; the triangular solve also has a causal structure. The win is that work on C tokens can be grouped into accelerator-friendly operations instead of executing an entirely separate full-state update for every token.

Here is how the key objects appear in the readable PyTorch implementation:

The project constructs the interaction coefficients using exp(g_i − g_j), solves (I − A)T = I with torch.linalg.solve_triangular, and carries a key-by-value state between chunks. The readable reference uses FP32 for its chunkwise core and enforces sequence length divisibility by the chosen chunk size; its fused backend has a different execution contract. The recurrence/chunkwise equivalence tests check that the two implementations agree within the tested numerical tolerances.

One useful distinction: the paper’s Kimi Linear formulation and my scaled Kimi K3 implementation share the same core recurrence, but not every neural parameterization is identical. My implementation uses a bounded negative log-decay, with a lower bound of −5 applied through a sigmoid, to limit extreme retention values. The Kimi Linear paper uses its own learned decay parameterization. Those are choices about how α is produced, not different definitions of the chunkwise correction algebra.

Another distinction: this article has described the mathematics of one head. The actual attention module has multiple heads, short causal convolutions on Q/K/V, learned write strengths and output gating, RMS normalization and a final projection. Those operations matter to model behavior, but none needs to be mistaken for the triangular core we have just derived.

We can now answer the original question. What makes KDA possible is not a way to ignore sequential dependence; it is a way to describe that dependence compactly and resolve it efficiently.

We started with an associative memory whose additive writes cannot reliably overwrite what it already knows. The delta rule turned each write into an error correction. Channel-wise decay gave the memory independent retention rates. The resulting transition had a special diagonal-plus-rank-one structure. The WY perspective suggested that chains of structured transformations need not be expanded into dense matrices. Finally, the UT-style triangular system collected the corrections inside each chunk, leaving one recurrent state to cross its boundary.

That is the path from the delta rule to Kimi Delta Attention: remember selectively, correct what the memory currently predicts, and reorganize the resulting causal dependencies into matrix operations that GPUs can execute efficiently.

The central source for the full derivation is Kimi Linear, Appendix B. For complementary explanations, I recommend Songlin Yang’s DeltaNet algorithm walkthrough, Jamie Dborin’s derivation-led KDA essay, and Haoran Zhang’s visual tour of linear attention. Each emphasizes a different part of the same underlying story.