Even Musk and Andrej Karpathy have been dropping approvals on this one: the “residual connection” — a component DeepSeek and Kimi both targeted back to back — is suddenly the most interesting corner of deep learning. Kimi (Moonshot AI) just released a heavyweight paper attacking a Transformer foundation that has gone untouched for a decade: the residual connection, proposed by He Kaiming in the 2015 ResNet paper and a fixture of deep learning ever since.
The metaphor that makes it legible: imagine a large model’s Transformer as a line of dozens of workers relaying a message. The residual connection is the rule that each worker, after hearing everything said before, adds one sentence of their own and passes the whole pile forward verbatim. The trouble: the worker at the end of the line receives everything everyone said, piled on top of each other — the further down the line, the messier and longer the message, the important early points get buried, and later contributions go unheard. The AI gets dumber. This is the dilution problem.
Kimi’s fix imports attention into the residual itself — a rule they call Attention Residuals: give the workers a smart filter so they stop accepting the whole pile and can instead choose to focus on the upstream content they care about.
The Results: Smarter and Cheaper
The model gets smarter: stable improvements across the board, and on the tasks that genuinely test capability — graduate-level exams, advanced mathematics reasoning, code generation, ultra-long-context understanding — direct jumps of 3 to 7.5 points. And it gets cheaper: validated on Kimi’s own model architecture, it saves 1.25x compute, with training’s end-to-end overhead under 4 percent and inference latency up less than 2 percent — essentially no added burden. Better still, it fits the scaling laws cleanly: the larger the model, the bigger the advantage.
Why the Residual Connection Persists — and Its Three Fatal Flaws
The residual connection survived a decade because it opens an information highway: it carries the error-correction signal that trains AI all the way down the stack, making deep networks trainable without losing what earlier layers learned. But three flaws are structural. You cannot choose what to hear: whether a layer extracts highlights or organizes content, it receives the whole pile — no selecting. Buried content is unrecoverable: once an early key point gets covered by later additions, it never resurfaces — which is why AI forgets your opening instruction in a long chat. The pile keeps growing: downstream workers must shout louder and add more to be heard at the end of the line, which is exactly what makes training deep models chaotic.
Previous fixes — PreNorm among them — merely reordered the housekeeping without touching the fixed accumulation rule. Kimi’s insight came from noticing that “the time problem along the sequence dimension and the layer problem along the depth dimension are, at bottom, the same problem.” When AI once read long sentences word by word and forgot the beginning, attention solved it by letting the model choose which words matter. AttnRes applies that same medicine to depth.
DeepSeek mHC vs. Kimi AttnRes: Two Roads Up the Same Mountain
Both teams innovated on the residual; both also shipped multi-head compression era architectures. DeepSeek’s mHC says: the highway is jammed, so widen it — build multiple parallel lanes with learned rules mixing traffic between them. Kimi’s AttnRes says: the highway jams because nobody can pick what matters, so hand out the filter. Both lift baseline performance, but their strengths diverge sharply:
- mHC: built for training stability — solid base-capability gains, weaker on complex tasks. Its core value: making models deeper without training collapse, solving the deep-network crash problem.
- AttnRes: stability plus complex reasoning — crushing improvements on long context and logic problems. It fixes deep-network training stability and excels at the high-difficulty tasks that demand precise recall and multi-step reasoning.
Deployment fit differs just as sharply. DeepSeek’s mHC is structural surgery — the existing single-stream architecture must be rebuilt into a multi-stream parallel one. Kimi’s AttnRes is a drop-in replacement part: swap the original residual block for Block AttnRes and leave everything else — architecture, training hyperparameters — untouched. Community members comparing the two in Kimi’s own ablations found that Kimi’s full-lossless Full AttnRes consistently outperformed mHC, while Block AttnRes performed on par — with only one-sixth of mHC’s data read-write volume.
For engineering teams, the practical significance is the deployment story rather than the benchmark delta. Architecture upgrades historically arrive as “retrain everything” propositions, which is why most die in committee — the accuracy gain never survives the retraining risk. AttnRes’s drop-in property inverts that calculus: the swap is local, the training recipe is unchanged, and the ablation table provides a direct A/B path — run Block AttnRes on your current checkpoint lineage, measure, and keep it only if the long-context and reasoning gains replicate on your datamix. The 1.25x compute saving compounds the case, since it is realized at every subsequent training run rather than once. And because the mechanism is orthogonal to the attention upgrades teams are already shipping, it stacks: the residual filter governs how information flows between layers, while the attention upgrades govern how information is gathered within them.
There is also a competitive reading that should not be missed. Kimi’s paper landed in the same window as its peers’ infrastructure pushes, and the ablation tables quietly benchmark against DeepSeek’s published method — the first time these two labs have traded architectural blows in public with measurable numbers rather than benchmark cherry-picks. When the underlying data recipe stops being the differentiator, the differentiators become the parts nobody thought to optimize. The residual connection was the least examined component in the stack precisely because it worked; now that two frontiers labs have published attacks on it, expect every serious lab’s next generation to ship some form of attention-gated residual — the same way PreNorm went from paper to default in eighteen months.
The Signal
For a decade, Transformer innovation concentrated on attention, FFN, MoE, positional encodings, and training data. The residual connection sat there like air — important, but so foundational that almost nobody dared touch it. Now DeepSeek and Kimi have moved in sequence, and the signal reads clearly: when the data dividend peaks, the next generation of large models will compete on how well they manage information — not just how much of it they swallow. Reference: the Kimi paper. Source: InfoQ.
