On the evening of July 27, Moonshot AI pushed Kimi K3’s complete weights onto Hugging Face: 2.8 trillion parameters, the largest open-weight model in the world, shipped together with a technical report and three pieces of core infrastructure — MoonEP, FlashKDA, and AgentEnv.
This was not a routine open-source release. Three days earlier, Jensen Huang had posted his first X message — an open letter titled “Open Weights and American AI Leadership,” initially signed by 25 companies, doubling to 50 within 48 hours, with OpenAI and Google adding their names afterward. The White House’s AI lead, David Sacks, publicly dissented, moving the dispute over open models from technical debate to open confrontation between policy and industry. Wired’s verdict was blunt: Silicon Valley has split apart over Chinese AI.
Skip the benchmark talk — that has been covered to death. The real question is what, inside the wall of chip export controls, Chinese AI companies built that made Jensen Huang side against the White House and dozens of Silicon Valley companies take sides publicly.
The Answer: Transformer’s Three Pillars, Each Rebuilt
What stunned the technical community was not the 2.8T parameter count. It was that K3 runs on roughly 50 billion active parameters — under 2% of the total — at a level approaching GPT-5.6 Sol and Claude Fable 5. Deep learning’s three pillars, in place for nine years, were each reworked:
1. Attention: from an ever-growing notebook to a whiteboard
A traditional Transformer’s KV cache is the session’s notebook: every generated token files away Key/Value vectors for the entire context. On Llama 3.1-70B that is about 320KB per token — so a 1-million-token session would demand 320GB of memory, more than NVIDIA’s flagship Rubin card holds in total (288GB).
Kimi’s answer is KDA (Kimi Delta Attention), a linear attention mechanism that replaces the ballooning cache with a fixed-size matrix state. K3 uses a 3:1 hybrid — three KDA layers to every one full-attention layer. Against a traditional MLA baseline, KV cache shrinks by up to 75%, and throughput at million-token contexts improves 6×. The result: K3 carries 40× the parameters of Llama 3.1-70B yet its per-token cache is 13× smaller — the foundation of its million-token context with dozens of concurrent clients.
2. Residual connections: from indiscriminate stacking to selective aggregation
Conventional residual connections add every layer’s output back onto the trunk indiscriminately. K3’s attention residuals (AttnRes) let each layer selectively aggregate the outputs of earlier layers — deep layers learn which prior information is actually useful right now. Training compute drops by about 20%, with all layers folded into 8 independently accounted blocks to keep overhead down.
3. The optimizer: the eight-dollar revolution from AdamW to Muon
In late 2024, researcher Keller Jordan ran the nanoGPT speedrun with the Muon optimizer at $8 per training run. Jordan joined OpenAI in January 2025; a month later Moonshot cited the blog post and validated Muon at roughly twice AdamW’s compute efficiency at the ten-billion-parameter scale. K3 refines it into MuonClip, tuned independently per attention head.
Here is the point of the whole exercise: none of these details needs to clear customs. Attention is an algorithm; residual connections are math; optimizers are composition rules. The compute wall stops chips and lithography machines. It does not stop architecture design.
Two Walls, Two Different Games
Understanding K3 — and China’s AI breakout generally — starts with two walls of different natures.
The compute wall pushed the problem from “compute” to “capacity.” Since October 2022, US export controls have kept the most advanced GPUs out of Chinese hands. But MoE sparsity, low-precision quantization, and hybrid attention reframed the problem from “I need one giant card I cannot buy” to “I orchestrate a fleet of ordinary cards I can buy.” Parameter shunting via MoE saves compute but not weights — K3 still weighs 1.4TB at MXFP4, with an official recommendation of 64+ cards. The buying criteria shifted accordingly: raw compute matters less; memory capacity and interconnect matter more.
The data wall stands in front of everyone. In March, at GTC, Yang Zhilin’s “How We Scaled Kimi K2.5” talk never mentioned chip controls. He talked about something else: high-quality data is finite. “Token efficiency isn’t just an efficiency question — it raises the ceiling of intelligence itself.” When the data runs out, how much a model extracts from each token decides how much further intelligence climbs.
The two narratives do not conflict: the data wall guarantees someone must walk this road eventually; the compute wall decides who walks it first, and how fast. Look back over four years — MLA, DSA, KDA, plus the extreme-sparsity MoE line — the densest stretch of architecture innovation on that timeline came from inside the compute wall.
Three Paths That Rewrite the Global Bill
Path 1: hybrid architectures become the 2026–2028 default
Full attention is not dying; it is being repositioned from fixed backbone to on-demand precision — most tokens flow through cheaper linear paths, with global retrieval summoned only for complex multi-hop reasoning. Expect the same allocation logic to spread to experts, reasoning depth, and precision. One telling signal: MiniMax went hybrid (linear + softmax) on M1, back to full attention on M2 and M2.5, then sparse attention again on M3 — a zigzag over three generations, because as their pretraining lead explained, hybrids shine on small models but lose on multi-hop reasoning at scale. K3’s 2.8T-parameter run settles the question: hybrid works at scale. The era of one architecture for all chips may be ending — future model families may fork at training time for different memory budgets, interconnects, and precision units, compiled per machine like software.
Path 2: compute splits up; memory collects the rent
Jevons’ paradox is playing out in AI: efficiency gains raise total consumption. Nineteenth-century steam engines got dramatically more efficient and coal consumption soared anyway — cheaper energy pulled whole new industries into burning it. Same story here: the cheaper the token, the faster million-token contexts, deep reasoning, and always-on agents pile in. K3 is the living proof — the generation with the most aggressive architecture compression carries the highest price: $15 per million output tokens, nearly 4× K2.6. Moonshot’s commercial lead put it plainly: the pricing is not aimed at cheap; it is a statement that open models, including Chinese ones, are not a discount label. DRAM contract prices rose roughly 90% quarter-over-quarter in Q1. Saved memory gets eaten by longer contexts, higher concurrency, bigger parameters, and demand spills from HBM into main memory and SSDs. Compute is splitting up; memory is collecting the rent.
Path 3: open source as the ticket to the new power center
Huang’s logic is coherent: the more open the weights, the more models behave like commodity parts, and the buyer base for cards widens from a few giants to the broad developer economy. What genuinely worries NVIDIA is its closed-source mega-customers (OpenAI, Anthropic) building their own silicon. Since the K3 upload, Microsoft has been evaluating it for Copilot and Azure integration; DoorDash and Coinbase have confirmed using K3 for cost-sensitive workloads. Open source has stopped being a moral stance and become infrastructure. DeepMind’s Demis Hassabis endorses open ecosystems while proposing a US-led standards body — and Anthropic, holding the closed line, finds itself increasingly isolated in Silicon Valley.
Three Layers of Takeaway
For AI practitioners: linear attention, sparse routing, and quantization-aware training are moving from “seen in papers” to daily tools. Track KDA, Muon, and FlashKDA — they are already shipping options in vLLM, SGLang, and TensorRT-LLM.
For technical decision-makers: token pricing competition is turning from “how cheap” toward “how smart about cost” — the real expense is not inference itself but the generation, storage, and reuse of state (KV cache). Kimi’s differential pricing (cache hit $0.3 vs miss $3 per million tokens) is a clear signal.
For enterprises: choice has never mattered more. Nadella’s line belongs on every CTO’s desk: “Bet the whole company on one AI and you may lose the whole company.” Open models are closing on closed ones in capability at a third to a half the price. Multi-model strategy is no longer a backup plan — it is cost control.
The siege holds, and the breakout accelerates. What the wall meant to block was what the people inside could still build. Four years on — from the shape of chips and the price of memory to the sticker on intelligence — the pen rewriting the global bill is held by the besieged. K3 is not the endgame: the papers and weights are public, the KDA kernels are open. Innovation happens in China, standards settle in the open-source community, and profits flow to memory vendors, clouds, and applications. For Chinese model companies to escape that split, selling weights and tokens is not enough — they need to move into managed runtimes and long-running agents.
That is Chinese AI’s next battle.
