CUDA has been chiseled open — not through its whole width, but at one decisive point. Understanding where that point sits, and why it sits there, tells you more about the next three years of AI infrastructure than the headline number does.
The raw fact: a one-year-old startup called Infinity used its in-house AI coding agent, Ignition, to build a CUDA-class software stack for chip startup d-Matrix in ten hours. What followed in the industry’s reaction — the attackers, the defenders, and the skeptics — sketches the real map of where Nvidia’s moat stands.
What Happened in Those 10 Hours
Ignition’s workflow is disarmingly simple. It generates a GPU kernel, compiles it, runs the tests, measures performance, and rewrites the code based on the feedback. Human engineers supply only the high-level direction; the agent loops through the drudgery on its own.
Infinity founder Jeremy Nixon, a former Google Brain researcher, told Business Insider the result shows one of Nvidia’s deepest moats being crossed. The company is now valued at around $100 million on a $15 million raise — numbers that say the market is not treating this as a science experiment.
And Infinity is not acting alone. DeepSeek founder Liang Wenfeng has said separately that AI coding agents combined with DeepSeek’s own TileLang language have made building low-level AI software dramatically easier. DeepSeek open-sourced TileKernels, a GPU operator library written in TileLang, eliminating a large volume of hand-written CUDA.
CUDA has been cracked — not everywhere, but at one critical seam.
Three Layers, One of Them Loosening
To see what the crack actually threatens, picture CUDA as three stacked layers.
Layer one, the kernel layer: GPU kernels, compilers, and the runtime — the code that makes silicon do work. Layer two, the library layer: heavily optimized math libraries like cuBLAS and cuDNN, plus debugging and verification tooling. Layer three, the ecosystem layer: frameworks like PyTorch and TensorFlow, millions of lines of enterprise code accumulated on top, and the developer community holding it all together.
Infinity went straight at layer one: automatically generating and optimizing inference kernels for arbitrary chips. That is the most hand-rolled stratum of the stack and the single biggest software obstacle between a new chip and its first customer. The historical cost of that layer for a new piece of silicon ran from months to years. An agent cycle measured in hours is not an incremental improvement — it changes who can plausibly enter the market.
Why the Inference Side Cracked First
Training and inference are different wars, fought with different economics.
Training means thousands of accelerators grinding in lockstep for months. It is merciless about performance, stability, and cluster-scale communication, and CUDA’s accumulated advantages there — communication libraries, fault recovery, debugging at supercomputer scale — remain close to irreplaceable. No large lab casually switches training platforms.
Inference runs on a single card or a small cluster. Buyers no longer care about absolute performance; they care about cost per answer. Anything that runs the model cheaply gets a trial.
Marshall Choy, chief commercial officer at Korean AI chip company Rebellions, put it bluntly: “On the inference side, CUDA is no longer the deciding factor. This is going to be a competition of open-source software.” Inference frameworks that run across chips mean users can switch hardware without rewriting code — and the moment switching is free, CUDA’s core lock-in mechanism stops functioning.
d-Matrix itself is an inference chip company. Cofounder Sid Sheth has argued the inference opportunity will ultimately exceed training — which is exactly why the company hired an AI agent rather than a hundred engineers to build its software base.
Attackers, Defenders, and a Skeptic
The attackers:
- Infinity — agent-accelerated chip enablement, aimed squarely at the inference market; a CUDA-class stack in ten hours.
- DeepSeek — TileLang and TileKernels replace hand-written CUDA with a higher-level language, dropping the floor on low-level development.
- Rebellions, Cerebras, Amazon Inferentia, Google TPU, AMD MI300X — chipmakers and cloud giants already positioned across the inference battlefield.
The defenders:
- Nvidia’s own AI agents. Ankit Patel, the company’s VP of developer ecosystem, says Nvidia is using agents internally “to develop CUDA faster and validate it at greater scale.”
- CUDA’s verification toolchain. Bing Xu, a former Nvidia engineer and founder of INT21, notes that agents can generate enormous volumes of code, but verification is the real bottleneck — and CUDA’s deepest asset is precisely its verification ecosystem.
And the skeptic in the middle: Chris Lattner, founder of Modular and creator of LLVM, poured cold water on the triumphalism. Writing code, he argues, is only a small slice of software engineering; production-grade optimization is what actually determines chip performance. Chip software is a small, elite field with scarce training data — a poor fit for models that learn from volume. And migrating millions of lines of existing enterprise code is an organizational decision, not a technical one.
The Moat Is Relocating, Not Disappearing
Weigh the arguments and a sharper picture emerges.
1. Training is safe for now. The quality bar — cluster coordination, stability, raw performance — is so high that every alternative faces a brutal validation curve. Lock-in there remains deep.
2. The inference gap is open, but open is not overturned. How fast inference shifts depends on a race: whether AI agents improve chip software productivity faster than Nvidia improves CUDA itself.
3. The moat is migrating from code stock to verification. This is the key insight in the whole episode. When generating code becomes nearly free, the scarce resource becomes proving the code is right — correct across edge cases, stable under production load. Bing Xu’s prediction that verification becomes CUDA’s next moat deserves to be taken seriously.
4. Inference will be an open-source battlefield. Cross-chip inference frameworks give users chip-agnostic mobility — the structural opposite of CUDA’s lock-in. Open source wins where switching is the product.
What Practitioners Should Watch
- AI agents in systems software. If agents keep cutting chip enablement costs, market entry gets cheaper — more competitors, faster iteration, lower prices.
- Inference costs may fall faster than training costs. Cheap inference silicon plus agent-built software stacks is a compounding combination, and the slope could outpace anything seen on the training side.
- Revalue verification. When code generation is a commodity, testing and validation become the new competitive barrier — a lesson that applies to every field where AI writes code, not just chips.
- Nvidia’s counter of fighting fire with fire. Nvidia’s own agents speed up its development too. That discounts the attackers’ relative advantage — and accelerates everyone’s clock.
The one-line summary: CUDA has not been disrupted, but its moat is migrating from “twenty years of code” to “twenty years of verification tools” — and inference is the first battlefield of that migration.
The final suspense of this fight is not who wins under the old definition of the moat. It is who is standing in the right place under the new one.
