The question of whether custom AI silicon can beat Nvidia's flagship hardware just stopped being theoretical. OpenAI published the first benchmark results for Jalapeño, its first in-house inference chip, and SemiAnalysis verified them in its own lab: across three public models, the chip delivers 1.5-1.9x more AI work per watt, 1.7-3.6x lower end-to-end latency, and up to 4.1x higher performance on highly interactive workloads than Nvidia GB200/GB300 systems. The deeper story is how it got there: from design to tapeout in nine months, with GPT-6 (GPT-Astra) acting as co-designer. The AI hardware race just turned into a flywheel.
The numbers: what Jalapeño actually beat
OpenAI tested Jalapeño on InferenceX, SemiAnalysis's public serving benchmark, using three open models that span the size range: GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. The official headline results:
- 1.5-1.9x more AI work per watt at peak throughput
- 1.7-3.6x lower end-to-end latency
- 2.1-4.1x higher performance on highly interactive workloads
- Rated at 700W, with measured sustained power at or below 550W on the tested workloads (GB300 is rated at 1400W)
SemiAnalysis's independent measurements add the color. On GPT-OSS 120B, Jalapeño produces 1,459 tokens per second versus 535 on GB200 — a 2.7x gap — and finishes a full request in 1.65 seconds versus 5.99 seconds on GB300. Push GB300 to its own maximum decode speed (169 tokens/s) and Jalapeño's throughput at that operating point is 104.3x higher: the gap grows from roughly 1.7x to 104.3x as interactivity demands rise. Full-system economics, with power, cooling and networking spread across every chip, come to $1.56 per chip-hour for Jalapeño — essentially identical to H100's $1.55 — versus $3.61 for Vera Rubin. And these results were achieved with speculative decoding and token prediction switched off, and no prefill/decode separation, while every comparison system ran its own best configuration. SemiAnalysis estimates that enabling speculative decoding alone could cut per-token cost by another two-thirds.
Why agents change the scoring system
Comparing chips by peak throughput at maximum batch size flatters big, power-hungry hardware. Agents break that frame. An agent task runs dozens of sequential steps, so a 4-second latency gap per step multiplies across the whole job. That is why OpenAI benchmarked at a matched user experience — how much useful work per watt while still meeting the latency an interactive agent requires — instead of raw throughput. It is also why the 104.3x extreme point matters: the place where the incumbent starts gasping at its decode ceiling is exactly where agent workloads live. Same model, higher interactivity demand, and the lead widens from 1.7x to 104.3x.
Nine months to tapeout: AI designed its own silicon
The most disruptive number is not throughput — it is time. OpenAI went from design to tapeout in nine months; typical ASIC cycles run 18-36 months. The accelerant was GPT-Astra, the model the outside world calls GPT-6, which worked as a co-designer: exploring implementation options, compressing the design-measure-verify loop, and micro-optimizing arithmetic circuits. SemiAnalysis reports the AI-assisted design shrank SIMD unit area by 8% and matrix engine area by 10%, with better timing and power than the first drafts. AI-written code for the attention and MoE blocks runs 1.5-1.8x faster than code from top human experts. The compute die is roughly 840mm² — 18mm² short of the 858mm² EUV reticle limit. The loop is now closed: AI designed the chip, the chip runs AI, and AI is already optimizing the AI running on the chip. Or as SemiAnalysis put it — GPT-5.6 Sol, a model running on Nvidia GPUs, was used to design a chip that threatens CUDA's moat. Nvidia's own GPUs are birthing their successor.
Is the CUDA moat dead?
For nearly twenty years, CUDA's lock-in was the industry's deepest moat: engineers' muscle memory, two decades of libraries, and the switching cost of moving anywhere else. SemiAnalysis's verdict is stark — the moat may be dead, and the reason is speed. Nvidia's Rubin actually taped out CoWoS a month before Jalapeño, yet no third party has published results on it; OpenAI let SemiAnalysis into its lab immediately. The hardware was never the problem — the pace of software enablement was. A clean-slate design also gave OpenAI an edge: no legacy baggage, an architecture drawn from a blank page. The moat debate has moved from opinion to measurement.
What to watch next: Gen2, 10GW, and the stack
Jalapeño is generation one of a multi-generational platform, and the roadmap is explicit: Gen2 is in deep development, Gen3 is taking shape, mass production ramps through 2027, and the next milestone is 100MW. Behind it sits the 10GW custom-accelerator deal OpenAI signed with Broadcom — roughly ten nuclear power plants' worth of capacity. At rack scale, one system packs 128 chips across CPU and ASIC racks at about 160kW. The efficiency frontier also separates sharply: per-watt HBM bandwidth of 22 versus Rubin's 11.1 and GB300's 5.71; per-watt FLOPs roughly tied with Rubin (19.1 vs 19.4), but at 700W versus 1800W+. The B0 stepping already in the fab adds about 25% per-watt performance over A0.
This is part of a broader pattern: the industry-wide race for equity stakes, long-term supply deals and custom silicon (see our analysis of the AI chip land grab), and the same flywheel appearing in EDA tools like Cadence ChipStack, where AI now handles chip verification. Nvidia still sits on the throne — but the challenger is no longer waiting outside the gate. It is already inside the datacenter, and it designs its own successor. For a closer look at how the chip actually achieves these numbers, continue with our Jalapeño architecture deep dive.
What to do about it
- Re-baseline inference cost models. If a 700W chip at H100-level TCO delivers 1.5-1.9x the work per watt, per-token economics shift — especially for agent-heavy workloads where latency compounds. See how the five-layer token cost breakdown changes once silicon efficiency stops being the bottleneck.
- Stop treating CUDA as a synonym for Nvidia's hardware advantage. The software ecosystem is the real asset; watch software enablement speed, not just spec sheets. Vendor risk calculus just changed.
- Benchmark interactively, not at peak batch. Measure tokens per watt at the latency your agents actually need — that is the number that will decide your next infrastructure contract.
- Expect the flywheel in your own toolchain. AI-assisted design just compressed an 18-36 month ASIC cycle into nine. When evaluating any silicon or hardware vendor, ask what AI did inside their design loop.
- Watch the 2027 ramp. When Gen2 lands with speculative decoding enabled and the 10GW deployment scales, today's per-token prices will start to look like a historical artifact.
FAQ
Is OpenAI really beating Nvidia in inference?
On SemiAnalysis's InferenceX benchmark across three public models (GPT-OSS 120B, DeepSeek R1 670B, Kimi K2.5 1T), Jalapeño delivered 1.5-1.9x more work per watt, 1.7-3.6x lower end-to-end latency, and 2.1-4.1x gains on highly interactive workloads versus GB200/GB300 systems, at a rated 700W (measured at or below 550W).
Does this mean CUDA is dead?
Not dead, but the debate is now measurable. SemiAnalysis argues the moat may be dead because software enablement speed matters more than stack depth: Rubin taped out a month earlier yet still has no independent results, while OpenAI opened its lab to third-party testing.
What makes Jalapeño so fast?
System-level co-design around agent workloads — explicit local placement of KV cache, integrated networking, an 840mm² die near the EUV limit — plus GPT-Astra-assisted design that cut SIMD unit area by 8% and matrix engine area by 10%. And it ran without speculative decoding, which SemiAnalysis estimates could cut per-token cost by another two-thirds.
