OpenAI has published the first benchmark numbers for Jalapeño, its first custom inference chip. On the public InferenceX benchmark from SemiAnalysis, the 700-watt part delivered 1.5 to 1.9 times more AI work per watt at peak throughput and 1.7 to 3.6 times lower end-to-end latency than systems built on Nvidia Blackwell. For highly interactive workloads the advantage widened to 2.1 to 4.1 times.
We covered these first results from the industry angle - whether the CUDA moat survives the challenge - in OpenAI's Jalapeño Chip Beats NVIDIA: First Benchmarks; this piece switches to a different lens, the architecture and full-stack integration behind the numbers. This is not just a product announcement. It is the first time a leading AI lab has shipped its own silicon and placed it on the Pareto frontier of inference efficiency in a single generation. That matters because inference cost is now the dominant line item in running large models, and whoever controls the cheapest inference stack controls the economics of the next wave of AI products.
What the numbers actually say
OpenAI tested Jalapeño against commercially available Nvidia GB200 and GB300 systems using InferenceX with an 8,000-token input and 1,000-token output, standard single-token prediction, and no speculative decoding. To keep the comparison honest, OpenAI normalized results by each accelerator's published thermal design power.
Jalapeño is rated at 700 watts, but its measured sustained power stayed at or below 550 watts on the tested workloads. Across three public models — OpenAI's own GPT‑OSS 120B, DeepSeek R1 670B, and Moonshot Kimi K2.5 1T — the chip beat the comparison systems on both throughput and latency. On Kimi K2.5, the largest model tested, Jalapeño delivered roughly 1.5 times higher peak performance per watt and 3.4 times lower end-to-end latency, with time-between-tokens dropping about 3.8 times.
The most eye-catching figure came when Jalapeño was asked to match the previous best single-user speed. At that latency target, it produced about 56 times more throughput per kilowatt than the comparison system. On DeepSeek R1, Jalapeño served roughly 700 tokens per second to a single user, versus 169 tokens per second on the rival system.
Why the architecture is different
Most inference hardware is optimized for either the prompt-processing stage, known as prefill, which is compute-bound, or the token-generation stage, known as decode, which is memory-bandwidth-bound. A system that is great at one often sits idle waiting for data during the other. Jalapeño tries to avoid that tradeoff by minimizing data movement.
The chip keeps model state, including the KV cache used during generation, explicitly local. The network is part of the architecture rather than an afterthought, so the entire workload can stay inside one connected domain. The result is a balanced accelerator that can shift resources between prefill and decode as agentic workloads demand. It is manufactured on TSMC's N3P process and offers 13.4 PFLOPS of MXFP4 compute per die.
The bigger picture: full-stack vertical integration
OpenAI also revealed that AI helped design the chip. The team moved from initial design to tapeout in about nine months by using models to explore implementations, shorten verification loops, and optimize arithmetic circuits. The chip was co-designed with Broadcom and is intended to be a predictable programming target for both human engineers and AI code generators.
This mirrors what Apple did with the A-series chips for the iPhone: owning the hardware, software, and workload lets a company optimize the whole stack instead of buying a general-purpose part. For OpenAI, that means models, serving software, memory, networking, and chips can be tuned together using real production traffic. For the industry, it means the inference cost pressure that DeepSeek's price hike recently exposed is now pushing labs to build their own hardware - a shift we mapped in our analysis of the AI chip land grab.
Caveats: first benchmarks are not a verdict
The results are promising, but the comparison window is narrow. InferenceX uses fixed sequence lengths, and OpenAI selected the models and baselines. The chip has not yet been tested on multi-turn agentic workloads such as AgentX, and by the time Jalapeño reaches volume deployment in late 2026 or 2027, Nvidia's Vera Rubin generation may have changed the comparison.
Supply-chain and manufacturing risks also remain. First-generation silicon from cloud labs has a long history of delays and limited availability. Meta and Microsoft have both struggled to ship competitive custom AI accelerators, so a single benchmark is not proof that Nvidia's dominance is over.
What this means for builders
For AI teams, Jalapeño is a signal to treat inference as a cost-and-latency engineering problem rather than a GPU procurement problem. Three practical takeaways stand out:
Measure cost per useful token, not per chip. The relevant metric is how much quality AI work a system can deliver per watt and per dollar at the latency your users actually feel. Raw FLOPS are no longer a reliable proxy. The five-layer token cost breakdown offers a ready framework for re-running those numbers.
Co-design will become the default. As models and chips are built together, the boundary between model research and infrastructure engineering is blurring. Teams that can iterate across both will have a structural advantage.
Do not bet against Nvidia yet. Jalapeño is scheduled for limited internal deployment by the end of 2026, with broader rollout in 2027. Until it is widely available, cloud builders should keep optimizing on existing hardware while watching custom-silicon roadmaps.
FAQ
What is Jalapeño?
Jalapeño is OpenAI's first custom AI inference chip, co-designed with Broadcom, aimed at serving large language models more efficiently than commercially available GPUs.
How much faster is it than Nvidia Blackwell?
On the public InferenceX benchmark, Jalapeño delivered 1.5 to 1.9 times more throughput per watt and 1.7 to 3.6 times lower end-to-end latency across GPT‑OSS 120B, DeepSeek R1, and Kimi K2.5.
When will Jalapeño be available?
OpenAI expects limited internal deployment by late 2026, with wider availability planned for 2027.
