A graphics card running at one-seventh of its speed
Eight ordinary PCIe GPUs running DeepSeek-V4.1-Flash produced an input throughput of 1,932 tokens per second on the community's Day 0 baseline. The same eight cards, the same model, with not a single weight changed, later delivered 13,274 tokens per second — a 6.87x jump.
What happened in between? No new algorithm. No new chip. Just the reconnection of acceleration paths that the hardware and the inference framework both already possessed, but that were silently bypassed because of adaptation gaps.
Most coverage of this story reads it as one company's optimization case study. The more important takeaway is structural: in inference, the silicon sets the ceiling, but the software stack decides how much of that ceiling you actually reach. And right now, that software stack is far rougher than most people assume.
The performance isn't missing — it's locked
To see where seven-fold headroom comes from, look at how mainstream inference frameworks actually work.
Frameworks like vLLM and SGLang ship with a large library of high-performance kernels. But those kernels only activate on hardware inside the framework's official validation matrix. A card outside the matrix — a consumer PCIe card, a domestic Chinese GPU, any long-tail accelerator — gets no error message. The framework silently falls back to slow, generic code paths. The service runs; it is just several times slower than it needs to be.
The subtler failure mode is that many high-performance kernels are not missing at all. Mismatched metadata definitions, wrong page-size configuration, or incompatibilities between the kernel and the target hardware simply prevent them from ever being triggered. The performance sits inside the card, locked.
That is exactly what the first phase of the DeepSeek-V4.1-Flash work addressed: aligning metadata between hardware and framework, correcting page-size settings, restoring the sparse-MLA prefill fast path, replacing a slow FP8 dense GEMM kernel, and widening the coverage of the PCIe inter-process communication fast path. Nothing new was invented — throughput rose from 1,932 to 5,850 tokens per second.
The same playbook reproduced on DeepSeek-V4-Flash and GLM5.3. Fixing redundant computation in attention-head padding, vectorizing KV-cache token logic, and extending CUDA graph coverage delivered gains of 20-33% across multiple models from that single phase alone.
Only then do the hardware's real constraints appear. Inter-card bandwidth over PCIe is far below NVLink, so applying the default parallelism strategies designed for high-speed interconnect hardware leaves GPUs idle while they wait on communication. The second phase — operator fusion, overlapping computation with collective communication, differentiating tensor and context parallelism between the prefill and decode phases, and retuning memory allocation — pushed throughput from 5,850 to 13,274 tokens per second.
"It runs" is a trap
Here is a simple lens for judging any deployment: the three tiers of usability — installable, runnable, sellable.
Installable means the weights download and the dependencies resolve. Runnable means the service starts and returns completions. Sellable means throughput, latency, and stability meet production service standards. Most long-tail hardware in the open-source community stops at tier two: a new model ships, the community confirms it runs, and the work is declared done. The path from tier two to tier three is treated as if it doesn't exist.
The lens travels well beyond GPUs. KTransformers, the open-source project from Tsinghua University's MADSys Lab, runs the 671-billion-parameter DeepSeek-R1 on a single 24GB RTX 4090 plus system RAM by keeping attention and the KV cache on the GPU while offloading mixture-of-experts weights to the CPU with fast AMX kernels — published at SOSP 2025, and again a matter of filling the software gap between "theoretically possible" and "actually usable." The reverse also holds: a top-end B300 running a parallelism and communication configuration that doesn't fit the workload wastes a substantial share of its compute.
The gap between the tiers is the profit band of inference economics. It doesn't belong to the chip vendors. It belongs to whoever does the unglamorous work.
Who should care, and why
Push the framework through a few scenarios.
Inference pricing is about to get less predictable. A B300 delivers roughly 4.9-5.6x the input throughput of an optimized 8-card PCIe machine on DeepSeek-V4-Flash — but the PCIe machine's cost can be a small fraction of the flagship's. Video generation makes the point sharper: by whole-machine throughput, about 2.6 RTX 6000D-class machines matched one B300 on text-to-video, shrinking to roughly 1.5 with caching and LoRA optimizations. PCIe does not replace flagship silicon — the remaining gap is real — but the substitution boundary has moved from "impossible" to "viable on some metrics."
For domestic Chinese GPUs, the bottleneck is software too. The same methodology was validated on domestic accelerators that sit outside mainstream frameworks' validation matrices: explicitly enabling platform-optimized sparse attention, disabling NVIDIA/AMD-oriented defaults like Shared Expert fusion, and splitting prefill and decode onto different communication modes. Merely submitting Shared Expert and Routed Expert computations in parallel delivered an 11.26% throughput gain across 35 paired tests. "It runs" is equally just the beginning for domestic silicon; systematic adaptation is the road from sovereign capability to production scale.
Compute scarcity shifts who owns the optimization premium. With high-end accelerator supply constrained and prices elevated, software optimization of installed hardware becomes a genuine source of additional supply. Whoever masters the tier-two-to-tier-three playbook holds several times the effective capacity on identical cards.
What to do about it
- If you buy compute: never evaluate by GPU model alone. Demand measured throughput for your model, your concurrency, on the vendor's actual stack. The same card can differ several-fold depending on software, and a quote built on an unoptimized framework may be quietly expensive.
- If you operate inference: audit framework versions, kernel activation paths, and communication configuration. Confirm high-performance kernels are actually executing rather than silently falling back — before you buy more capacity. Splitting prefill and decode tuning is usually the cheapest win available.
- If you build deployment tooling: long-tail hardware adaptation is an undervalued niche. The open ground between "runs" and "sells" in the open-source ecosystem is exactly where engineering effort converts to value.
- If you're an individual developer: consumer hardware plus heterogeneous scheduling and quantization already runs surprisingly large models locally. Data never leaves the machine, and the marginal cost per token is your electricity bill — for sensitive workloads, that is decisive.
Silicon sets the ceiling; software decides the realization rate. In 2026's inference market, that realization rate is still startlingly low — and a low realization rate is precisely the opportunity left open to engineering teams.