Why Your Local LLM Feels Dumber: Inference Stack Drift

Self-host an open-weight model, and sooner or later you hit the same confusion: this is supposedly the same model everyone online is raving about, but on your box it makes dumber calls, misses obvious steps, and occasionally breaks an agent workflow for no apparent reason. The usual suspects — bad prompting, weak hardware, a too-aggressive quant — get the blame. A new logit-level investigation points at a quieter culprit that sits below all of them: your inference stack itself. If the setup side is still new to you, our guide to running Qwen locally covers the install basics; this test is about what quietly goes wrong after the install.

Level1Techs forum user thr3e ran a punishing test with Qwen3.6-27B on an RTX PRO 6000 Blackwell GPU, capturing full-vocab logits across more than 100,000 tokens. The setup is the key to why this matters: the same weights, the same GPU, the same prompt — and only one inference setting changed. The result was that the model picked different tokens at critical decision points, and in at least one trace it executed the wrong commands on a real Cisco router because a tool call targeted the wrong interface.

Where the drift actually lives

Logits are a pure arithmetic product: matrix multiplies, attention, and activations stacked into floating-point numbers. Same weights plus same input should — in theory — produce identical logits. In practice, floating-point precision, accumulation order, and hardware instruction sets introduce tiny numerical differences. When those differences are large enough to flip the highest-probability token, two "identical" deployments start behaving like different models.

thr3e isolated three sources of that drift:

• Attention backends. vLLM ships three full-attention backends for Qwen3.6-27B: FlashAttention 2, Flash Inference, and Triton Attention. Switching only the backend — nothing else — produced Top-1 flips: the greedy-decoding token diverged from baseline. In one traced failure, FlashAttention 2 resolved a Cisco router interface as GigabitEthernet0/1/4 instead of GigabitEthernet0/0/1.201, then executed two more wrong commands on the back of that mistake.
• KV-cache precision. Holding weights at BF16 and fixing the backend to Triton, the test varied only the KV-cache quant: BF16, INT8, and INT4. INT4 caused the Top-1 flip rate to spike sharply in long context and drove tool calls to a point of no recovery. INT8 flipped too but eventually clawed its way back. Only BF16 stayed stable end to end.
• Reproducibility controls. Repeated runs of the same backend were bit-identical down to every hidden state — so the divergence is purely a product of which CUDA kernels execute the prefill math, not randomness or thermal throttling.

The workload mattered as much as the findings: it was a real ~100k-token agent workflow with multiple tool calls, deliberately absent from any public benchmark or training set, so nobody had tuned or quant-calibrated against it.

The model is not the product. The stack is.

Think of it like an engine. Two cars with the identical engine block can feel completely different depending on ignition timing, fuel mapping, and exhaust tuning. Open weights are the engine block; your 734-package deployment stack is the tune. Change one package, one CUDA kernel path, one quantization decision — and the same "engine" runs a different personality. The flip side is equally useful: a well-calibrated official stack can make a mid-tier model outperform an uncalibrated local one, the same logic behind why Qwen3.8-Flash was added to Qwen Office with most workloads on a standard tier.

This reframes the whole open-weight debate. The official API is not just a convenience layer; it is the reference implementation of a model, with a fixed, calibrated stack that produced the benchmark numbers you read about. A local deployment is a hand-assembled approximation of that reference — and every layer, from attention kernels to KV-cache precision to multi-GPU NCCL config, can diverge from it. In long contexts the differences snowball until the model makes a confidently wrong decision at exactly the moment it matters.

What this means for teams

Several practical consequences follow:

• "Same model, different reviews" is a measurement artifact. When one team says a model is brilliant and another calls it broken, both can be right — the divergence may be in the stacks, not the model or the prompt.
• Agent reliability on self-hosted models is capped by stack drift. Tool calls are exactly where a single flipped token does real damage, which is why agent workloads on local models feel flaky in ways chat never does.
• Quantization is now a business decision, not a memory hack. INT4 KV-cache is fine for short chat and dangerous for long-context agent runs. The savings have a reliability price you only discover after a failure.
• Inference engines are part of the model's spec. vLLM, kernels, and backend choice belong in your model changelog, not your footnote.

What to do before you trust a local model

• Freeze your stack. Pin the inference engine, the attention backend, and CUDA version. Treat "it worked on my machine" as a versioned claim, not a general truth.
• Test with your own long agent traces. Run the exact tool-calling workflows you depend on, at production context lengths, before you route real work through a self-hosted model.
• Keep BF16 KV-cache for anything with tool calls. If you must quantize, verify with your own failure-prone traces — not a static benchmark.
• Demand reproducibility notes. When a model release claims a benchmark, ask what stack produced it. If the answer is vague, assume the number doesn't transfer to your box.
• Default to the API when a task matters. Use local for latency-critical or privacy-critical paths; use the calibrated official stack for correctness-critical ones.

thr3e is packaging the test harness and dataset for public distribution so anyone can reproduce the measurements on their own hardware. That is the right instinct: until bit-level reproducibility becomes a normal part of how we ship and evaluate local deployments, "your local model is dumber" will keep being a mystery that is actually just arithmetic.