For almost a decade, the Transformer has been the default answer to one question: how do you build a language model? Its component choices — QK normalization, grouped-query attention (GQA), sliding-window attention, a fixed pretraining context length, the output projection layer — were copied from model to model until nobody stopped to ask whether they were actually good.
At COLM 2026, three independent research groups asked exactly that. Their papers target different parts of the stack: one audits long-context architecture choices, one audits the output layer during training, one audits layer pruning. The conclusions converge on the same uncomfortable pattern: the industry's "safe" defaults carry real, measurable costs that standard benchmarks never surface.
Here is what the three papers found, and what it changes for anyone training, pruning, or deploying LLMs.
Cracks in the Foundation: Long-Context Choices That Backfire
Paper one, "Cracks in the Foundation" from Ai2 and collaborators, targets the architectural choices that have become standard equipment: QK normalization, grouped-query attention (GQA), sliding-window attention, and pretraining context length.
To test them properly, the team trained 26 models — the "OlmPool" — all 7B to 8B parameters, all with identical data, tokenizer, and context-extension recipes, differing only in which of the four switches were flipped. Each model received 140B tokens of pretraining plus 10B tokens of long-context finetuning. Total bill: more than 170,000 GPU hours.
The results are striking. On the HELMET 32K long-context benchmark, the best configuration scored 56.4 and the worst 29.9 — a 47% relative gap from identical data and near-identical architectures. Flipping any single switch alone barely matters: pretraining at 4096 vs 8192 context, or adding sliding-window attention, moves scores by only one or two points. The damage comes from combinations. Put sliding-window attention and GQA in the same model and the average drop is nine points — far more than the sum of their individual effects. The worst configuration tested, GQA plus sliding-window attention plus per-head QK normalization, was worse still.
One finding was so consistent it became a diagnostic: counting how many of these "harmful choices" a model contains predicts its long-context score more reliably than analyzing the architecture piece by piece.
The most counterintuitive result involves QK normalization. It was introduced to stabilize training, and the industry treats it as strictly beneficial. But removing QK-norm and post-norm from OLMo 3 and switching to pre-norm raised its HELMET score by six points — while the identical change on Llama 3 cost 3.8 points. The same "standard" component, two different base models, opposite outcomes.
The paper also flips a common assumption about attention sinks — the tendency of models to dump a large share of attention on the first few tokens. Sink behavior is usually treated as wasted compute, and plenty of methods sell themselves on eliminating it. In the OlmPool models the opposite held: models with stronger attention sinking scored better on long-context tasks. QK normalization suppresses that sinking — which may be exactly why it drags long-context performance down.
Lost in Backpropagation: The Output Layer Eats the Gradient
Paper two, "Lost in Backpropagation" from Cornell, examines a component every language model shares: the output projection layer that converts hidden states into word scores (logits).
The mismatch between vocabulary size and hidden-state size is well known as the "softmax bottleneck": the vocabulary is tens of thousands of tokens wide while the hidden dimension is only a few thousand, limiting what distributions the model can express. This paper shows the same mismatch has a second, less visible consequence — in backpropagation.
When error signals flow backward, the output layer must pass information about every vocabulary entry down to the layers below. But the layer's structure caps how much information can actually get through, to roughly the hidden dimension — far smaller than the vocabulary. The researchers measured this information loss on pretrained GPT-2, Pythia, Llama 3, OLMo 2, and Qwen 3 models by projecting error signals into the null space of the output layer's weights. In every model, 95% to 99% of gradient norm is destroyed at this single step. The signal that survives has only 0.1 to 0.2 cosine similarity with the true gradient.
In other words, on every single backward pass, the output layer quietly erases most of the training signal — and the remainder points in roughly the wrong direction. It is the one layer no model can skip, and almost nobody audits it.
When Fewer Layers Break More Chains: Pruning Kills Reasoning
Paper three, "When Fewer Layers Break More Chains" from the University of Tübingen, targets layer pruning — deleting whole Transformer layers to cut compute. Earlier work had shown some layers contribute little, and pruning a quarter of them can retain more than 80% of original accuracy on knowledge benchmarks. That made pruning look like a free lunch.
The catch: knowledge benchmarks ask for short answers that the model can recall. The paper instead tested long-chain reasoning, using test-time scaling — giving the model more thinking tokens or multiple sampling attempts, which normally raises accuracy. The team pruned one to two layers from s1.1-7B and Qwen3-8B using three mainstream pruning methods, then re-ran both kinds of evaluation.
Knowledge scores barely moved. But on AIME24, a competition-level math benchmark, pruning a single layer from s1.1-7B caused a sharp drop in accuracy, and two layers pushed it toward zero. Worse, test-time scaling stopped working on pruned models: more thinking tokens no longer helped, and in some cases longer thinking made results worse. The entire "think longer to get smarter" assumption silently broke.
Fine-tuning cannot cleanly restore the damage. LoRA fine-tuning barely helped the one-layer models, and while full fine-tuning recovered some ground on two-layer models, they stayed far below their unpruned baseline.
What This Means for the Industry
Read together, the three papers describe a single structural pattern: the Transformer stack runs on inherited settings that were propagated, not validated. Every major lab and most open-weight model families carry combinations of these choices, and the interaction effects mean you cannot reason about them one knob at a time.
The implications are concrete. First, evaluation design is now a strategic decision: standard knowledge benchmarks cannot see the long-context, gradient-health, or reasoning costs these papers measured. Second, the long-context race is about to collide with architecture reality — as everyone stretches context windows, the QK-norm/GQA/SWA combinations embedded in current models will determine who hits the ceiling first. Third, efficiency pressure is rising across the industry, pushing more teams toward pruning and quantization; this paper suggests pruned models may be silently losing exactly the reasoning capability that agentic workloads depend on — the same workloads driving model companies to re-architect inference and training together. And as inference demand reshapes the hardware mix, with analysts now expecting CPU and GPU counts to converge toward 1:1 in agentic workloads, squeezing more performance per parameter becomes a competitive necessity, not an optimization nicety.
What You Can Do About It
- Audit your architecture choices as a set, not as switches. If you train or continue pretraining, treat QK-norm, GQA, and sliding-window attention as interacting variables. The OlmPool result suggests the count of "harmful defaults" is a decent first-order predictor of long-context quality.
- Do not let knowledge benchmarks bless a pruning recipe. If you prune, validate on reasoning and math tasks (AIME-style), and re-check whether test-time scaling still works on the pruned model. If it does not, the recipe is hiding a capability regression.
- Reconsider "fixing" attention sinks. The OlmPool models suggest attention sinking correlates with better long-context performance. Before adopting a method that removes sinks, measure its effect on long-context evals.
- Budget test-time compute only for models whose reasoning chain is intact. Test-time scaling is an amplifier: on an unpruned model it amplifies reasoning, on a pruned one it amplifies noise.
- Follow the papers. All three are under COLM 2026, and the OlmPool suite is a public resource for reproducing the 26-model comparison.
For more context on the infrastructure and economics side of this shift, see our guides on speeding up local inference and the 2026 AI trend map.
