When everyone else is working to make RL training groups bigger, researchers at Tsinghua said — maybe the smallest group is the best one.
Reinforcement learning is becoming the core engine of LLM post-training. From DeepSeek-R1 to OpenAI’s o-series, RL has taught models to reason, code, and use tools. But RL’s efficiency bottleneck is becoming the new ceiling. In long-horizon agent tasks especially, the model needs multi-turn interaction, tool calls, and environment feedback — a single rollout can run to thousands or tens of thousands of tokens. And most RL pipelines still use synchronous batching: wait for the whole group of rollouts to finish, then update the model together.
The consequence: within a batch, fast rollouts wait for slow ones. GPUs idle; the model stands by.
Asynchronous RL could theoretically fix this — update as soon as each rollout completes. But asynchronous RL has long carried a core dilemma: once the model updates, rollouts still in flight become “stale data,” introducing off-policy bias, and training stability collapses. A research team from Tsinghua’s KEG lab just posted an arXiv paper with a counterintuitive solution — Single-rollout Asynchronous Optimization (SAO). Not making the group bigger. Making the group minimal.
GRPO’s Group Dilemma: Bigger Isn’t Necessarily Steadier
To see SAO’s value, start with why GRPO dominates today. GRPO (Group Relative Policy Optimization) is the RL method DeepSeek-R1 uses: for each prompt, sample a group of rollouts and use their relative performance as the advantage signal for updating the model. The group serves two purposes — a stable advantage estimate (in-group comparison cancels bias) and noise dilution across multiple rollouts.
But GRPO rests on two hidden assumptions that long-horizon agent tasks break. First, the whole group must finish before updating. Relative advantage requires every rollout in the group to complete — and in long-horizon tasks, execution times diverge wildly. One rollout may end in 10 steps; another needs 50. GPU resources waste away in waiting. Second, group sampling is naturally incompatible with asynchrony. Async RL’s core is “one arrives, one trains.” GRPO needs the group — arrive-one-train-one cannot compute relative advantage. The two approaches conflict at the root. So the industry faced a multiple-choice question: sacrifice efficiency for stability (synchronous GRPO) or sacrifice stability for speed (async). SAO’s contribution was turning that either/or into “both.”
The Single-Rollout Design: Four Mechanisms Against One Contradiction
SAO’s core idea is direct: generate one rollout per input and feed it into training immediately upon completion. But a single rollout faces two lethal problems — advantage estimation becomes hostage to that rollout’s quality (high variance), and the constantly shifting model parameters in an async environment continuously manufacture off-policy bias. The team answered with four mechanisms:
1. Direct bilateral importance sampling (DIS). Because rollout generation and model training are out of sync, SAO records token-level log probabilities during the rollout phase itself. At update time it computes an importance-sampling ratio per token, and any token outside the set interval skips its gradient update. In essence: “I only learn from tokens that haven’t expired.”
2. Double-rate value-model updates. Single-rollout training has higher gradient variance, demanding a more reliable value model for advantage estimation. SAO decouples the update cadence — for every policy-model update, the value model updates twice. More frequent value calibration lowers the noise in gradient estimates.
3. Freeze the attention modules. The team found the value model’s training volatility came mainly from the full-attention layers while MoE layers stayed relatively stable. So during value-model training, SAO freezes attention parameters and optimizes only the MoE projections — a precise “don’t touch the messiest part” strategy.
4. Skip observation tokens in GAE. In multi-turn agent trajectories, model actions and environment feedback alternate. Folding observation tokens into advantage computation injects heavy noise, so SAO excludes environment-feedback tokens from GAE (generalized advantage estimation), propagating advantage only between the model’s own generated tokens.
All four designs share one goal: compress single-rollout variance and bias to parity with group sampling. The result: SAO held stable through 1,000 training steps while the VAPO baseline collapsed at 90.
“Stream RL” Three Principles: From Batch Thinking to Stream Thinking
As a reusable framework, SAO amounts to three principles of “Stream RL” — against today’s mainstream “Batch RL” thinking:
| Dimension | Batch RL (old thinking) | Stream RL (new thinking) |
|---|---|---|
| Sampling granularity | Update after the group completes | Update immediately per rollout |
| Bias control | Canceled by in-group statistics | Token-level importance-sampling clipping |
| Value estimation | Same-frequency updates | High-frequency calibration (2:1) |
Batch RL’s core assumption is robustness through statistics. Stream RL’s is efficiency through precise control. The switch is essentially choosing between “trusting statistics” and “trusting engineering” — and SAO proved the latter superior in long-horizon agent settings: control every source of bias finely enough and you no longer need a large group to average the noise away.
Online Learning: SAO’s Real Long Game
The most notable SAO experiment is not AIME 2025’s 97.3% or SWE-Bench’s 29.8% — it is the online-learning simulation. The team designed a mock writing task whose reward preference switched among “cute style,” “chuunibyou style,” and “classical style.” After each preference switch, SAO realigned to the new objective quickly, while a moving-average baseline lagged visibly.
Why does that matter? In real deployments, user preferences, business goals, and safety constraints all shift dynamically. Batch RL needs a fixed reward function — change the reward and the entire previously-sampled group of rollouts becomes waste. But single-rollout granularity fits dynamic environments naturally: do one, evaluate one, update one — every cycle reacts to the latest reward signal. That is SAO’s more fundamental advantage over GRPO: it moves RL training from batch-based historical learning to stream-based real-time adaptation.
GLM-5.2 (750B-A40B) has already deployed SAO for its agent RL training — meaning one of China’s largest MoE models is now being RL-trained with the minimal-granularity method.
Implications for LLM Training Teams
SAO is a beginning — the team acknowledges current experiments center on Qwen3-30B-A3B and agent-type tasks; applicability to smaller models and non-agent RLHF settings needs verification. But the direction is clear:
-
Building an agent RL training pipeline? Prioritize async architecture and watch rollout granularity rather than batch size. The efficiency bottleneck is usually synchronous waiting, not compute. SAO’s four mechanisms reuse directly — DIS and double-rate value updates especially.
-
Building RL tooling (veRL, OpenRLHF)? Async single-rollout is a training mode worth native support. Today’s frameworks assume synchronous batching, so the changes are nontrivial — but the payoff is training latency dropping from hours to minutes.
-
Thinking about RL’s next direction? Real-time online adaptation is the capability to watch. When a model can learn while deployed, adjusting as it goes, RL stops being a one-time post-training investment and becomes a core runtime capability of the product itself.
Finally, SAO offers a counterintuitive lesson every AI training engineer should memorize: sometimes the smallest group is the best group. The question was never how big the group is — it is how much care you spend controlling bias on every single token.
Paper: Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning (Tsinghua KEG lab) · Authors: Zhenyu Hou et al. · Deployment: GLM-5.2 (750B-A40B)
