Tsinghua SEED Fills the RL Supervision Gap With Self-Evolving Distillation

Outcome correct, therefore every step was correct? That is the blind spot in agent training.

The dominant paradigm for training agents that autonomously complete long-horizon tasks is post-training with reinforcement learning. The logic is intuitive: the agent executes a sequence of actions, the environment returns a final reward — success or failure — and RL updates the policy against that signal.

But a fundamental problem hides inside: a correct result does not mean every step was right. A failed trajectory may contain brilliant reasoning and tool calls; a successful one may be laced with luck and wasted moves. Outcome-based RL carries only one bit of signal — pass or fail — and cannot tell you “the tool call at step 37 was clever, reinforce it” or “step 12 went the wrong direction; don’t do that again.”

This is what researchers call the supervision gap — between episode-level outcome rewards and token-level policy learning, there is no fine-grained, process-oriented feedback signal. Tsinghua’s Tao Jianhua team recently proposed SEED (Self-Evolving On-Policy Distillation) aimed straight at that gap. The name reveals the idea: let the agent self-evolve during training, converting the trajectories it has already walked into “hindsight skills,” then distilling them back into the policy. The paper is at arXiv:2607.14777, with code open-sourced on GitHub.

How SEED Works: a Two-Stage Self-Evolution Loop

Stage one: hindsight skill SFT. First, the base policy collects a batch of interaction trajectories. Then an external analyzer (which can be a separate LLM) summarizes each complete trajectory into natural-language “skills” — success trajectories yield reusable workflows and strategies; failed ones abstract the patterns to avoid and the corrections to apply. These trajectory-to-skill pairs drive a supervised fine-tune teaching the model two roles: as analyzer, extracting high-quality hindsight skills from trajectories; as actor, borrowing those skills to improve decisions on later tasks.

Stage two: self-evolving on-policy distillation — the core innovation. During RL, the current policy model plays both actor and analyzer:

2. Collect trajectories: the actor executes tasks in the environment, generating on-policy trajectories.

3. Analyze skills: the analyzer extracts hindsight skills from them.

4. Contrast scoring: for the same action, SEED compares the model’s probability under “with skill context” versus “without” — the probability shift is the skill’s measured influence.

5. Distill: that probability shift becomes a dense, token-level on-policy distillation signal, jointly optimized with GRPO.

Then the loop turns: as the policy updates, next-round trajectories improve, the extracted skills sharpen, and the distillation signal evolves with them. The key insight: this hindsight supervision comes from the model’s own current trajectories and evolves dynamically with the policy — rather than a static body of knowledge instructing a dynamically updating model.

Results: Dense Supervision Beats Terminal Rewards

The team evaluated on three canonical long-horizon agent tasks — embodied interaction (ALFWorld), web navigation (WebShop), and search-based QA:

• Beats GRPO across the board: on ALFWorld, SEED outscores pure outcome-driven GRPO by 45.9 percentage points.

• Internalization beats bolting on: compared with supplying skill prompts at inference (Skill-Prompt), SEED performs better with no extra input at all — the skills are inside the parameters.

• Dynamic beats static: against fixed-skill-library distillation methods (OPSD, Skill-SD), self-evolution wins clearly, because it adapts to new trajectories and new failure modes appearing during training.

• Double sample efficiency: SEED surpasses GRPO trained on the full dataset while using only 60% of it.

• Cross-task generalization: on ALFWorld’s unseen task set, macro-average success rises from 70.9% to 86.2%.

• Multimodal extension: on visual agent benchmarks (Sokoban, EZPoints), average success climbs from 77.0% to 91.0%.

The Hindsight-Review Era of Agent Training

SEED’s core contribution is less a new algorithm than an underrated insight: the path the agent itself has walked is the best teacher. A year of agent discussion has fixated on planning, tool use, and long-horizon execution; almost nobody seriously asked how an agent learns from its own errors afterwards. Terminal RL signals are too sparse — “you failed, score zero” — with no hint of where. Human-labeled preference data (RLHF) gives finer feedback but costs too much to scale.

SEED’s cleverness: the agent is its own best supervisor. Every trajectory it walks — success or failure — contains reusable experience. Translating action sequences into natural-language skills, then distilling the skills’ behavioral influence back into the policy, is essentially an automated post-game review. It mirrors how humans learn: a top chess player always reviews the game — why was this move strong, why was that one a blunder — and the review’s lessons harden into next game’s intuition. SEED installs an automatic review engine on the agent.

Limits and Directions to Watch

• Complex tasks unproven: on longer-workflow, sparser-reward benchmarks like DeepPlanning and OdysseyArena, SEED’s performance still needs testing.

• Self-evolution carries risk: actor and analyzer share one model, so mistakes risk being summarized as “experience” and repeatedly reinforced — like reviewing your games alone, with no external validation, a self-verification loop waiting to happen.

• Training cost: inference is zero-overhead, but training requires trajectory analysis and paired scoring, which can grow with task complexity.

The limits point to promising directions: verifiable state changes as anchors for skill quality, uncertainty-aware mechanisms to filter unreliable experience, and analyzing only pivotal trajectories to compress training cost.

For Teams Building Agents

2. Stop staring at final success rates. Record every agent trajectory — the failures are as valuable as the successes.

3. Internalize experience rather than bolting it on. Few-shot examples in prompts treat symptoms; capabilities distilled into parameters actually transfer.

4. Let agents learn from their own experience. No expensive human labeling required — the roads already walked are training data. The question is how to extract that experience automatically and dynamically.

The next battlefield for AI agents may not be whose model is bigger or whose toolbox is fuller — it will be who taught their agent to review its own games.