Robot foundation models keep getting bigger, which means slower inference. Real deployments hide that latency with asynchronous inference: the robot keeps executing the current action chunk while the model computes the next one. It's smooth, it's necessary — and it quietly breaks online reinforcement learning.
Here's the problem. With async execution, the actions the model planned and the actions the robot actually performed are no longer the same thing. By the time a new chunk arrives, the robot has already moved partway through the old one; before the new chunk finishes, the next inference result overwrites its tail. If you run gradient-based RL over the whole chunk uniformly, you credit actions the robot couldn't change — and blame ones it never executed.
That's exactly what SmoothRL (arXiv:2608.29768, from Astribot's foundation-model team) fixes. It's an online RL framework that fine-tunes a pretrained policy inside the async inference loop, and its core trick is ruthlessly simple: only reinforce the actions the robot actually executed.
Three regions, one gradient path
SmoothRL splits every action chunk by frame index into three zones based on real execution status:
- Committed region — locked in by the previous inference round; unchangeable and guaranteed to execute.
- Execution region — the newly generated actions the robot will genuinely perform this round.
- Discarded region — generated but never executed; the next inference round overwrites it.
Treating these three equally during training is wrong. SmoothRL's insight: let value gradients pass only through the Execution region, so the optimization objective matches the trajectory distribution the robot actually experiences.
Reinforce in Deployment
The second principle is about temporal rhythm. SmoothRL runs asynchronous inference directly during training rollouts — model computation and robot execution happen in parallel, and the replay buffer records trajectories under that real-time relationship. No "train in a clean synchronous world, then switch to async at deploy time." You confront the execution dynamics you'll face in production from the first gradient step.
The concrete setup
Per task, the team fine-tunes π0.5 as the base policy, builds on the RLT framework, and adds a lightweight TD3-style actor-critic that predicts residual corrections in the original action space. A quick sketch of the training loop:
# Pseudo-structure of SmoothRL's per-chunk update
for chunk in rollout:
committed, execution, discarded = split_chunk(chunk, execution_frames)
# residual policy corrects base policy in raw action space
residual = actor_critic.policy(base_policy(obs))
a = base_policy(obs) + residual
# value-gradient update, but ONLY through executed frames
q = critic(obs, a)
actor_loss = -q.mean() # pass gradient only over `execution`
# mask everything outside the Execution region
masked = execution_frames_mask(actor_loss)
update_actor(actor_loss * masked)
update_critic(replay_buffer.sample())On the S1 tendon-driven robot: actions at 30 Hz, inference requests at 5 Hz (a new chunk every 200 ms). The base policy predicts 32-frame chunks; under the latency budget, Committed + Execution occupy 12 frames, with 6 frames in the actual Execution region — the remaining 20 get overwritten before execution.
Results on real hardware
Three real tasks, and the gains are the point:
- Dynamic throwing: 39% → 94% — needs continuous velocity build-up and precise release; any pause bleeds away accumulated speed.
- Pen capping: 8% → 83% — dual-arm alignment within ~5 mm relative pose error.
- Package opening: 30% → 90% — inserting a ~1 mm blade into a 2–3 mm seam.
And it's not just success rate. After online RL, a real autonomous throwing rollout saw the right end-effector's acceleration RMS drop 52% and jerk drop 47%. The robot gets more accurate and smoother.
The failures tell the real story: before RL, all three tasks showed fixed systematic biases (release velocity wrong for target distance, blade drifting left, overly-similar capping motions). Online RL used real execution outcomes to correct exactly those deviations — a deployment problem, not a from-scratch relearning one.
Practical takeaways
- Async inference and online RL are only incompatible if you don't model the timing. Mask gradients to the truly-executed frames.
- Train under the same async rhythm you deploy with — replay buffers should carry real-time relationships, not idealized ones.
- A lightweight residual (TD3-style) policy on top of a frozen base policy is a pragmatic middle ground; it can't rescue a base policy that's fundamentally wrong.
- Expect non-monotonic curves. Real online exploration (package opening dipped 30%→20% before recovering) doesn't improve steadily.