Real-World Robot Learning

STEAM: Self-Supervised Temporal Ensemble Advantage Modeling for Real-World Robot Learning

A label-free advantage model that reads progress, stalls, failures, and recoveries frame‑by‑frame — learned from nothing but the temporal order of expert demos.

Zhihao Liu^1,4,5,* Qiuyi Gu^2,3,6,* Yitao Wang² Dongming Qiao² Yixian Zhang² Shuaihang Chen^7,5 Liangzhi Shi² Tianxing Zhou^8,5 Zefang Huang^9,5 Kang Chen^10,5 Zhen Guo¹¹ Quanlu Zhang¹¹ Jincheng Yu² Xiaodan Liang⁶ Guoliang Fan¹ Yu Wang² Feng Gao^2,3 Xinlei Chen^2,† Chao Yu^2,†

¹ Institute of Automation, Chinese Academy of Sciences ² Tsinghua University ³ Striding AI ⁴ School of Artificial Intelligence, University of Chinese Academy of Sciences ⁵ Zhongguancun Academy ⁶ Pengcheng Laboratory ⁷ Harbin Institute of Technology ⁸ Beijing Institute of Technology ⁹ Zhejiang University ¹⁰ Peking University ¹¹ Infinigence AI * Equal contribution · † Corresponding author

arXiv Code

Scroll

At a glance

Good and bad behavior live in the same trajectory.

Label-free advantage

Frame-level advantage estimated from expert temporal structure — normalized temporal offsets between frame pairs become dense self-supervised targets. No hand-crafted rewards or annotations.

Conservative ensemble

An ensemble of distributional offset predictors, aggregated by the minimum advantage. Members agree in-distribution but diverge on OOD rollouts, so the min suppresses overestimated, false-positive advantages.

+59%

Real-world gains

On four real-robot tasks, paired with CFGRL, STEAM lifts success rate by +59%, +54.3%, +23%, and +16.2% — towel folding, chip checkout, cola restocking, and pick-and-place.

How it works

From temporal order to a refined policy.

STEAM turns the order of expert frames into a progress signal, learns it with an ensemble, then uses it to score data and refine the policy.

STEAM framework: (a) frame pairs from expert demos for normalized temporal offset, forward and reversed; (b) ensemble of M predictors mapping frame pairs and instructions to categorical distributions over temporal bins, converted to scalar advantages; (c) the trained ensemble scores mixed-quality data, guiding a VLA policy through CFGRL. — **(a)** Frame pairs → normalized temporal offsets (forward & reversed). **(b)** An ensemble of M predictors → categorical distributions → scalar advantages. **(c)** The ensemble scores mixed-quality data → CFGRL policy refinement.

01
Label frame pairs

Within an expert episode, the temporal offset of two frames is just j − i. Forward pairs supervise progress; reversed pairs synthesize regression. Offsets are scaled by trajectory length, so efficient demos score higher.
02
Train the ensemble

Each predictor maps a frame pair + instruction to a categorical distribution over N signed temporal bins (cross-entropy). The advantage is the expected bin minus the ground-truth offset — its sign tells progress from regression. M predictors, aggregated by the minimum.
03
Score, label and refine

The trained ensemble scores every frame of mixed-quality data — expert demos, rollouts, and human corrections. A per-source quantile threshold turns scores into positive / negative optimality labels, which then condition a flow-matching π₀ policy through CFGRL — guiding sampling toward the positive, high-advantage behavior.

Step 1Select a frame pair Step 2Length-scale the offset · Δ → Δ̃

predictor p_θ

Step 3Predict a distribution over temporal-offset bins

predicted distribution · ▼ one-hot training target

Click a frame pair above to light up its Step 1 → 2 → 3.

The signal, frame by frame

Watch the advantage move.

A good advantage stays high during proficient motion, dips on hesitation or failure, and recovers when progress resumes. Below, STEAM's frame-level advantage is overlaid on four episode types — pick a task and watch each curve track exactly what the robot is doing.

Comparison

STEAM vs. Baselines.

STEAM performance

Results

STEAM (trained on expert + non-expert data) against Behavior Cloning, HG-DAgger, and RECAP.

Main results

Method	Towel Folding			Chip Checkout			Cola Restocking			Pick-and-Place
	Succ.	Score	Thr.	Succ.	Score	Thr.	Succ.	Score	Thr.	Succ.	Score	Thr.
BC	33.3	3.3	42	39.5	4.6	16	52.0	2.36	71	63.8	1.5	230
HG-DAgger	40.0	3.7	48	53.3	6.0	22	58.3	2.6	84	—	—	—
RECAP	55.6	2.9	39	53.3	5.33	24	52.9	2.1	46	53.8	1.5	161
STEAM	92.3+59	4.9	58	93.8+54.3	7.5	48	75.0+23	3.0	90	80.0+16.2	1.8	254

Succ. = average success rate (%); Score = average completed sub-stages; Thr. = successful episodes / hour. Green = gain in success rate over the BC baseline. HG-DAgger is not evaluated on pick-and-place, where BC already does reasonably well, so no human corrective demonstrations were collected.

Ablations (towel folding, full data)

Two design choices drive STEAM's gains: finer-grained temporal classification and a conservative ensemble. Both lift the success rate substantially.

Bin count N — granularity helps

N	Succ.	Score	Thr.
2	27.3	2.8	41
8	54.6	3.8	51
32 (default)	92.3	4.9	58

Finer-grained bins sharpen the progress signal: success climbs 27.3 → 54.6 → 92.3% as N grows from 2 to 32. A coarse N=2 collapses to a mere forward/backward signal and loses most of the gain.

Ensemble size M — a small ensemble suffices

M	Succ.	Score	Thr.
1	72.7	3.9	53
3 (default)	92.3	4.9	58
5	90.9	4.6	55

The worst-of-M aggregation suppresses overestimated advantages on OOD frames: going from M=1 to 3 lifts success 72.7 → 92.3%. A small ensemble already captures the benefit — M=5 adds nothing further.

Conclusion

Real-world robot learning increasingly relies on heterogeneous data, but demonstrations and rollouts often mix useful progress with stalls, corrections, and suboptimal behavior. Effective policy learning therefore requires frame-level advantages that distinguish reliable local progress from failures and regressions. We propose Self-supervised Temporal Ensemble Advantage Modeling (STEAM), a label-free method that learns such advantages from expert demonstrations. STEAM trains an ensemble of temporal-offset predictors on frame pairs within expert trajectories, using the normalized temporal offset between two frames as a self-supervised signal. Each predictor maps a frame pair to a distribution over temporal offsets, which is converted into a scalar advantage. STEAM then takes the minimum advantage across the ensemble to score mixed-quality rollout data conservatively. Across real-world bimanual towel folding, chip checkout, cola restocking, and single-arm pick-and-place tasks, STEAM identifies stalls, failures, and recoveries. When combined with CFGRL, STEAM further improves policy success rate by 59%, 54.3%, 23% and 16.2% over baselines, respectively.

BibTeX

@article{liu2026steam,
  title={STEAM: Self-Supervised Temporal Ensemble Advantage Modeling for Real-World Robot Learning},
  author={Liu, Zhihao and Gu, Qiuyi and Wang, Yitao and Qiao, Dongming and Zhang, Yixian and Chen, Shuaihang and Shi, Liangzhi and Zhou, Tianxing and Huang, Zefang and Chen, Kang and Guo, Zhen and Zhang, Quanlu and Yu, Jincheng and Liang, Xiaodan and Fan, Guoliang and Wang, Yu and Gao, Feng and Chen, Xinlei and Yu, Chao},
  journal={arXiv preprint arXiv:2606.29834},
  year={2026},
  url={https://arxiv.org/abs/2606.29834},
}