Apple researchers release REVERSAL-BENCH to measure reset-free RL failures

Apple researchers Riyaaz Shaik and Chandru Venkataraman have released REVERSAL-BENCH, a simulation benchmark for measuring how reset-free reinforcement-learning agents behave as simulated environments become harder to recover from. The benchmark varies reversibility along a controlled axis and uses a reset oracle to label whether a state can return to the task-relevant starting distribution.

In the authors’ experiments, reset-free and safe-RL methods were increasingly absorbed into irreversible states as the reversibility parameter, rho, moved toward its most severe setting. By contrast, episodic baselines with external resets maintained stable learning. The results come from simulations and a preprint; they have not been independently replicated or demonstrated on physical robots.

The paper defines rho on a scale from 0 to 1 within each task. It changes the severity or extent of an irreversible region while preserving an achievable optimal path. The authors caution that rho is a domain-specific control, not a score that can be compared directly across tasks.

REVERSAL-BENCH covers eight manipulation settings across five physics engines. Its simulator-grounded oracle saves a queried state, runs task-specific recovery routines, determines whether the system can return to the relevant initial-state distribution, and restores the original state so the evaluation does not alter the trajectory. The team also reports releasing a dataset of 44.9 million transitions across 11 task classes and five simulators, with ground-truth recoverability labels.

Like Apple’s separate DiscoSign research on discourse-aware ASL translation, REVERSAL-BENCH is a research release rather than a product announcement.

The paper’s paired tests hold geometry, spawn distributions and rewards constant while changing whether a trap is physically escapable. At rho equal to 1, the authors report that Naive, R3L and Leave-No-Trace runs approached complete absorption in the irreversible setup but recorded zero absorption in reversible counterparts. A model-based method, MoReFree, was a notable exception: after incurring absorption during exploration, its converged policy found a high-momentum route through the force sink and reached zero absorption.

In a ManiSkill3 tabletop task, a learned PPO policy showed a similar cliff in the authors’ tests. Across eight seeds and 10 million training steps, reset-free absorption rose from 0.10 to 0.95 as the table-edge margin shrank toward rho equal to 1, while the episodic baseline remained at zero.

The researchers also tested a Reversibility Shield that can override actions when estimated recoverability falls below a threshold. In point-mass navigation, they report that it reduced absorption from 0.63 to zero at rho 0.25 and from 0.85 to 0.21 at rho 0.50. The benefit fell sharply at more severe settings, with absorption reaching 0.88 at rho 0.75 and 0.91 at rho 1.

A tabletop stress test separated detection from physical control. The learned detector posted AUROC values between 0.98 and 0.996 for rho values from 0.1 through 0.4. At rho 0.8, however, the unshielded, learned-shield and oracle-shield conditions all reached about 0.95 absorption because the robot could not intercept the sliding object. The reset oracle depends on saving and restoring simulator states, which the authors say is unavailable on real robots; transferring the approach to physical hardware remains future work.