hero
RecovaAI Research Design

Agent-guided
failure recovery

For autonomous robotic manipulation

Develop task execution and recovery in a digital twin.
Verify and refine both through physical experience.
Agent panel: illustrative
Isabella Liu1, An-Chieh Cheng1, Johan Bjorck3, Zhiding Yu3, Hongxu Yin3
Jan Kautz3, Linxi Fan3, Yuke Zhu2,3, Sifei Liu3
1 University of California, San Diego2 University of Texas at Austin3 NVIDIA

Motivation

Task instruction

“Stack the rings in size order.”

Task execution
Real robot stacking rings on a peg in a nominal task rollout
Failure state
Actual failure state: a green ring is tilted and caught near the top of the peg.
Recovery
Real recovery of the pictured failure: the gripper presses the caught ring onto the stack
02

Motivation

A twin makes repeated exploration practical; physical rollouts verify recovery and supply experience for learning.

01Explore in the twin

Matched MuJoCo ring-stacking replayTwin replay · 2×
Reproduce failures and test
candidate corrections.

02Verify on the robot

Matched physical ring-stacking recordingReal camera · 2×
Check restoration under real
observations and contact dynamics.

03Retain the experience

TaskRecovery πθρφ
Refine both policies; use new
failures to develop further skills.
03

Related work: recovery

RecoveryChaining

IROS 2025

Learn recovery around given
task controllers.

Original RecoveryChaining paper figure: recovery from a shelf collision and slip, followed by return to task control

RecovaDevelop and improve task execution
and recovery together.

RACER

ICRA 2025

One actor handles task execution
and recovery under language guidance.

Original RACER architecture: a vision-language supervisor gives instructions to a shared visuomotor actor

RecovaA separate recovery policy learns
the correction each failure needs.

04

Method overview

The agent connects twin development with verified execution and separate task and recovery updates.

Recova workflow Recova builds and explores a digital twin. Simulation recovery knowledge prepares recovery skills. A transparent, outlined Real robot rollouts group contains the base policy, recovery skills, and Human. Failures invoke recovery skills, which restore the scene and resume the base policy. For unresolved failures, Recova requests human assistance, shown by an upward arrow from Recova to Human. Human recovery demonstrations improve recovery skills. A circular success-data loop improves the base policy. Real robot rollouts Build &explore Delegate Sim recovery knowledge Failure Unresolved Resume Success data Recovery demonstration Digital twin Base policy Recovery skills Human Recova
05

Real → sim

Fit object poses and contact parameters so that replayed robot trajectories align with real camera views.

Camera
observations

Camera calibrationwhen available

Robot
trajectories

Digital twin

Same recorded moment
Real camera
Real camera2×
MuJoCo replay
MuJoCo replay2×
01

Replay trajectory

02

Compare views

03

Refine poses + contact

06

Simulation development

Simulation develops task execution and recovery together, with separate policy supervision.

Twin recoveryTwin recovery
Clear the grasp corridor, then resume stacking.

Attempt the task

Successful task rollouts → πθ

Diagnose the failure

Here, a neighboring ring blocks the grasp.

Test a correction

z: “Move the neighboring ring aside.”

Retain the skill

Reusable program + successful rollout → ρφ

07

Programs and training data

The program retains the strategy; successful rollouts train the recovery policy.

Reusable program

locate(ring)
move_above(ring)
press_down()
verify_restoration()

Code-as-policy skill library

Successful rollout

Beginning of a simulated recovery rolloutEnd of a simulated recovery rollout

“Push the stuck ring down”

Instruction-labeled rollouts supply training data for recovery policy rho. The learning curve is a schematic training symbol. RolloutsTrainρ
08

Recovery skills

Ring-stacking failures require different corrections: recover a grasp, realign the ring, or clear its path.

Recover a missed grasp
Recover a missed grasp
Realign the ring
Realign the ring
Level a caught ring
Level a caught ring
Clear a neighboring ring
Clear a neighboring ring
Retrieve an unreachable ring
Retrieve an unreachable ring
Move the idle arm clear
Move the idle arm clear
09

Sim → real

Twin rollouts initialize task and recovery policies; physical experience refines both.

Simulated recovery nudges a misplaced pencil into its tray
Twin recovery rollout
Instruction-labeled rollouts supply training data for recovery policy rho. The learning curve is a schematic training symbol. RolloutsTrainρ
Learned physical recovery realigns and reinserts the pencil
Learned physical recovery
10

Agent monitor

The agent selects and verifies recovery; learned policies execute the motion.

The green ring is caught near the top of the peg
Recova agentIllustrative trace
Requirement
Stack rings in order
Completion
Task unfinished
Intervention Restoration

Diagnose the obstruction

Ring caught on the peg

Agent command Select a known recovery

“Push the stuck ring down”

Agent selects · policy executes

11

Real-world rollout

Monitor execution, recover or request help, then learn from the resulting experience.

Recova real-robot rollout and learning loop, adapted from Figure 2B The task policy is monitored by Recova, which selects a recovery or requests a human demonstration. Verified recovery allows the task to resume. Between rounds, task and recovery experience update separate policies. A new recovery instruction is demonstrated, trained, and registered; it also guides program development in the twin. recover verified unresolved intact scene, stalled task Task policy “Stack rings in size order” → Recova monitor · decide · verify Recovery policy “Push the stuck ring down” → Human demo Task or recovery Update task policy Successes + task demos Update recovery policy Verified recoveries + recovery demos Betweenrounds New instructionz Train → Register Develop in the twin Program + test the new correction available next round

Task execution → recovery → verified task restart

Human demonstrations resolve stalled tasks and unresolved failures

Between rounds, task and recovery experience update separate policies

New instructions become skills through demonstration, training, and registration

12

Real-world DAgger

Recova coordinates four workstations; task and recovery policies improve between collection rounds.

Parallel real-robot data collection
RecovaMonitor
and delegate
Run task policy
Invoke recovery
Request a human
Collect
data
Task data → task policyRecovery data → recovery policy

Station 1Waiting to start a rollout.

YAM 1 top camera
YAM 2 top camera
YAM 3 top camera
YAM 4 top camera
13

Learning over rounds

Four collection rounds on mahjong draw

Task-policy completionHuman takeover
025507510012.5%57.1%75%85.7%87.5%28.6%25%0%Round 1Round 2Round 3Round 4Collection episodes (%)
14

Physical evaluation

Four physical tasks · 20 trials per task and configuration

Base task policy+ DAgger+ Recovery
0255075100207085Pencil box357590Stack rings258590Draw tile158085Discard tileTask success (%)
Mean success23.8%→77.5%→87.5%
15

Simulation benchmarks

MolmoAct2 task policy + programmatic recovery · published baseline results

LIBERO-Pro

Six settings · mean task success

ASPIRE
71.7%
Recova
78.8%

+7.1 percentage points

MolmoSpaces

Four categories · mean task success

RATs
38.0%
Recova
64.9%

+26.9 percentage points

16

Real-world recoveries

Learned corrections restore task progress across four physical tasks.

Physical failure and recovery example: push down a caught ring
Push down a caught ring
Physical failure and recovery example: reinsert the pencil
Reinsert the pencil
Physical failure and recovery example: restore the tile wall
Restore the tile wall
Physical failure and recovery example: remove a misplaced ring
Remove a misplaced ring
Physical failure and recovery example: reinsert the tray
Reinsert the tray
Physical failure and recovery example: stand the hand tiles
Stand the hand tiles
17

Future directions

Automatic registration

New skillEvidenceRegistry
When is a new skill reliable
enough to deploy?

Sharing across tasks

One correctionTask ATask BTask C
Can the same correction
work in another task?
18
mahjong-film
RecovaConclusion

Task execution and recovery
learn together.

01

Develop in a twin

Learn task execution
and recovery together.

02

Verify on the robot

Check that recovery
restores task progress.

03

Improve with experience

Refine policies and turn
new failures into corrections.

LIBERO-Pro

Task success (%) · 50 initial states per task · published baseline results

MethodObject
position		Object
task		Goal
position		Goal
task		Spatial
position		Spatial
task		Mean
CaP-Agent027.031.029.016.013.023.023.2
RATs61.063.043.036.029.031.043.8
ASPIRE98.095.081.045.051.060.071.7
π₀0.00.00.00.00.00.00.0
π₀.₅17.01.038.00.020.01.012.8
MolmoAct2-LIBERO52.024.020.021.027.057.033.5
Recova91.287.282.267.062.283.078.8
B1

MolmoSpaces

Task success (%) · four simulation categories · published baseline results

MethodPickPick & placeOpenCloseMean
CaP-Agent023.011.014.036.021.0
RATs37.022.020.073.038.0
π₀14.07.010.047.019.5
π₀-FAST20.010.09.035.018.5
π₀.₅28.010.018.062.029.5
MolmoAct2-DROID41.032.03.867.036.0
Recova78.740.556.583.964.9
B2

Recovery examples

Learned recoveries on two physical tasks

Real robot: pencil-recovery
Reinsert the pencil tray
Real robot: draw-recovery
Return a displaced tile
B3

References

RecoveryChaining

Learning Local Recovery Policies for Robust Manipulation.

Vats et al. · IROS 2025

RACER

Rich Language-Guided Failure Recovery Policies for Imitation Learning.

Dai et al. · ICRA 2025

ReSYNC

Recover, Discover, Plan: Learning Skills and Concepts from Robot Failures.

Li et al. · CoRL 2026

SIRIUS

Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment.

Liu et al. · RSS 2023
B4