Bundled RubiksCubeAndBananaTask trajectory drops objects on Isaac Sim 5.0 / Isaac Lab 2.2

[SetasAditya]

Hi, thanks for the recent playback fix. I pulled the latest changes and retested the bundled RubiksCubeAndBananaTask demonstration.

Environment

  - RoboLab commit: 0aef241
  - Isaac Sim: 5.0.0
  - Isaac Lab: 2.2.0
  - GPU: NVIDIA GeForce RTX 5080 Laptop
  - Driver: 580.173.02
  - Ubuntu 24.04

Command

  python examples/run_recorded.py \
    --task RubiksCubeAndBananaTask \
    --headless \
    --validate-states

Observed behavior

• The replay starts from the recorded environment configuration and initial scene state. Both the banana and Rubik’s cube are detected as successfully grasped, but they are subsequently dropped. The episode finishes with score 0.0 and does not reproduce the recorded success.
• State validation shows that the first simulated state is initially very close to the recording, but the trajectory begins diverging shortly afterward. I also tested enabling enhanced determinism and rebuilding the PhysX contact state before starting the actions; neither prevented the drops.

Expected behavior

The bundled trajectory, recorded with Isaac Sim 5.0 / Isaac Lab 2.2, should reproduce the successful episode on the same software stack.

Could you confirm whether the trajectory depends on the recording machine’s GPU/driver or any runtime settings not stored in env_cfg.json? I can attach the replay MP4, event log, and state-drift output.

[WANGXinyiLinda]

I also noticed this issue -- it appears that there are additional uncontrolled randomness on the renderer side so you cannot exactly reproduce a policy's outcome

[xuningy]

Hi @SetasAditya , thanks for the detailed report.

We reproduced your setup on the machine the bundled trajectory was recorded on (RTX 6000 Ada, driver 535, Isaac Sim 5.0 / Isaac Lab 2.2.0). The replay is bit-exact there: max drift 0.0000 across all 648 steps. Since your software stack is identical, the difference comes down to hardware.

This is a known limitation: determinism is not guaranteed across different hardware. With a different GPU or driver, recorded trajectories can behave differently on a different machine — small floating-point differences compound through the contact solver, which is why your replay tracks closely at first and then diverges around the grasp. Unfortunately, no runtime setting or env_cfg.json field can compensate for this. See the Isaac Lab reproducibility notes: https://isaac-sim.github.io/IsaacLab/main/source/features/reproducibility.html
We bundle a trajectory replay just for verification purposes, but generally the scholasticism is fine for evaluation since the real-world is inherently noisy.

We'll update the checklist in docs/replay.md to state that reproducing a recorded outcome also requires the same GPU architecture and driver, and add GPU/driver info to the recording's provenance attributes so playback can warn on mismatch.