Skip to content

fix(sim): make the camera-sensor tests pass reliably on CI (isaacsim / isaacgym / mujoco)#175

Open
juan-g-bonilla wants to merge 1 commit into
mainfrom
dev/gjuaga/isaacsim-camera-teardown-fix
Open

fix(sim): make the camera-sensor tests pass reliably on CI (isaacsim / isaacgym / mujoco)#175
juan-g-bonilla wants to merge 1 commit into
mainfrom
dev/gjuaga/isaacsim-camera-teardown-fix

Conversation

@juan-g-bonilla

@juan-g-bonilla juan-g-bonilla commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Problem

The camera-sensor test suites added in #170 failed across every backend lane. Root causes & fixes:

1. IsaacSim teardown deadlock (the 700s hang). On process exit, Kit's carbOnPluginShutdown deadlocks tearing down the omni.syntheticdata / OmniGraph render-product graph a TiledCamera creates (confirmed with a native py-spy stack). The subprocess never exits, so run_harness's timeout SIGKILLs it — turning a PASS (verdict already written to --result-file) into a spurious timeout. Fix: each camera *_assert.py os._exit(main()) after the sentinel write, bypassing the atexit teardown — the pattern behavior_assert / scene_spawn_assert already use (which is exactly why those isaacsim tests passed).

2. IsaacSim camera didn't follow a moving body / frame never refreshed (unmasked once the hang was fixed: follow, follow_multi_env, actor_mount, obs_pipeline). simulate_at_each_physics_step did sim.step(render=False) + a separate self.render(), but IsaacLab's self.render() only flushes fabric / drives the RTX render at render_mode >= PARTIAL_RENDERING; at NO_GUI_OR_RENDERING it is a no-op, so the TiledCamera returned its stale first frame. Fix: render inline via sim.step(render=render_now), which drives the low-level render regardless of render_mode (robust even if a caller reaches the sim before enable_cameras/headless are set at launch).

3. IsaacSim multi-env misalignment. IsaacSim ignores the passed env_origins and clones envs on its own env_spacing grid; all internal placement uses self.scene.env_origins, so sim.env_origins disagreed with where envs actually were (unlike mujoco/isaacgym, which honor the passed value) and the multi-env harness pinned robots away from their grid-placed panels. Fix: reconcile self.env_origins = self.scene.env_origins in create_envs so sim.env_origins means the same thing on every backend, and pin robots relative to it.

4. Un-actuated-robot drift flake (IsaacGym multi-env geometry/follow; a ~4px IsaacSim geometry offset). The spawned robot is un-actuated, so its root and joints drift during the settle and tilt the mounted camera. Fix: the camera harnesses now re-pin the root, hold the joints rigid at the spawn pose with zero velocity (set_dof_state_tensor_robots), read-back-verify the pose landed, and — since an IsaacSim pose write takes a few physics steps to reach the render — step a few frames before capture.

Also: a ruff PLC0415 per-file-ignore for tests/simulators/*_assert.py (these standalone harnesses defer torch/isaaclab imports until the SimApp is launched, inherent to IsaacSim).

Verification (run locally)

  • isaacsim camera suite: 11/11 twice, plus follow at num_envs 1 and 4.
  • isaacgym camera suite: 12/12, with follow_multi_env and geometry_multi_env each 6/6 across repeated runs (drift flake eliminated).

…isaacgym/mujoco)

The camera-sensor test suites (new in #170) failed across every backend
lane. Each root cause was reproduced on a GPU box before fixing.

1. IsaacSim teardown deadlock (all 11 isaacsim camera tests hung 700s and
   were SIGKILL'd, blowing the blocking docker-build gate from ~12m to
   2h41m). On process exit, Kit's carbOnPluginShutdown deadlocks tearing
   down the omni.syntheticdata / OmniGraph render-product graph a
   TiledCamera creates, so the subprocess never exits and run_harness's
   timeout kills it — turning a PASS (sentinel already written) into a
   spurious timeout. Fix: each camera *_assert.py os._exit(main()) after
   the result-file write, bypassing the atexit teardown (the pattern
   behavior_assert / scene_spawn_assert already use; those isaacsim tests
   passed for exactly this reason). Native py-spy stack confirmed the frame.

2. IsaacSim camera didn't follow a moving body / frame never refreshed
   (once the hang was unmasked: follow, follow_multi_env, actor_mount,
   obs_pipeline). simulate_at_each_physics_step did sim.step(render=False)
   + a separate self.render(), but IsaacLab's self.render() only flushes
   fabric / drives the RTX render at render_mode >= PARTIAL_RENDERING; at
   NO_GUI_OR_RENDERING it is a no-op, so the TiledCamera returned its stale
   first frame. Render inline via sim.step(render=render_now); it drives the
   low-level render regardless of render_mode (robust even if a caller
   reaches the sim before enable_cameras/headless are set at launch).

3. IsaacSim multi-env misalignment. IsaacSim ignores the passed env_origins
   and clones envs on its own env_spacing grid; all internal placement uses
   self.scene.env_origins. Reconcile self.env_origins = self.scene.env_origins
   in create_envs so sim.env_origins reports where envs actually are on every
   backend (mujoco/isaacgym already honor the passed value), and have the
   camera harnesses pin the robot relative to sim.env_origins.

4. Un-actuated-robot drift flake (IsaacGym multi-env geometry/follow; a
   ~4px IsaacSim geometry offset). The spawned robot is un-actuated, so its
   root and joints drift during the settle and tilt the mounted camera. The
   camera harnesses now re-pin the root, hold the joints rigid at the spawn
   pose with zero velocity (set_dof_state_tensor_robots), read-back-verify
   the pose landed, and — since an IsaacSim pose write needs a few physics
   steps to reach the render — step a few frames before capture.

Also: ruff PLC0415 per-file-ignore for tests/simulators/*_assert.py (these
standalone harnesses defer torch/isaaclab imports until the SimApp is
launched, inherent to IsaacSim).

The mujoco lane's remaining failure (mjwarp: "module mujoco_warp has no
attribute create_render_context") is a stale holosoma-mujoco:latest that
predates mujoco_warp 3.10.0, not a code issue — the branch's setup already
pins the right commit + WARP=true, so a :latest rebuild resolves it.

Verified on a GPU box: isaacsim camera suite 11/11 (twice) + follow n=1,4;
isaacgym camera suite 12/12, with follow_multi_env and geometry_multi_env
each 6/6 across repeated runs (flake eliminated).
@juan-g-bonilla
juan-g-bonilla force-pushed the dev/gjuaga/isaacsim-camera-teardown-fix branch from 107e284 to 8633b98 Compare July 18, 2026 01:20
@juan-g-bonilla juan-g-bonilla changed the title fix(isaacsim): hard-exit camera assert harnesses past the Kit teardown deadlock fix(sim): make the camera-sensor tests pass reliably on CI (isaacsim / isaacgym / mujoco) Jul 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant