fix(sim): make the camera-sensor tests pass reliably on CI (isaacsim / isaacgym / mujoco)#175
Open
juan-g-bonilla wants to merge 1 commit into
Open
fix(sim): make the camera-sensor tests pass reliably on CI (isaacsim / isaacgym / mujoco)#175juan-g-bonilla wants to merge 1 commit into
juan-g-bonilla wants to merge 1 commit into
Conversation
…isaacgym/mujoco) The camera-sensor test suites (new in #170) failed across every backend lane. Each root cause was reproduced on a GPU box before fixing. 1. IsaacSim teardown deadlock (all 11 isaacsim camera tests hung 700s and were SIGKILL'd, blowing the blocking docker-build gate from ~12m to 2h41m). On process exit, Kit's carbOnPluginShutdown deadlocks tearing down the omni.syntheticdata / OmniGraph render-product graph a TiledCamera creates, so the subprocess never exits and run_harness's timeout kills it — turning a PASS (sentinel already written) into a spurious timeout. Fix: each camera *_assert.py os._exit(main()) after the result-file write, bypassing the atexit teardown (the pattern behavior_assert / scene_spawn_assert already use; those isaacsim tests passed for exactly this reason). Native py-spy stack confirmed the frame. 2. IsaacSim camera didn't follow a moving body / frame never refreshed (once the hang was unmasked: follow, follow_multi_env, actor_mount, obs_pipeline). simulate_at_each_physics_step did sim.step(render=False) + a separate self.render(), but IsaacLab's self.render() only flushes fabric / drives the RTX render at render_mode >= PARTIAL_RENDERING; at NO_GUI_OR_RENDERING it is a no-op, so the TiledCamera returned its stale first frame. Render inline via sim.step(render=render_now); it drives the low-level render regardless of render_mode (robust even if a caller reaches the sim before enable_cameras/headless are set at launch). 3. IsaacSim multi-env misalignment. IsaacSim ignores the passed env_origins and clones envs on its own env_spacing grid; all internal placement uses self.scene.env_origins. Reconcile self.env_origins = self.scene.env_origins in create_envs so sim.env_origins reports where envs actually are on every backend (mujoco/isaacgym already honor the passed value), and have the camera harnesses pin the robot relative to sim.env_origins. 4. Un-actuated-robot drift flake (IsaacGym multi-env geometry/follow; a ~4px IsaacSim geometry offset). The spawned robot is un-actuated, so its root and joints drift during the settle and tilt the mounted camera. The camera harnesses now re-pin the root, hold the joints rigid at the spawn pose with zero velocity (set_dof_state_tensor_robots), read-back-verify the pose landed, and — since an IsaacSim pose write needs a few physics steps to reach the render — step a few frames before capture. Also: ruff PLC0415 per-file-ignore for tests/simulators/*_assert.py (these standalone harnesses defer torch/isaaclab imports until the SimApp is launched, inherent to IsaacSim). The mujoco lane's remaining failure (mjwarp: "module mujoco_warp has no attribute create_render_context") is a stale holosoma-mujoco:latest that predates mujoco_warp 3.10.0, not a code issue — the branch's setup already pins the right commit + WARP=true, so a :latest rebuild resolves it. Verified on a GPU box: isaacsim camera suite 11/11 (twice) + follow n=1,4; isaacgym camera suite 12/12, with follow_multi_env and geometry_multi_env each 6/6 across repeated runs (flake eliminated).
juan-g-bonilla
force-pushed
the
dev/gjuaga/isaacsim-camera-teardown-fix
branch
from
July 18, 2026 01:20
107e284 to
8633b98
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The camera-sensor test suites added in #170 failed across every backend lane. Root causes & fixes:
1. IsaacSim teardown deadlock (the 700s hang). On process exit, Kit's
carbOnPluginShutdowndeadlocks tearing down theomni.syntheticdata/ OmniGraph render-product graph aTiledCameracreates (confirmed with a native py-spy stack). The subprocess never exits, sorun_harness's timeout SIGKILLs it — turning a PASS (verdict already written to--result-file) into a spurious timeout. Fix: each camera*_assert.pyos._exit(main())after the sentinel write, bypassing the atexit teardown — the patternbehavior_assert/scene_spawn_assertalready use (which is exactly why those isaacsim tests passed).2. IsaacSim camera didn't follow a moving body / frame never refreshed (unmasked once the hang was fixed:
follow,follow_multi_env,actor_mount,obs_pipeline).simulate_at_each_physics_stepdidsim.step(render=False)+ a separateself.render(), but IsaacLab'sself.render()only flushes fabric / drives the RTX render atrender_mode >= PARTIAL_RENDERING; atNO_GUI_OR_RENDERINGit is a no-op, so theTiledCamerareturned its stale first frame. Fix: render inline viasim.step(render=render_now), which drives the low-level render regardless ofrender_mode(robust even if a caller reaches the sim beforeenable_cameras/headlessare set at launch).3. IsaacSim multi-env misalignment. IsaacSim ignores the passed
env_originsand clones envs on its ownenv_spacinggrid; all internal placement usesself.scene.env_origins, sosim.env_originsdisagreed with where envs actually were (unlike mujoco/isaacgym, which honor the passed value) and the multi-env harness pinned robots away from their grid-placed panels. Fix: reconcileself.env_origins = self.scene.env_originsincreate_envssosim.env_originsmeans the same thing on every backend, and pin robots relative to it.4. Un-actuated-robot drift flake (IsaacGym multi-env
geometry/follow; a ~4px IsaacSim geometry offset). The spawned robot is un-actuated, so its root and joints drift during the settle and tilt the mounted camera. Fix: the camera harnesses now re-pin the root, hold the joints rigid at the spawn pose with zero velocity (set_dof_state_tensor_robots), read-back-verify the pose landed, and — since an IsaacSim pose write takes a few physics steps to reach the render — step a few frames before capture.Also: a ruff
PLC0415per-file-ignore fortests/simulators/*_assert.py(these standalone harnesses defertorch/isaaclabimports until the SimApp is launched, inherent to IsaacSim).Verification (run locally)
followat num_envs 1 and 4.follow_multi_envandgeometry_multi_enveach 6/6 across repeated runs (drift flake eliminated).