Add benchmarking for Relation Solver

[zhx06]

Summary

Add benchmarking for Relation Solver

Detailed description

• Adds reproducible performance measurements for relation solving and end-to-end object placement.
• Covers bbox/mesh collision, object and batch scaling, multiple GPUs, robot variants, and environment build/reset.
• Prints results by default, with optional JSON and CSV reports for comparison and capacity planning.
• solver measures one seeded RelationSolver.solve() call; placer measures ObjectPlacer.place() including candidate generation, retries, and validation.

Local benchmark results

Command:

/isaac-sim/python.sh isaaclab_arena_examples/relations/relation_solver_benchmark.py --suite comprehensive

Hardware: NVIDIA RTX 6000 Ada (48 GiB), PyTorch 2.10.0, CUDA 12.8
Runtime: 456 seconds

Batch-size Scaling

Fixed Workload: 3 objects.

• BBox solver: 0.657 layouts/s (1), 5.437 (8), 21.441 (32), 84.976 (128); all passed.
• BBox placer: 0.655 layouts/s (1), 5.322 (8), 20.113 (32), 66.692 (128); all passed.
• Mesh solver: 0.737 layouts/s (1, passed), 5.253 (8), 21.233 (32), 83.169 (128); batches 8–128 exceeded the final-loss threshold.
• Mesh placer: 0.577 layouts/s (1), 4.202 (8), 14.514 (32), 36.275 (128); all passed.

Object-count Scaling

Fixed workload: batch size 1.

• BBox solver: 1.522 s (3 objects), 2.164 s (5, 1.42×), 4.010 s (10, 2.63×); all passed.
• BBox placer: 1.527 s (3), 2.184 s (5, 1.43×), 4.037 s (10, 2.64×); all passed.
• Mesh solver: 1.358 s (3, passed), 2.264 s (5, 1.67×), 4.115 s (10, 3.03×); 5 and 10 objects exceeded the final-loss threshold.
• Mesh placer: 1.735 s (3), 2.576 s (5, 1.49×), 5.185 s (10, 2.99×); all passed.

Environment Bring-up

• Banana-in-bowl BBox: 2.535 s with robot, 2.416 s without robot.
• Kitchen BBox: 5.180 s with robot, 4.597 s without robot.
• Kitchen mesh: 10.651 s with robot, 8.822 s without robot.
• Banana-in-bowl mesh was rejected because this BBox-oriented environment has no usable mesh collision pair.

[greptile-apps]

Greptile Summary

Adds a relation-solver benchmarking framework with synthetic solver/placer workloads, environment build/reset measurements, multi-GPU execution, capacity search, and JSON/CSV reporting.

• Introduces benchmark scenarios, measurements, metadata, reporting, and runtime utilities.
• Adds bbox/mesh, object-count, environment-count, robot, and multi-GPU benchmark matrices.
• Exposes per-environment solver convergence and collision-pair metrics.
• Adds unit and persistent-simulation benchmark coverage.

Confidence Score: 4/5

The environment benchmark’s printed latency must be corrected before merging because it reports build-only time alongside throughput calculated from build plus reset.

Environment measurements capture build and reset separately, but the terminal table presents only build latency under a generic median column while deriving throughput from the combined duration, making the default report internally inconsistent.

Files Needing Attention: isaaclab_arena/relations/benchmark/reporting.py

Important Files Changed

Filename	Overview
isaaclab_arena/relations/benchmark/environment.py	Adds repeated environment construction/reset timing, correctness gating, cleanup, and live-memory sampling.
isaaclab_arena/relations/benchmark/solver.py	Adds deterministic synthetic workloads and solver/placer timing with correctness and memory metrics.
isaaclab_arena/relations/benchmark/reporting.py	Adds run envelopes, capacity search, scaling summaries, and report writers, but the terminal environment latency is inconsistent with its throughput.
isaaclab_arena/relations/benchmark/models.py	Defines validated benchmark scenarios, measurements, device metadata, serialization, and distributed-run completeness checks.
isaaclab_arena/relations/relation_solver.py	Changes convergence to require every environment below threshold, refreshes final loss, and exposes separate collision-pair counts.
isaaclab_arena_examples/relations/relation_solver_benchmark.py	Adds the runnable CLI, GPU worker orchestration, capacity probing, report generation, and environment benchmark launching.
isaaclab_arena/tests/test_relation_solver_benchmark.py	Adds broad coverage for scenarios, timing, convergence, serialization, reports, distributed workers, and CLI behavior.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  CLI[Benchmark CLI] --> Scenarios[Scenario matrix]
  Scenarios --> Solver[RelationSolver target]
  Scenarios --> Placer[ObjectPlacer target]
  Scenarios --> Environment[Environment build/reset target]
  CLI --> Workers[Per-GPU workers]
  Workers --> Solver
  Workers --> Placer
  Workers --> Environment
  Solver --> Measurements[Benchmark measurements]
  Placer --> Measurements
  Environment --> Measurements
  Measurements --> Table[Terminal table]
  Measurements --> JSON[JSON report]
  Measurements --> CSV[CSV report]
  Measurements --> Capacity[Capacity search]

Loading

_{Reviews (1): Last reviewed commit: "add software meta" | Re-trigger Greptile}

[greptile-apps]

Environment latency excludes reset

When an environment benchmark is printed, median_ms contains only build_ms while throughput uses build_ms + reset_ms, causing the default report to understate end-to-end latency and show a latency that cannot be reconciled with its env/s value.

Suggested change

	median_ms = {
	"solver": result.solve_ms,
	"placer": result.place_ms,
	"environment": result.build_ms,
	}[result.target]
	median_ms = {
	"solver": result.solve_ms,
	"placer": result.place_ms,
	"environment": (
	result.build_ms + result.reset_ms
	if result.build_ms is not None and result.reset_ms is not None
	else None
	),
	}[result.target]

[arena-review-bot]

🟡 Convergence semantics changed for every solver caller

This switches the stop condition from aggregate mean loss to per-env max loss (and adds a full _compute_total_loss after the loop), so every batched RelationSolver.solve() in the framework — not just benchmarks — now keeps iterating until the worst environment converges and pays an extra forward pass per solve. For num_envs > 1 that generally means more iterations and shifted placement/reset behavior on the default path. Is this intended to ride along in a benchmarking PR? If so, it seems worth calling out in the PR description, since it changes behavior for all placement callers, not only the benchmark.

[arena-review-bot]

🟡 Sim-spinning benchmark inside the sim-free relations package

run_environment_benchmark builds a full Arena env, resets it, and closes it — real sim lifecycle orchestration — and it's exported from isaaclab_arena.relations.benchmark, a domain package that's otherwise kept simulator-free. The deferred imports keep the import graph clean, but the responsibility of owning a live env belongs to the runtime/example layer, not the relations domain. Would a top-level benchmarks area (or the examples package) be a better home for this piece, so the relations public API doesn't gain a sim entry point?

[arena-review-bot]

🤖 Isaac Lab-Arena Review Bot

Summary

This PR adds a relation-solver benchmarking subsystem: a sim-free measurement library under isaaclab_arena/relations/benchmark/ (scenarios, solver/placer/environment runners, reporting) plus a 531-line multi-GPU CLI in the examples package. The measurement code is clean, well-typed, and thoroughly unit-tested. My two concerns are about scope and layering, not correctness of the timing logic.

Design, Boundaries & Scope

• Core solver behavior change bundled in a benchmark PR. relation_solver.py changes convergence from aggregate mean-loss to per-env max-loss and adds a post-loop _compute_total_loss. This affects every RelationSolver.solve() caller (all batched object placement), not just the benchmark — for num_envs > 1 it generally runs more iterations and pays an extra forward pass per solve. It looks like a deliberate improvement, but changing the default placement path deserves a call-out in the PR description so downstream users aren't surprised.
• Sim-orchestration in the sim-free relations package. run_environment_benchmark builds/resets/closes a live Arena env and is exported from isaaclab_arena.relations.benchmark. Deferred imports keep the import graph clean, but the responsibility of owning a live env belongs to the runtime/example layer. Worth asking whether this piece (and the heavier harness — multi-GPU workers, capacity search, run manifests, git/CSV/JSON tooling) should live in a benchmarks/examples area rather than the shipped core package, keeping relations/ lean.

Findings

🟡 Warning: relation_solver.py:300 — convergence criterion changed to per-env max loss for all solver callers; call out the default-path change.
🟡 Warning: benchmark/environment.py:113 — sim-spinning env benchmark exported from the otherwise sim-free relations domain package; consider a better home.

(Note: Greptile already flagged reporting.py:168 — the results table shows build_ms as median_ms for environment targets while throughput uses build_ms + reset_ms, so the two columns can't be reconciled. Worth fixing.)

Test Coverage

Excellent unit coverage — scenario validation, seeding/determinism, timing with injected clocks, multi-GPU aggregation, capacity search, report round-trips, and the worker CLI (with_subprocess). The sim path uses the persistent-simulation-app inner/outer pattern with deferred imports. One question: test_environment_benchmark_builds_robot_variants_sequentially carries no marker (Phase 1) — do the banana_in_bowl / kitchen specs it builds instantiate any cameras? If so it needs with_cameras.

Verdict

Minor fixes needed