What Are We Actually Benchmarking in Robot Manipulation?

Public repository scaffold for the paper "What Are We Actually Benchmarking in Robot Manipulation?"

Project website: https://ripl.github.io/manipulation_benchmark_audit/

Purpose

This repository will host public code, diagnostics, result summaries, and artifact documentation for the paper. The current version is a minimal release scaffold while code and artifacts are prepared.

Status

This repository is being prepared for public release. Code, benchmark diagnostics, result summaries, and artifact access instructions are coming soon.

The current repository intentionally contains only lightweight documentation scaffolding. It does not include model weights, datasets, simulator caches, rollout videos, generated result bundles, or private filesystem paths.

Repository Layout

.
├── artifacts/      # Release artifact documentation; files coming soon.
├── diagnostics/    # Benchmark diagnostic descriptions; code coming soon.
├── results/        # Lightweight paper result summaries; files coming soon.
├── LICENSE
└── README.md

What Will Be Released

1. Diagnostic definitions and minimal code needed to run public benchmark audit checks.
2. Lightweight result summaries for paper-facing claims.
3. Artifact documentation with download or reproduction instructions for data that is too large to store in git.

Contact

For questions, contact Tianchong Jiang via GitHub: https://github.com/Tianchong-Jiang.