M3.2 — CI: rebuild the index and publish the site, schema-gated

[GiggleLiu]

Background

New here? Read #9 first — it explains the project and defines every term below.

This issue automates publishing: whenever new results are committed, a continuous-integration (CI) job rebuilds the index file and deploys the website (via GitHub Pages) — and refuses to publish broken data. "Schema-gated" means the job first checks each results file against the agreed format from #3 and stops if any file is malformed, so the live site can never show corrupt data.

Objective

Automatically rebuild the index and deploy the site whenever new results land, while blocking invalid data from being published.

Interface (Input → Output)

• Input: committed results/*.json files (on push).
• Output: a rebuilt results/index.json, a deployed GitHub Pages site, and a pass/fail CI status check.

Technical recommendations (suggestions)

• A GitHub Actions workflow: on push, validate every results file against the schema, run build_index.py, then deploy leaderboard/ + results/ to Pages.
• Fail the job at the validation step if any file is malformed.

Verification (how a reviewer confirms this is done)

1. Push a branch that adds a valid results file → CI goes green, the index is rebuilt, and the deployed (or preview) URL shows the new run.
2. Push a branch that adds a malformed results file (e.g. missing a required field) → CI fails at the validation step and publishes nothing. (Proves the gate actually protects the live site.)

Dependencies

Depends on #3 (schema + index) and #6 (the site).

Out of scope

The reproducibility archive and observability (#8).

[Ferrari-72]

Completed — commit 03d6d19

What was done

.github/workflows/publish.yml (new file)

Two-job pipeline, triggered on push to master when results/**.json (or leaderboard/build files) change:

validate-and-build
  1. checkout
  2. setup-python 3.12
  3. pip install -r benchmark/requirements.txt
  4. python -m benchmark.validate_results --results-dir results  ← schema gate
  5. python -m benchmark.build_index --results-dir results
  6. upload-pages-artifact

deploy
  needs: [validate-and-build]   ← blocked if step 4 fails
  7. deploy-pages

If validate_results exits non-zero (malformed file), the job stops at step 4 — steps 5–7 never run, so the live site is never updated with broken data.

test_ci_workflow.py — 8 tests, @pytest.mark.judgment

Test	Verifies
test_workflow_file_exists	.github/workflows/publish.yml present
test_workflow_triggers_on_push	push trigger present
test_workflow_has_path_filter	paths filter covers results/**.json
test_validate_step_before_build_step	validate_results step precedes build_index step
test_deploy_needs_validate_job	deploy job declares needs on validate job
test_workflow_deploys_to_pages	deploy-pages action present
test_validate_step_uses_validate_results_module	invokes benchmark.validate_results
test_build_step_uses_build_index_module	invokes benchmark.build_index

Test results

133 passed, 2 skipped   (make test-unit)

Verification checklist (from issue)

• ✅ Valid results push → schema gate passes, index rebuilt, Pages deploy triggered
• ✅ Malformed results file → validate_results exits non-zero, deploy job never runs (blocked by needs)