Replace legacy architecture with simplified benchmark

.github/workflows/ci.yml

@@ -4,74 +4,15 @@ on:
   push:
   pull_request:
 
-permissions:
-  contents: read
-
 jobs:
-  python:
-    runs-on: ubuntu-24.04
+  test:
+    runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
         with:
           python-version: "3.12"
-          cache: pip
-      - run: python -m pip install --upgrade pip
-      - run: python -m pip install -e '.[test,qualification]' ruff
-      - run: ruff format --check src tests tools setup.py modal_app.py qualification_modal.py
-      - run: ruff check src tests tools setup.py modal_app.py qualification_modal.py
+      - run: python -m pip install -e '.[dev]'
+      - run: ruff check src tests
+      - run: ruff format --check src tests
       - run: pytest
-      - run: python tools/freeze_manifests.py
-      - run: git diff --exit-code -- data/dev-v1.json data/ranked-v1.json
-
-  minimum-dependencies:
-    runs-on: ubuntu-24.04
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.12"
-          cache: pip
-      - run: python -m pip install -e . numpy==2.0.2 scipy==1.14.1 pytest
-      - run: pytest tests/test_dataset.py tests/test_pilot_data.py tests/test_ns_dominance.py tests/test_ns_refinement.py
-
-  installed-wheel:
-    runs-on: ubuntu-24.04
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.12"
-          cache: pip
-      - run: python -m pip install build
-      - run: python -m build
-      - name: Exercise installed resources outside the checkout
-        shell: bash
-        run: |
-          python -m venv "$RUNNER_TEMP/wheel-venv"
-          "$RUNNER_TEMP/wheel-venv/bin/python" -m pip install dist/*.whl
-          cd "$RUNNER_TEMP"
-          "$RUNNER_TEMP/wheel-venv/bin/linear-solver-bench" dataset summary
-          "$RUNNER_TEMP/wheel-venv/bin/python" -c 'from linear_solver_bench.paths import native_dir; assert (native_dir() / "src/driver.cpp").is_file(); assert (native_dir() / "include/nsl_hypre_solver.h").is_file()'
-
-  native-smoke:
-    runs-on: ubuntu-24.04
-    timeout-minutes: 20
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.12"
-          cache: pip
-      - run: sudo apt-get update && sudo apt-get install -y ninja-build
-      - run: python -m pip install build
-      - run: python -m build --wheel
-      - run: python -m pip install dist/*.whl
-      - name: Build and validate the native runtime from the installed wheel
-        shell: bash
-        run: |
-          cd "$RUNNER_TEMP"
-          linear-solver-bench runtime build --output runtime --jobs 2
-          linear-solver-bench runtime build --output runtime-check --jobs 2
-          python -c 'import json; a=json.load(open("runtime/manifest.json")); b=json.load(open("runtime-check/manifest.json")); assert a == b, "clean runtime builds must be identical"'
-          linear-solver-bench candidate validate "$GITHUB_WORKSPACE/submissions/starter.c" --runtime runtime

.gitignore

@@ -1,17 +1,11 @@
 .venv/
-.uv-cache/
-__pycache__/
 .pytest_cache/
 .ruff_cache/
-.coverage
-htmlcov/
+__pycache__/
 *.egg-info/
 build/
 dist/
 results/
-workspace/
-secrets/
-data/prepared/
-data/downloads/
+.cache/
 .env
 .DS_Store

BENCHMARK.md

@@ -0,0 +1,80 @@
+# Benchmark contract
+
+## Candidate interface
+
+A candidate is one UTF-8 C file of at most 65,536 bytes. It includes exactly
+`HYPRE_parcsr_ls.h` and exports:
+
+```c
+HYPRE_Int solver_create(HYPRE_Solver *solver,
+                        HYPRE_Real relative_tolerance,
+                        HYPRE_Real absolute_tolerance);
+```
+
+The returned object provides setup, solve, and destroy callbacks. Candidates may
+assemble a solver from the HYPRE operations allowlisted by the benchmark. Stock
+HYPRE Krylov outer solvers are reserved for the fixed reference. Candidates may
+not perform I/O, networking, process creation, dynamic loading, or timing. Inputs
+must not be modified.
+
+Submissions use a reviewed-source threat model. The compiler and symbol audit
+enforce the ordinary C contract; they are not a security boundary against hostile
+inline assembly or raw system calls. Modal still isolates the evaluation from the
+host and blocks sandbox networking.
+
+## Paired execution
+
+The fixed reference and candidate are built against the same pinned HYPRE runtime
+inside one environment. Each numerical case is run in two fresh processes:
+
+1. Run and verify the fixed reference.
+2. Run and verify the candidate.
+3. Compute `speedup = reference_seconds / candidate_seconds` if both pass.
+
+A reference failure invalidates the evaluation. A candidate failure is recorded
+with `passed: false` and no speedup. Failed or omitted cases never disappear from
+an aggregate.
+
+The native monotonic clock measures factory creation, setup, and solve. It excludes
+compilation, input loading, transport, destruction, and trusted verification.
+The reference is measured again for every candidate; timings are never reused
+between sandboxes. Each solver process has a 600-second wall-time limit.
+
+The complete-split score is the geometric mean of scored-case speedups, provided
+the candidate passes every scored and control case. Partial evaluations and
+evaluations with failures have no aggregate score.
+
+## Accuracy
+
+For residual `r = b - A x`, the trusted evaluator computes normwise and
+componentwise backward error and the relative residual. NS cases also compare
+against the evaluator-held manufactured target.
+
+NS requires:
+
+- normwise backward error ≤ `1e-10`;
+- componentwise backward error ≤ `1e-8`;
+- relative L2 and Linf forward errors ≤ `1e-5`.
+
+FLASH requires:
+
+- normwise backward error ≤ `max(tolerance, 20 * float64_epsilon)`;
+- relative residual ≤ `tolerance * (1 + 1e-6)`.
+
+Both families require a finite solution of the correct size, a successful solver
+status, and unchanged inputs.
+
+## Modal venue
+
+Modal is the default and the only official scoring venue. Local runs are useful
+development checks, but their reports are marked `official: false`.
+
+The Modal venue uses a versioned immutable image, exact CPU and memory
+request/limit pairs, and a network-disabled sandbox. It compiles the two solvers
+once and transfers one public case at a time. Only the numerical `CaseInput`
+crosses the execution boundary; targets and correctness remain on the trusted
+client.
+
+Changing the reference source, HYPRE runtime, native driver, image, resources,
+case membership, timing boundary, or accuracy rules creates a new venue or
+benchmark version.

CITATION.cff

@@ -1,9 +1,8 @@
 cff-version: 1.2.0
-message: "If you use this benchmark, please cite this software and the source matrices."
+message: "If you use this benchmark, please cite this software and its source matrices."
 title: "LinearSolverBench"
 type: software
 version: 0.1.0
-date-released: 2026-08-29
 authors:
   - name: Autodidakt
 license: Apache-2.0

CONTRIBUTING.md

@@ -1,35 +1,16 @@
 # Contributing
 
-Changes should preserve the benchmark's scientific identity and keep the
-trusted path small. Bug fixes, portability improvements, new public tests, and
-proposals for future tracks are welcome.
-
-## Development checks
+Keep the participant workflow small and readable. New abstractions need more than
+one real use before they are introduced.
 
 ```bash
-uv venv --python 3.12
-uv pip install -e '.[dev]'
-ruff format --check .
-ruff check .
+python -m pip install -e '.[dev]'
+ruff format --check src tests
+ruff check src tests
 pytest
 ```
 
-To exercise the native boundary, build the pinned runtime and audit the starter
-submission:
-
-```bash
-linear-solver-bench runtime build --output build/runtime
-linear-solver-bench candidate validate submissions/starter.c --runtime build/runtime
-```
-
-Run `python tools/freeze_manifests.py` after intentionally changing selection
-logic. Both generated manifest digests and the reason for a dataset-version
-change must be reviewed. Do not rewrite a published manifest in place; create a
-new benchmark version.
-
-## Contract changes
-
-A pull request that changes the dataset, accuracy gates, timed boundary,
-submission ABI, resource limits, HYPRE identity, or ranking rule must call that
-out explicitly. Such changes normally belong in a new track or major benchmark
-version so existing results remain comparable.
+Changes to numerical inputs, accuracy, the native ABI, the fixed reference,
+timing, or Modal resources change benchmark results and require an explicit
+version decision. Keep private test data, credentials, full private reports, and
+dataset-construction artifacts outside this repository.