diff --git a/README.md b/README.md index 0ad98f53..34a5bcf0 100644 --- a/README.md +++ b/README.md @@ -65,26 +65,22 @@ segment was recomputed bit for bit on a second machine. The other terms are defi ollama run ryoari/openverifiablellm "Tell me about the Moon." ``` -**2. Consistency check, no GPU (10 minutes).** Needs Python 3.12 or newer, `git`, `openssl`, -`sha256sum` and network access. On macOS, `sha256sum` comes with `brew install coreutils` (as -`gsha256sum`). +**2. Consistency check, no GPU (about 2 minutes).** Needs Python 3.12 or newer, `git`, `openssl` +and network access. One command sets up a CPU-only Python environment, downloads the release without +its checkpoints, checks the downloaded files against their hash list, and checks the record and the +weights: ```bash -pip install huggingface_hub torch numpy -hf download AOSSIE/openverifiable-enwiki-20260901-20260918-r1-evidence \ - --repo-type dataset --include "release/simple-run/*" --exclude "*/ckpt-*" --local-dir ovl -cd ovl/release/simple-run -sha256sum -c --ignore-missing release-hashes.txt -git clone -q --depth 1 --branch v1.1.0 https://github.com/AOSSIE-Org/OpenVerifiableLLM ovl-code -OVL_DEVICE=cpu python ovl-code/simple/ovl.py verify --run . --weights conversation-model.pt +git clone -q --depth 1 https://github.com/AOSSIE-Org/OpenVerifiableLLM +bash OpenVerifiableLLM/simple/check_release.sh # anchor anchor.json: chain head MATCH (38 entries) # anchor timestamp freetsa: OK (Oct 20 14:38:43 2026 GMT) # anchor timestamp digicert: OK (Oct 20 14:38:43 2026 GMT) # weights conversation-model.pt: MATCH chain entry 37 +# OVL release check PASS: the released model matches its timestamped training record. ``` -Use `ovl.py` from the v1.1.0 tag (v1.0.2 or newer works too). The copy in the download's `code/` -folder is older and does not check the anchor. +To run each step yourself, pinned to the v1.1.0 code, see [docs/verify.md](docs/verify.md#1-consistency-check-without-a-gpu). **3. Verification, on an RTX 5090 (about 1 hour, about US$1.50).** [Deploy the RunPod template](https://console.runpod.io/deploy?template=mai1ozh35g), wait, and read diff --git a/docs/verify.md b/docs/verify.md index 9b60b2c5..c2ea050c 100644 --- a/docs/verify.md +++ b/docs/verify.md @@ -17,12 +17,16 @@ folder `release/simple-run/`. The terms (segment, boundary, chain entry, anchor) ## 1. Consistency check, without a GPU -Needs Python 3.12 or newer, `git`, `openssl`, `sha256sum` and network access. On macOS, -`sha256sum` comes with `brew install coreutils` (as `gsha256sum`). Download `release/simple-run/` +`bash simple/check_release.sh`, from a clone of this repository, does all of this in one command (about +2 minutes). By hand, it needs Python 3.12 or newer, `git`, `openssl`, `sha256sum` and network access. On +macOS, `sha256sum` comes with `brew install coreutils` (as `gsha256sum`). Download `release/simple-run/` without the `ckpt-*.pt` files, then from that folder: ```bash -pip install huggingface_hub torch numpy +# On Linux, add --index-url https://download.pytorch.org/whl/cpu to the torch install: the default +# download includes CUDA (several GB), and this check needs only the CPU. +pip install torch +pip install huggingface_hub numpy sha256sum -c --ignore-missing release-hashes.txt # every downloaded file matches the hash list git clone -q --depth 1 --branch v1.1.0 https://github.com/AOSSIE-Org/OpenVerifiableLLM ovl-code OVL_DEVICE=cpu python ovl-code/simple/ovl.py verify --run . --weights conversation-model.pt diff --git a/simple/README.md b/simple/README.md index 4f7aa672..b6c2f7b2 100644 --- a/simple/README.md +++ b/simple/README.md @@ -10,6 +10,7 @@ see [verify](../docs/verify.md). | I want to… | Run | Needs | |---|---|---| +| Check the release in one command (download, hashes, record, weights) | `bash simple/check_release.sh` | any CPU, ~2 minutes | | Check the released weights and record | `python simple/ovl.py verify --run RUN --weights RUN/conversation-model.pt` | any CPU | | Check one training segment of the release | `bash simple/verify_segment.sh` | RTX 5090 | | Do that in one click on a fresh pod | the RunPod template in [`checker_template.md`](checker_template.md), which runs [`checker_pod.sh`](checker_pod.sh) | RTX 5090 pod | @@ -34,6 +35,7 @@ On a CPU, set `OVL_DEVICE=cpu`. | [`ovl.py`](ovl.py) | The loop. Commands: `train`, `verify` (`--weights`, `--segment`, `--random`, `--all`, `--data`; it needs the run's `anchor.json` and its two timestamps, or `--anchor FILE`; `--no-anchor` and `--no-timestamps` are only for a run that hasn't been anchored yet), `fork`, `chat`, `bench` | | [`model.py`](model.py), [`data.py`](data.py) | The GPT model, and the token ids, batch layout and seeding shared with the data preparation (copied unchanged from commit `3f882ad`) | | [`stamp.py`](stamp.py) | RFC 3161 timestamps (FreeTSA and DigiCert) for each chain entry while training runs; `--check` verifies them. `OVL_TIMESTAMP=0` turns stamping off | +| [`check_release.sh`](check_release.sh) | The laptop check in one command: a CPU-only Python environment, the release without its checkpoints, the hash list, then `ovl.py verify` | | [`verify_segment.sh`](verify_segment.sh) | Downloads the release's data, record and the needed checkpoints, then checks `K` random segments (or `SEGMENTS="7 8"`) | | [`checker_pod.sh`](checker_pod.sh) | One-click check for a fresh RTX 5090 pod: fetches the release code at a pinned revision, runs `verify_segment.sh`, prints one result line to share | | [`checker_template.md`](checker_template.md) | The description and start command of the public RunPod template | diff --git a/simple/check_release.sh b/simple/check_release.sh new file mode 100755 index 00000000..59354f5e --- /dev/null +++ b/simple/check_release.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# The consistency check of the released model in one command, on any computer, no GPU. +# +# git clone --depth 1 https://github.com/AOSSIE-Org/OpenVerifiableLLM +# bash OpenVerifiableLLM/simple/check_release.sh # works in ./ovl-check +# +# It makes a Python environment (CPU-only PyTorch), downloads the release without its +# checkpoints (about 330 MB; the Python environment takes about 1 GB), checks every downloaded file against release-hashes.txt, then runs +# `ovl.py verify`: the record's hash chain, its two timestamps, and the released weights. +# Takes about 2 minutes. Needs Python 3.12 or newer, git, openssl and network access. Set OVL_PYTHON to use an existing +# Python that already has torch, numpy and huggingface_hub, and skip the install. +# It recomputes no training: that needs an RTX 5090 (docs/verify.md). +set -euo pipefail + +WORK=${1:-ovl-check} +RUN_PATH=${RUN_PATH:-release/simple-run} +REPO=AOSSIE/openverifiable-enwiki-20260901-20260918-r1-evidence +CODE=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +mkdir -p "$WORK"; WORK=$(cd "$WORK" && pwd) +fail() { echo "OVL release check FAIL: $*" >&2; exit 1; } + +command -v openssl >/dev/null || fail "openssl not found; it checks the record's timestamps" +if [ -n "${OVL_PYTHON:-}" ]; then + PY=$OVL_PYTHON +else + BASE=$(command -v python3 || command -v python) || fail "Python 3.12 or newer not found" + "$BASE" -c 'import sys; sys.exit(sys.version_info < (3, 12))' || fail "needs Python 3.12 or newer, found $("$BASE" -V)" + if [ ! -x "$WORK/venv/bin/python" ]; then + echo "== Python environment in $WORK/venv" + "$BASE" -m venv "$WORK/venv" + "$WORK/venv/bin/python" -m pip install -q --upgrade pip + # On Linux the default PyTorch download includes CUDA (several GB); this check needs only the CPU. + [ "$(uname -s)" = Linux ] && TORCH_INDEX=(--index-url https://download.pytorch.org/whl/cpu) || TORCH_INDEX=() + "$WORK/venv/bin/python" -m pip install -q "${TORCH_INDEX[@]}" torch + "$WORK/venv/bin/python" -m pip install -q numpy huggingface_hub + fi + PY=$WORK/venv/bin/python +fi + +echo "== Downloading $RUN_PATH (without checkpoints)" +"$PY" - "$REPO" "$RUN_PATH" "$WORK" <<'PY' +import sys +from huggingface_hub import snapshot_download +repo, path, out = sys.argv[1:] +snapshot_download(repo, repo_type="dataset", allow_patterns=[f"{path}/*"], ignore_patterns=[f"{path}/ckpt-*"], + local_dir=out) +PY +RUN=$WORK/$RUN_PATH + +echo "== Checking downloaded files against release-hashes.txt" +"$PY" - "$RUN" <<'PY' || fail "a downloaded file differs from release-hashes.txt" +import hashlib, sys +from pathlib import Path +run, bad, n = Path(sys.argv[1]), 0, 0 +for line in (run / "release-hashes.txt").read_text(encoding="utf-8").splitlines(): + want, name = line.split(maxsplit=1) + f = run / name.lstrip("*") + if not f.is_file(): + continue # not downloaded, like the checkpoints + h = hashlib.sha256() + with f.open("rb") as fh: + for chunk in iter(lambda: fh.read(1 << 20), b""): + h.update(chunk) + n += 1 + if h.hexdigest() != want: + print(f"{name}: FAILED"); bad += 1 +print(f"{n} files: {'all match' if not bad else f'{bad} differ'}") +sys.exit(1 if bad or not n else 0) +PY + +echo "== Checking the record and the released weights" +OVL_DEVICE=cpu "$PY" "$CODE/simple/ovl.py" verify --run "$RUN" --weights "$RUN/conversation-model.pt" \ + || fail "verify did not pass (see above)" +echo "OVL release check PASS: the released model matches its timestamped training record."