Speed up publication byte/history checks by 12% (no new κ)

research/lazy-publication-digests/README.md

@@ -0,0 +1,131 @@
+# Defer publication mismatch digests
+
+This contribution was produced by Nick Petalas.
+
+The complete publication preflight is about 12% faster on the measured large-input
+fixture after deferring two diagnostic SHA256 digests until a byte mismatch.
+This is a measured verifier improvement with a scoped equivalence argument,
+not a finite mathematical certificate or a new multiplication theorem. No new
+kappa is claimed.
+
+The production change is one conditional in `scripts/check_research_commit.py`.
+Python previously evaluated the mismatch message before calling `require`, even
+when `working == committed`. The message hashes both buffers. The changed code
+compares those same buffers first and constructs the same message only on failure.
+
+## Equivalence argument
+
+If the buffers agree, both versions continue to the next file. The old version
+also computes two digests that cannot affect the equality decision. If they
+differ, both raise `ValueError` with exactly the same filename, working digest,
+committed digest and punctuation. The surrounding `closing(git_blobs(...))`
+context still closes the reader on failure. Successful iteration still advances
+the reader through its final Git exit-status check.
+
+Inventory, literal path spelling, regular-file requirements, optional executable
+modes, committed source metadata, raw ancestry, replacement/graft isolation and
+resource bounds are untouched. The comparison remains exact byte equality;
+digest equality never decides acceptance. As with the original command, the
+checkout must stay stable during a check. Exceptional failure of the SHA256
+implementation itself is outside the equivalence claim on successful comparisons.
+
+## Measured evidence
+
+Canonical base is `721c03822c421b7f7fcbb2598993b947c4d88419`, priority revision 8.
+The unchanged `research/physical-gaussian-core` fixture supplies 42 compared files,
+54,975,541 bytes and two declared source commits. Its mathematical verifier was
+not run. Every timed sample runs the complete byte/history preflight through
+`check(..., check_source_history=True)` in a fresh interpreter, including imports,
+all Git subprocesses, exact comparison and the full returned receipt.
+
+Ten alternating trials per side on CPython 3.12.15, macOS arm64 gave:
+
+| Implementation | Median | Range |
+| --- | ---: | ---: |
+| Original | 280.56 ms | 277.11 to 300.94 ms |
+| Deferred digests | 246.91 ms | 241.00 to 256.76 ms |
+
+The median reduction is 11.99%. All 20 complete receipts are byte-identical,
+with zero command errors. Both sides use the same shared reader, fixture,
+flags and warmup. The host has 18 logical CPUs; the recorded initial one-minute
+load is in the generated benchmark receipt. Other processes were running, so
+the sides alternate. These are warm-filesystem timings, not cold-disk or
+cross-platform performance claims.
+
+An untimed complete-command profile attributes 34 ms of 274 ms to SHA256 and
+133 ms to subprocess calls. Removing 84 diagnostic digests is consistent with
+the observed 34 ms saving; Git and file transport still limit the command.
+The generated JSON receipt binds both implementation hashes and records every
+sample, host details and the complete common stdout. It is retained in the PR
+body; reproduction writes a new receipt only into the requested scratch directory.
+
+The focused verifier runs 36 actual Git regression tests. Two new tests require
+zero successful-comparison digests and the exact mismatch message, two mismatch
+digests and reader closure. The no-digest test rejects the original implementation.
+Existing controls cover normalized/binary bytes, working drift, missing files,
+inventory changes, literal wildcard/Unicode paths, committed links, reader final
+failure, immutable objects, source-history failures and executable modes.
+
+## Reproduction
+
+From the repository root, use the same interpreter for both sides:
+
+```sh
+/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -B scripts/research_package.py --package research/lazy-publication-digests
+/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -I -W error -B research/lazy-publication-digests/verify.py
+/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -B scripts/check_research_commit.py --package research/lazy-publication-digests --check-source-history
+```
+
+The registered commands run normal and optimized Python with warnings as errors.
+They exercise transport contracts, not the predecessor package's mathematics.
+The test fixtures are temporary synthetic Git repositories, not research checkouts.
+
+For timing, materialize `research/physical-gaussian-core` and all paths in its
+`VERIFY.json` before running. Full Git history and the pinned base are required.
+Pass an absolute scratch directory outside the checkout:
+
+```sh
+/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -I -W error -B research/lazy-publication-digests/benchmark.py --scratch-dir "$RUN_DIR" --rounds 10
+```
+
+`RUN_DIR` names a directory selected by the reviewer. The benchmark writes the
+unchanged baseline script and `benchmark.json` there, never into the checkout.
+Source provenance is in `SOURCES.json`; current implementation/test changes are
+declared replay inputs in `VERIFY.json`.
+
+## Scope and engineering choices
+
+The autonomous local run playbook stops at this first measured improvement.
+Build the Lever supplies the reproducible complete-command benchmark. Prove It
+Works requires complete receipts and real Git rejection tests. Power of Ten
+keeps the production edit to one existing comparison and bounds the benchmark
+to 5 through 30 pairs with a 60-second timeout per subprocess. Python allocation
+and the existing input-size-dependent checker loops remain managed-runtime
+deviations; this change introduces no buffer or new production loop. The
+benchmark deliberately imports exactly the selected before/after implementation
+files to use the same complete API, rather than modifying either source.
+
+No native replay, supplier admission, physical machine implementation, selected
+frontier replay, formal proof-kernel check or independent human review is claimed.
+Timing on small packages or another host may show little benefit. A complete
+multiplier and the five retained framework assumptions remain outside this work.
+
+## Open next questions
+
+How much does deferred diagnostic work save across a representative publication
+workload, including small packages? This matters because fixed Git overhead can
+dominate. First choose a fixed package list and preserve every complete receipt
+under alternating trials. A representative workload and a separate measurement
+budget are needed. The structured question is in `TASK.json`.
+
+## Credit and coordination
+
+Credit ywz65's publication comparator, Nick Petalas's earlier source-history
+preflight, roger-sato's immutable-object isolation, Douglas Colkitt's source
+reader/runner and the retained executable-mode and reader-cleanup contributors.
+The Gaussian package is only an unchanged transport fixture; its contributors
+retain its mathematical and implementation credit.
+
+[Advance claim](https://github.com/CrocSwap/integer-mult-bounds/discussions/1166#discussioncomment-19246971).
+The concurrent joint-ancestry and committed-metadata-reuse claims change separate
+preflight stages. Their performance is not measured here. Apache-2.0.

research/lazy-publication-digests/SOURCES.json

@@ -0,0 +1,66 @@
+{
+  "schema_version": 1,
+  "base_commit": "721c03822c421b7f7fcbb2598993b947c4d88419",
+  "files": [
+    {
+      "path": "certificates/selected-result.json",
+      "usage": "provenance",
+      "git_blob": "d59b5035e7b4cbaa9d688a1d32dd0d6eb79c466a",
+      "sha256": "9b8d0eea95b3bf0dfcb7938a732a6439b87a2685dec2a5ab9e525b3b34debbfa"
+    },
+    {
+      "path": "docs/research/agent-task-catalog.json",
+      "usage": "provenance",
+      "git_blob": "3404fbdd5e7529a0674eef26114b5071e4586cec",
+      "sha256": "ead6cdc9d0af6d58e8a49744d8ec8c7c15e13d2ba93c276c434bcf9b51b93a81"
+    },
+    {
+      "path": "scripts/check_research_commit.py",
+      "usage": "provenance",
+      "git_blob": "dc0be6c743f79b3131e674d92c81b75b069812c9",
+      "sha256": "63026ec3805f2a3236cc071fc78cf1c54bc6528c43e2f3d82548dc68bf346bce"
+    },
+    {
+      "path": "scripts/research_package.py",
+      "usage": "provenance",
+      "git_blob": "48fe9feb4699c09e95505d95c327ea998ac8b3a7",
+      "sha256": "84a45190a6a96c20993a0d4a1045fbf24de660d2e6fef8f8a447bb3d1aa78db6"
+    },
+    {
+      "path": "tests/test_research_commit_bytes.py",
+      "usage": "provenance",
+      "git_blob": "3b367bfbc33e39fdad583e817a554652f58d4123",
+      "sha256": "1b0c66576f1948213dbfca502e1ccbfd2566840be697270c9197fff0abf8093a"
+    },
+    {
+      "path": "tests/test_research_source_history.py",
+      "usage": "provenance",
+      "git_blob": "aaaa47c607ce1ee317f0c281586016920ff40eac",
+      "sha256": "0175206fa52dff514fea4d625a072bb66735eb5214be9396dc42b874f0918454"
+    },
+    {
+      "path": "tests/test_research_commit_modes.py",
+      "usage": "provenance",
+      "git_blob": "0d619bf141f7a3d681143243a3582cd09449e5e0",
+      "sha256": "718dbd7b6e857bc250f37b4d4ea26d47909b720e69aca06071f39507fbf55462"
+    },
+    {
+      "path": "tests/test_research_git_replacements.py",
+      "usage": "provenance",
+      "git_blob": "2b8e1edac9755ce5258aeadd9e4e6a69d0450fcc",
+      "sha256": "534929cae3719b452db79054c92736365eb21bad7f77e036c17126e2eeb2c809"
+    },
+    {
+      "path": "research/physical-gaussian-core/VERIFY.json",
+      "usage": "provenance",
+      "git_blob": "4fb267c2490d7d79ffe140713ce3a4e891b3c60f",
+      "sha256": "681d32de565b01c2a253f5c7650916dbbebb00abe871acbb23e02dd22fdb8643"
+    },
+    {
+      "path": "research/physical-gaussian-core/SOURCES.json",
+      "usage": "provenance",
+      "git_blob": "76c6b24324af919cef0887aa7d2aeff4c48832ba",
+      "sha256": "d3ac712b80fa771f87c0127eaf22debdb290dc544f0bc7b218912e50a8710966"
+    }
+  ]
+}

research/lazy-publication-digests/TASK.json

@@ -0,0 +1,61 @@
+{
+  "schema_version": 1,
+  "task_id": "replay-runtime",
+  "priority_revision": 8,
+  "base_commit": "721c03822c421b7f7fcbb2598993b947c4d88419",
+  "original_card": {
+    "id": "replay-runtime",
+    "question": "Can one measured replay bottleneck be sped up without dropping coverage?",
+    "starting_points": [
+      "scripts/run_isolated_tests.py",
+      "scripts/verify_replay_coverage.py",
+      "certificates/selected-result.json"
+    ],
+    "methods": [
+      "Profile one isolated replay and optimize the measured bottleneck.",
+      "Design source-bound reuse with explicit dependency invalidation and negative controls."
+    ],
+    "acceptance_gate": "Report before/after wall time on the same fixture, retained checks and invalidation behavior.",
+    "first_step": {
+      "command": [
+        "python3",
+        "-B",
+        "scripts/research_selected_baseline.py"
+      ],
+      "expected": "PASS_RESEARCH_BASELINE_BINDINGS_ONLY",
+      "inspect": "Resolve the current selected review, separate complex/ordinary BIT/admitted BIT/outer budgets, and compare the card against live handoffs before starting. Historical ceilings require exact source/model equivalence.",
+      "scope": "Source-binding orientation only; not a scientific replay or all-size proof. Materialize selected inputs if this checkout is sparse."
+    },
+    "first_experiment": "Measure one unchanged fixture, preserve check coverage, and test one optimization with invalidation controls.",
+    "bottleneck_link": "Faster trustworthy replays increase review throughput and let mechanism candidates be checked sooner.",
+    "allowed_variations": [
+      "Profile one stage or design one source-bound cache boundary."
+    ],
+    "scope_change_triggers": [
+      "Skipping coverage or comparing different fixtures to claim speedup."
+    ],
+    "overlap_terms": [
+      "replay runtime",
+      "cache invalidation",
+      "isolated tests"
+    ]
+  },
+  "scope_status": "changed",
+  "actual_scope": "Defer mismatch-only SHA256 diagnostics in the publication byte preflight, preserving exact comparisons and failure behavior.",
+  "scope_change_reason": "A verifier-engineering result at the publication boundary rather than a mathematical replay or supplier change.",
+  "bottleneck_link": "Reduce repeated publication-check work without changing its acceptance predicate.",
+  "budget_minutes": 60,
+  "stop_rule": "Ship the first measured complete-command improvement with identical receipts and passing rejection controls.",
+  "outcome": "completed",
+  "evidence": "README.md records ten alternating complete runs; benchmark.py regenerates a source-bound receipt outside the checkout.",
+  "author": "Nick Petalas",
+  "ai_assistance": "Independent human and formal proof-kernel review are unclaimed.",
+  "open_questions": [
+    {
+      "question": "How much does this save across a representative publication workload, including small packages?",
+      "why_it_matters": "The measured benefit depends on compared byte count and Git overhead.",
+      "first_step": "Benchmark a fixed declared package list with the same complete receipts and alternating runs.",
+      "blocked_by": "A representative workload and separate measurement budget."
+    }
+  ]
+}

research/lazy-publication-digests/VERIFY.json

@@ -0,0 +1,41 @@
+{
+  "schema_version": 1,
+  "status": "ready",
+  "python": "3.12",
+  "requirements": [],
+  "inputs": [
+    "scripts/check_research_commit.py",
+    "scripts/research_package.py",
+    "tests/test_research_commit_bytes.py",
+    "tests/test_research_source_history.py",
+    "tests/test_research_commit_modes.py",
+    "tests/test_research_git_replacements.py"
+  ],
+  "commands": [
+    {
+      "argv": [
+        "{python}",
+        "-I",
+        "-W",
+        "error",
+        "-B",
+        "{package}/verify.py"
+      ],
+      "timeout_seconds": 120,
+      "checks": "Actual Git byte/inventory, exact mismatch digests, zero success-path digests, reader finalization, immutable objects, source history and POSIX execute modes."
+    },
+    {
+      "argv": [
+        "{python}",
+        "-I",
+        "-W",
+        "error",
+        "-O",
+        "-B",
+        "{package}/verify.py"
+      ],
+      "timeout_seconds": 120,
+      "checks": "The same complete behavior under optimized Python."
+    }
+  ]
+}