Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions research/replay-planner-ancestor-cache/BENCHMARK.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
{
"schema_version": 1,
"claim_type": "empirical tooling improvement",
"timestamp_utc": "2026-10-29T13:11:10.625011+00:00",
"head": "3f0152a32eeebac0df11946dbd50cabe5a7f7839",
"python": "3.12.15 (main, Oct 14 2026, 00:55:37) [Clang 23.1.3 ]",
"platform": "macOS-27.0.1-arm64-arm-64bit",
"tracked_paths_sha256": "e2d42dd7de100ce4155860425a3b9d0f1ef943c1e70bd26b5ae9fc4ccd445022",
"sparse_checkout_sha256": "5230ac40920ff55e2fb4be70ddb3075488002bf79dddccae95d54ea7b2def5c6",
"source_sha256": {
"scripts/research_package.py": "e1256016e7f51827d04fa2179255989f639ae1d97afaa9c16265f85a0c6abd70",
"research/replay-planner-ancestor-cache/planner.py": "54ed1ecd59ae2aef90755e96d304f445407763907c55e526294b01c7cf47cd49",
"research/replay-planner-ancestor-cache/benchmark.py": "1cca5ee36d6a17603bc2ebb9b4a6074bf6ab271eb3ffbec7678800856be4e2e6",
"research/replay-planner-ancestor-cache/VERIFY.json": "91f0aa22c030ef0ee00b65bf91d4c3d65531e4795b3110d46948e00a42185bde"
},
"measurement": {
"commands": [
[
"/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12",
"-I",
"-W",
"error",
"-B",
"/Users/nick/git/imb-research-11/scripts/research_package.py",
"--changed-since",
"0000000000000000000000000000000000000000"
],
[
"/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12",
"-I",
"-W",
"error",
"-B",
"/Users/nick/git/imb-research-11/research/replay-planner-ancestor-cache/planner.py",
"--changed-since",
"0000000000000000000000000000000000000000"
]
],
"pairs": 9,
"samples_seconds": [
[
0.89816625,
0.884204958,
0.970985542,
0.888555875,
0.890791583,
0.902384959,
0.928754333,
0.8907095,
0.905556583
],
[
0.817896708,
0.801496583,
0.852495667,
0.823127125,
0.826971417,
0.819598375,
0.83802175,
0.823856,
0.867351208
]
],
"median_seconds": [
0.89816625,
0.823856
],
"range_seconds": [
[
0.884204958,
0.970985542
],
[
0.801496583,
0.867351208
]
],
"fraction_less_time": 0.0827355180624968,
"errors": 0,
"complete_outputs_equal": true,
"selected_packages": 3788,
"stdout_sha256": "e0a4c3bcf5545c3334dc8a36b2509c7b57df4c25cbdb9395898db385d56e82d0",
"stderr_sha256": "0ccfcc32adec026de1f50bb05cf4d5ac9e555a7589d844786ace39f5ed9c5e93",
"stderr": "Planning includes 3759 unchanged sparse manifests from HEAD; materialize packages and inputs before replay.\n"
}
}
151 changes: 151 additions & 0 deletions research/replay-planner-ancestor-cache/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
# Reuse checked ancestors within one complete replay plan

The opt-in planner reduces complete local planning time by **8.27%**, from
median **0.898166250 s** to **0.823856000 s** over nine alternating pairs.
All eighteen commands succeed and produce identical selection JSON and sparse
warnings for **3,788 packages**, including **3,759** manifests authenticated from
sparse HEAD. Before times range from 0.884204958 to 0.970985542 seconds; after
times range from 0.801496583 to 0.867351208 seconds.

This is a scoped equivalence proof with finite performance evidence, not a
mathematical bound or supplier-admission result. This contribution was
produced by Nick Petalas. The selected frontier is unchanged.

## The change

[JoyboyBrian's PR #6087](https://github.com/CrocSwap/integer-mult-bounds/pull/6087)
already avoids checking an identical full input path repeatedly within a plan.
Different inputs and manifests still share ancestors. The new [planner.py](planner.py)
checks each distinct existing ancestor once and reuses that result only within
the same invocation of `plan_details`.

```text
begin plan with an empty prefix table
for each canonical path:
reuse already checked prefixes
inspect new components for symlinks and junctions
stop at a missing component or a regular-file ancestor
run the unchanged selection and manifest-hash logic
discard the table on success or failure
```

The command authenticates the exact shared runner against [SOURCES.json](SOURCES.json),
loads those bytes into a private module without consulting bytecode, and temporarily
replaces only that module's path guard during a synchronous plan. Its parser,
sparse-manifest authentication, Git environment, exact HEAD/diff spelling,
complete-input cache, selection, hashes and output logic remain in use. It does
not modify the shared runner or accelerate the default CI command. A changed
runner fails authentication until this package is deliberately updated.

## Scoped equivalence proof

The inherited contract requires the checkout to remain stable throughout one
plan. Each `lstat` therefore returns the same metadata or exception for a given
path during that plan. Every call first runs the original canonical-path validator.
The first uncached lookup performs the original symlink and junction tests.
Only a checked directory permits descent. A checked regular file permits the
same return as the original guard: a descendant would raise the suppressed
`NotADirectoryError`. A missing prefix similarly suppresses deeper lookups.
Thus the cached result reproduces every later decision under that prefix.
Permission and other unexpected errors propagate with their original type and
message and are not cached. Rejected symlinks and junctions are never cached.

Induct on the finite components of each path and then on the paths visited by
the unchanged planner. The replacement guard returns the same path or raises
the same error at each visit. Every manifest read, input-validation result,
selection cause and manifest hash is consequently unchanged. A table is created
inside each `plan_details` call and the guard is restored in `finally`; the next
plan must inspect the filesystem again. Concurrent filesystem mutation is
outside the inherited contract. The private module has one synchronous caller.

## Evidence and reproduction

[BENCHMARK.json](BENCHMARK.json) is generated by [benchmark.py](benchmark.py).
It binds implementation, harness, manifest and shared-source hashes, Git path
inventory and sparse configuration, interpreter, platform, all eighteen samples,
error count and stdout/stderr digests. Each timed command includes process
startup, candidate source authentication, all planning and captured output.
Every pair reverses which side runs first. No precomputed result is timed.

An untimed profile put repeated component traversal and `lstat` work in the
original guard at about one quarter of complete baseline planning time. The
measured end-to-end reduction is smaller; JSON, Git blob reads and timeout
supervision still run. The machine had 18 logical CPUs and load averages 2.92,
3.19 and 2.79 at the initial profile. Both sides use identical interpreter flags
and the existing caches. These are warm-filesystem macOS measurements on a
shared machine, not cold-cache or Linux/Windows speedup claims.

From the repository root, using the local interpreter:

```sh
/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -I -W error -B research/replay-planner-ancestor-cache/planner.py --changed-since origin/main
/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -I -W error -B research/replay-planner-ancestor-cache/verify.py
/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -I -O -W error -B research/replay-planner-ancestor-cache/verify.py
/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -B scripts/research_package.py --package research/replay-planner-ancestor-cache
/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -B scripts/check_research_commit.py --package research/replay-planner-ancestor-cache --check-source-history --check-modes
```

Another standard-library CPython 3.12 executable may replace the absolute path.
The direct marker is `PASS_REPLAY_PLANNER_ANCESTOR_CACHE`; the registered marker
is `PASS_REGISTERED_RESEARCH_PACKAGE`. Each direct run generates a receipt bound
to every package file and writes no package files. Optional timing reproduction,
which CI does not run:

```sh
/Users/nick/git/integer-mult-bounds/.venv/bin/python3.12 -I -W error -B research/replay-planner-ancestor-cache/benchmark.py --pairs 9 --output /absolute/scratch/benchmark.json
```

Future checkouts and hosts need not reproduce exact times or package counts.
Registered replay checks the committed measurement's bindings and arithmetic.
The direct verifier covers 18 path cases, four symlink/junction cases, two lookup
failure cases, eight complete fixture-plan cases, changed-source rejection and
measurement-receipt validation. It checks every fixture consumer, duplicate
inputs, component ordering, package changes, shared controls and manifest hashes.
Later calls reject a new alias and recover after its removal. Both successful
and failed calls restore the guard. Normal and optimized modes pass. Fixture
Git inventory is explicit; the real timed commands perform original Git and
sparse authentication.

## Resource and diagnostic limits

Added loops visit finite path components, manifest inputs or fixed test cases.
The cache stores at most one entry per distinct prefix encountered in one plan;
memory is proportional to the total stored prefix characters and is released
when that plan ends. Registered commands have 120-second timeouts. Each optional
benchmark subprocess has a 60-second timeout and the harness permits at most
twenty pairs.

Python's managed allocation is a Power-of-Ten rule 3 deviation limited to this
finite table. Its compensating controls are the finite prefix bound and release
at the end of the call; tests check reuse and next-plan invalidation. Heap peak
was not measured. Shared integration requiring a hard memory limit must first
measure and bound this table for its repository-size limit.

Executing the authenticated existing runner is a rule 8 deviation: the input is
one SHA256-pinned source, not generated code. A changed-source control rejects
before execution, and the namespace is private. This preserves inspectable
shared checks without duplicating them. Reassess this loader if the runner's
dependencies or execution model change.

Warning-strict normal/optimized and in-memory syntax checks are used. No Python
linter was installed locally. Native Windows junction execution, a fully
materialized checkout benchmark, CI timing, selected supplier replay, Lean,
Docker and complete multiplication obligations are unrun. Independent human
and formal proof-kernel review are unclaimed.

## Open next questions

Should this validator be incorporated into the shared planner? First exercise
real Windows junctions and measure complete plans in a fully materialized
checkout. Those platform and workload checks remain the blockers. The present
opt-in command provides a reviewable implementation and reproduction harness.

## Credit

JoyboyBrian retains PR #6087's complete-input reuse and tests. Douglas Colkitt
retains the shared replay runner. Ali Romman retains the earlier complete-plan
benchmark context; the exact-path, symlink/junction, Git-source, sparse-manifest
and deadline contributors retain their original source notices. The shared
source is consumed unchanged. The new prefix validator, private CLI, scoped
equivalence proof, behavioral controls and timing harness were produced by
Nick Petalas. Apache-2.0.
24 changes: 24 additions & 0 deletions research/replay-planner-ancestor-cache/SOURCES.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{
"schema_version": 1,
"base_commit": "3f0152a32eeebac0df11946dbd50cabe5a7f7839",
"files": [
{
"path": "certificates/selected-result.json",
"usage": "provenance",
"git_blob": "d59b5035e7b4cbaa9d688a1d32dd0d6eb79c466a",
"sha256": "9b8d0eea95b3bf0dfcb7938a732a6439b87a2685dec2a5ab9e525b3b34debbfa"
},
{
"path": "docs/research/agent-task-catalog.json",
"usage": "provenance",
"git_blob": "3404fbdd5e7529a0674eef26114b5071e4586cec",
"sha256": "ead6cdc9d0af6d58e8a49744d8ec8c7c15e13d2ba93c276c434bcf9b51b93a81"
},
{
"path": "scripts/research_package.py",
"usage": "input",
"git_blob": "091b4c8f58bac9a9580d76025b887dc5c873d05f",
"sha256": "e1256016e7f51827d04fa2179255989f639ae1d97afaa9c16265f85a0c6abd70"
}
]
}
61 changes: 61 additions & 0 deletions research/replay-planner-ancestor-cache/TASK.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{
"schema_version": 1,
"task_id": "replay-runtime",
"priority_revision": 8,
"base_commit": "3f0152a32eeebac0df11946dbd50cabe5a7f7839",
"original_card": {
"id": "replay-runtime",
"question": "Can one measured replay bottleneck be sped up without dropping coverage?",
"starting_points": [
"scripts/run_isolated_tests.py",
"scripts/verify_replay_coverage.py",
"certificates/selected-result.json"
],
"methods": [
"Profile one isolated replay and optimize the measured bottleneck.",
"Design source-bound reuse with explicit dependency invalidation and negative controls."
],
"acceptance_gate": "Report before/after wall time on the same fixture, retained checks and invalidation behavior.",
"first_step": {
"command": [
"python3",
"-B",
"scripts/research_selected_baseline.py"
],
"expected": "PASS_RESEARCH_BASELINE_BINDINGS_ONLY",
"inspect": "Resolve the current selected review, separate complex/ordinary BIT/admitted BIT/outer budgets, and compare the card against live handoffs before starting. Historical ceilings require exact source/model equivalence.",
"scope": "Source-binding orientation only; not a scientific replay or all-size proof. Materialize selected inputs if this checkout is sparse."
},
"first_experiment": "Measure one unchanged fixture, preserve check coverage, and test one optimization with invalidation controls.",
"bottleneck_link": "Faster trustworthy replays increase review throughput and let mechanism candidates be checked sooner.",
"allowed_variations": [
"Profile one stage or design one source-bound cache boundary."
],
"scope_change_triggers": [
"Skipping coverage or comparing different fixtures to claim speedup."
],
"overlap_terms": [
"replay runtime",
"cache invalidation",
"isolated tests"
]
},
"scope_status": "changed",
"actual_scope": "Reuse successful filesystem ancestor checks within one complete replay-planning invocation; deliver a source-pinned opt-in CLI with exact output and failure controls.",
"scope_change_reason": "The profiled coordination planner is a bounded prerequisite for replay rather than a scientific verifier. This adds prefix reuse beyond PR6087 exact-full-input reuse.",
"bottleneck_link": "Less planning overhead lets unchanged complete replay coverage reach its isolated jobs sooner; no mathematical or supplier saving is inferred.",
"budget_minutes": 60,
"stop_rule": "Ship the first reliable complete-command improvement after exact equivalence and rejection controls; do not extend into shared-runner changes.",
"outcome": "completed",
"evidence": "verify.py behavioral differentials; BENCHMARK.json generated complete-command timing receipt; README.md scoped equivalence argument.",
"author": "Nick Petalas",
"ai_assistance": "Independent human and formal proof-kernel review are unclaimed.",
"open_questions": [
{
"question": "Should this invocation-local ancestor validator be incorporated into the shared planner?",
"why_it_matters": "The present opt-in command does not accelerate the default CI path.",
"first_step": "Run the same differential controls on native Windows junctions and benchmark a fully materialized checkout before proposing shared integration.",
"blocked_by": "Native Windows and full-checkout performance have not been measured."
}
]
}
36 changes: 36 additions & 0 deletions research/replay-planner-ancestor-cache/VERIFY.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"schema_version": 1,
"status": "ready",
"python": "3.12",
"requirements": [],
"inputs": [
"scripts/research_package.py"
],
"commands": [
{
"argv": [
"{python}",
"-I",
"-W",
"error",
"-B",
"{package}/verify.py"
],
"timeout_seconds": 120,
"checks": "Exact path-guard and complete-plan differential controls, next-call invalidation, errors and source authentication."
},
{
"argv": [
"{python}",
"-I",
"-O",
"-W",
"error",
"-B",
"{package}/verify.py"
],
"timeout_seconds": 120,
"checks": "The same behavioral controls with assertions disabled."
}
]
}
Loading