Llm experimental

CHANGELOG.md

@@ -5,11 +5,26 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [0.63.1] - Unreleased
+## [0.64.0] - Unreleased
+
+### Added
+  - **Independent LLM artifact verifier**: Added `script-examples/verify_llm_result.py` for all eight current CPU/Metal decode/prefill and contiguous/paged profiles. Independent Python arithmetic checks geometry, work, checksums, timing, rates, accepted sample populations, and statistics. Optional `--binary` checks the executable file's SHA-256, and `--require-raw-timing` requires original CPU timing evidence. Verdicts distinguish consistent accepted results, inconsistent or unaccepted results, and unsupported evidence or resource limits. Added `make test-llm-verifier` to the aggregate `make test-all` gate.
+  - **LLM build provenance and original CPU timing**: Results now retain a build manifest with available Git revision and dirty state, compiler, build flags, architecture, SDK, deployment target, and executable SHA-256. CPU measurements and excluded attempts retain original Mach start/stop/delta ticks and timebase for elapsed-time reconstruction. Python 3 is now required for build provenance generation; missing evidence remains explicitly unavailable or partial.
+  - **CPU and Metal checksum fault coverage**: Added independent arithmetic goldens, corruption and work-mutation tests, real-GPU reduction-boundary tests, and wider prefill boundary coverage. Documented profile-specific checksum collisions and final-state sampling limits, with acceptance validation across all eight profiles, file/stdout output, graceful interruption, and persistent multi-GiB Metal resources.
 
 ### Changed
-  - **CLI help updated**.
-  - **Test coverage cleaned up**: Removed redundant tests.
+  - **LLM output advances to JSON schema 2 and v2 methodologies**: All eight profiles now use `llm-memory-v2-<backend>-<phase>-<layout>`. Canonical scenario plans and expected checksums are stored once, measurements and calibration attempts reference those plans, and aggregates identify their accepted measurements. Geometry, model context, layout, resources, and component identities have explicit owners, with strict reference and null/status semantics. The API documents the schema-1-to-schema-2 field map; no compatibility aliases or fallback reader are provided.
+  - **LLM validation retains separate observations**: Named checks now preserve applicability, completion, validity, and reasons for structure, final KV writes or unchanged append state, and applicable padding checks. A checksum match cannot hide a failed final-state check, and unevaluated checks remain unresolved instead of appearing successful.
+  - **LLM file checkpoints are bounded by loop count**: Files now receive progress snapshots every `K=max(1,ceil(count/8))` completed loops, normally at most eight progress snapshots plus one terminal snapshot. Abrupt termination can lose up to `3K` completed attempts since the last successful snapshot. Exact `--output -` prepares only the terminal document, and disabled output skips JSON construction. Writer counters report actual prior persistence attempts; checkpoint failures remain terminal and are not retried.
+  - **LLM result collection avoids repeated retained data and statistics work**: Calibration keeps compact actual checksum evidence, canonical plans own expected values, and exact statistics are prepared at snapshot boundaries from accepted measurement IDs. Memory admission accounts for retained evidence, canonical-plan storage, statistics scratch, and simultaneous JSON construction and serialization peaks.
+  - **CPU task preparation validates inputs once per call**: Removed duplicate plan/resource validation within expected-checksum calculation while retaining fresh validation of borrowed plans and materialized resources before each executor call starts workers or timing.
+  - **LLM acceptance and comparison quality are separate**: `results_complete` describes the measured population, while `run_accepted` also requires valid execution evidence and no known command or checkpoint failure. Position balance, sample count, CV, duration, and environment remain separate comparison criteria. Documentation clarifies that cyclic order balances scenario positions without guaranteeing predecessor-pair balance, and that comparisons must account for conditioning and output cadence.
+  - **CLI help and reference documentation updated**: Refreshed general and LLM help, the machine API, manual, whitepaper, and supporting references for schema 2, verification, checkpoint behavior, and comparison requirements. Clarified the synthetic meaning of query heads, prefill tiles, and theoretical attention quantities.
+  - **Test coverage cleaned up**: Removed redundant contract tests and replaced brittle source-text assertions with focused semantic coverage. Hardware-dependent LLM cases are classified as integration tests, and the Makefile now builds Objective-C++ test sources with test flags and ARC.
+
+### Fixed
+  - **CPU contiguous decode validates final KV append bytes**: After timing stops and workers join, KV-bearing scenarios now verify every final K/V append byte against the expected pattern. Corruption produces an invalid measurement with retained diagnostics and cannot enter accepted aggregates, even when the timed checksum matches.
+  - **Theoretical prefill overflow no longer rejects valid memory work**: Overflowing model-context attention-pair or FMA quantities become nullable values with an explicit arithmetic-overflow reason. Exact byte, work, allocation, and execution guardrails remain enforced.
 
 ## [0.63.0] - 2026-08-23
 

CITATION.cff

@@ -9,8 +9,8 @@ authors:
     family-names: "Heimonen"
     orcid: "https://orcid.org/0009-0004-0023-2407"
 
-version: "0.63.0"
-date-released: 2026-08-23
+version: "0.64.0"
+date-released: 2026-09-06
 
 repository-code: "https://github.com/timoheimonen/macOS-memory-benchmark"
 url: "https://github.com/timoheimonen/macOS-memory-benchmark"

Makefile

@@ -71,6 +71,19 @@ TARGET = memory_benchmark
 # Default target: build the executable
 all: $(TARGET)
 
+# Recompute provenance before considering objects. A changed source revision,
+# dirty state or build flags invalidates every object, avoiding mixed manifests.
+export PROVENANCE_CXX = $(CXX)
+export PROVENANCE_CXXFLAGS = $(CXXFLAGS)
+export PROVENANCE_TEST_CXXFLAGS = $(TEST_CXXFLAGS)
+export PROVENANCE_ASFLAGS = $(ASFLAGS)
+export PROVENANCE_LDFLAGS = $(LDFLAGS) $(APPLE_FRAMEWORKS)
+.build-provenance.h: FORCE
+	python3 build-support/generate_provenance.py $@
+
+.PHONY: FORCE
+FORCE:
+
 # Rule for linking the executable from object files
 $(TARGET): $(OBJ_FILES)
 	@echo "Linking $(TARGET)..."
@@ -80,7 +93,8 @@ $(TARGET): $(OBJ_FILES)
 # Test directory and files
 TEST_DIR = tests
 TEST_SRCS := $(sort $(wildcard $(TEST_DIR)/*.cpp))
-TEST_OBJS := $(TEST_SRCS:.cpp=.o)
+TEST_OBJCXX_SRCS := $(sort $(wildcard $(TEST_DIR)/*.mm))
+TEST_OBJS := $(TEST_SRCS:.cpp=.o) $(TEST_OBJCXX_SRCS:.mm=.o)
 
 # Dependency files generated by DEPFLAGS. Test dependencies include tests/*.h
 # helpers through the compiler's actual include graph rather than a manually
@@ -98,13 +112,18 @@ TEST_LIB_OBJS := $(filter-out main.o, $(OBJ_FILES))
 # Recompile once after a Makefile change so an existing pre-dependency-file
 # workspace cannot keep stale objects. Subsequent header changes are tracked by
 # the generated .d files.
-$(OBJ_FILES) $(TEST_OBJS): Makefile
+$(OBJ_FILES) $(TEST_OBJS): Makefile .build-provenance.h
 
 # Rule for compiling test files (must come before generic %.o rule)
 $(TEST_DIR)/%.o: $(TEST_DIR)/%.cpp
 	@echo "Compiling test $< -> $@..."
 	$(CXX) $(TEST_CXXFLAGS) $(DEPFLAGS) -c $< -o $@
 
+# Objective-C++ tests retain test flags and ARC, including generated dependencies.
+$(TEST_DIR)/%.o: $(TEST_DIR)/%.mm
+	@echo "Compiling Objective-C++ test $< -> $@..."
+	$(CXX) $(TEST_CXXFLAGS) -fobjc-arc $(DEPFLAGS) -c $< -o $@
+
 # Objective-C++ production boundaries share the normal C++ build settings.
 %.o: %.mm
 	@echo "Compiling Objective-C++ $< -> $@..."
@@ -135,11 +154,16 @@ test-integration: $(TARGET) $(TEST_TARGET)
 test-script-examples:
 	python3 -m unittest -v tests/test_script_examples.py
 
-# All tests (unit tests + integration tests + bundled script examples)
+# Independent LLM artifact contract and mutation tests.
+test-llm-verifier:
+	python3 -m unittest -v tests/test_llm_result_verifier.py
+
+# All tests (unit, integration, bundled examples, and independent LLM verifier)
 test-all: $(TARGET) $(TEST_TARGET)
-	@echo "Running all tests (unit + integration + bundled script examples)..."
+	@echo "Running all tests (unit + integration + bundled examples + LLM verifier)..."
 	./$(TEST_TARGET)
 	$(MAKE) test-script-examples
+	$(MAKE) test-llm-verifier
 
 # Reproducible production C++ source coverage in an isolated /tmp build.
 coverage-unit:
@@ -217,7 +241,7 @@ uninstall:
 	@echo "$(TARGET) uninstalled successfully."
 
 # Define targets that don't correspond to files
-.PHONY: all clean test test-integration test-script-examples test-all \
+.PHONY: all clean test test-integration test-script-examples test-llm-verifier test-all \
 	coverage-unit coverage-all clean-test docs clean-docs install uninstall
 
 # Missing dependency files are expected on a clean tree. Existing files carry

README.md

@@ -34,7 +34,7 @@ See [Measurement Capabilities](documents/CAPABILITIES.md) for the full measureme
 - macOS 26 or later on Apple Silicon (ARM64)
 - Xcode Command Line Tools for source builds
 - GoogleTest from Homebrew for the test suite
-- Python 3 for the script-example entry test included in the aggregate `make test-all` gate; `jq` is optional for JSON
+- Python 3 for build provenance and the Python tests included in the aggregate `make test-all` gate; `jq` is optional for JSON
   inspection and the jq-backed latency-script path
 - Metal modes: a unified-memory device with `MTLGPUFamilyApple7` or compatible later-family capability; LLM Metal also
   requires Tier 2 argument buffers and `maxBufferLength >= 256 MiB`
@@ -139,6 +139,9 @@ memory_benchmark --llm-memory --weight-size-mb 64 --layers 4 \
   --kv-layout paged --kv-block-tokens 16 --iterations 1 --count 3 --seed 42
 ```
 
+Query heads classify the model; they do not add executed attention math. The prefill query tile
+defines synthetic prefix rereads, not a real inference kernel’s cache or SRAM tiling.
+
 Run one full-prompt prefill operation per scenario with two-token attention query tiles:
 
 ```bash
@@ -176,7 +179,7 @@ checkpoints are required; see the [Machine-Readable CLI API](documents/API.md) s
 | `--analyze-core2core` | Calibrated two-thread acquire/release token-protocol round-trip latency under best-effort macOS scheduler hints. |
 | `--gpu-bandwidth` | Standalone Metal GPU read/write/copy effective compute-payload bandwidth. |
 | `--llm-memory` | Standalone synthetic LLM memory profile: CPU or Metal decode/prefill with contiguous or paged KV. |
-| `--sweep <key=a,b>` | Cartesian parameter sweep for supported CPU, pattern, TLB, and core-to-core modes; requires `--output`. GPU schema 1 and LLM schema 1 do not support sweeps. |
+| `--sweep <key=a,b>` | Cartesian parameter sweep for supported CPU, pattern, TLB, and core-to-core modes; requires `--output`. GPU schema 1 and LLM schema 2 do not support sweeps. |
 
 Primary modes are intentionally separate and accept different option sets. Use `memory_benchmark -h` or the [User Manual](documents/MANUAL.md) for defaults, valid combinations, and the complete option reference.
 
@@ -243,7 +246,7 @@ memory_benchmark --gpu-bandwidth --buffer-size 512 --count 3 --seed 42 --output
   >gpu_bandwidth.json 2>gpu_bandwidth.log
 ```
 
-Reproducible fixed-work LLM memory profile with atomic scenario and command-terminal file checkpoints:
+Reproducible fixed-work LLM memory profile with bounded completed-loop and command-terminal atomic file snapshots:
 
 ```bash
 caffeinate -i -d memory_benchmark --llm-memory --weight-size-mb 4096 --layers 32 \
@@ -265,7 +268,7 @@ For prefill, replace the decode context with explicit prompt/tile geometry:
 
 Add `--kv-layout paged --kv-block-tokens 16` to combine that prefill geometry with deterministic paged KV.
 
-The same schema 1 payload can be captured once from final-only stdout:
+The same LLM schema 2 payload can be captured once from final-only stdout:
 
 ```bash
 memory_benchmark --llm-memory --weight-size-mb 64 --layers 4 \
@@ -322,9 +325,11 @@ They retain the top-level `version` as provenance but do not require a particula
 translate released standard schema 2, unversioned historical standard JSON layouts, or other methodology identities.
 Consumers making conclusions should reject incomplete or interrupted runs according to the mode-specific status fields.
 Every result-producing direct command or CPU sweep using `--output -` reserves stdout for one final JSON document and
-routes its post-parse human transcript to stderr; file output is atomic. LLM file output checkpoints after each terminal
-scenario measurement and at command terminal, while its stdout checkpoints remain logical lazy transitions followed by
-one final document. Exact process acceptance rules are in the
+routes its post-parse human transcript to stderr; file output is atomic. LLM files receive a progress snapshot every
+`K=max(1,ceil(count/8))` completed loops plus a terminal snapshot: at most eight progress writes, with up to `3K`
+completed attempts potentially missing after abrupt termination. Stdout emits one final document. A correct complete
+one-loop LLM run can have `run_accepted: true` while position balance is incomplete and quality is insufficient for a
+comparison. Inspect sample count, observed CV, duration and environment separately. Exact process acceptance rules are in the
 [Machine-Readable CLI API](documents/API.md), with schema and checkpoint details in the
 [User Manual](documents/MANUAL.md), [Technical Specification](documents/TECHNICAL_SPECIFICATION.md), and mode
 whitepapers.
@@ -369,7 +374,7 @@ recognizes the current console labels only and is neither JSON-schema nor histor
 - [TLB Analysis Whitepaper](documents/TLB_ANALYSIS_WHITEPAPER.md): paired analysis, boundary rules, confidence model, and JSON verification contract.
 - [Core-to-Core Whitepaper](documents/CORE_TO_CORE_WHITEPAPER.md): LDAR/STLR handoff protocol, scheduler-hint scenarios, and JSON schema.
 - [GPU Bandwidth Whitepaper](documents/GPU_BANDWIDTH_WHITEPAPER.md): Metal methodology, timing, validation, resource model, and interpretation limits.
-- [LLM Memory Profile Whitepaper](documents/LLM_MEMORY_PROFILE_WHITEPAPER.md): generic schema-v1 vocabulary plus the
+- [LLM Memory Profile Whitepaper](documents/LLM_MEMORY_PROFILE_WHITEPAPER.md): schema-2 vocabulary plus the
   active CPU and Metal decode/prefill traffic, timing, checksum, and interpretation contracts.
 - [Apple M5 LLM CPU-decode working-set samples](results/0.63.0/AppleM5_LLM_working_set_scaling.md): two complete
   0.63.0 JSON runs and their observed working-set scaling.
@@ -393,7 +398,7 @@ make test-integration
 make test-all
 ```
 
-`make test-all` requires Python 3; it runs all GTest cases followed by the focused script-example entry test. `jq` is
+`make test-all` requires Python 3; it runs all GTest cases followed by the script-example and independent LLM verifier tests. `jq` is
 not required by the test gate.
 
 Generate isolated LLVM production-source coverage reports under `/tmp`:
@@ -403,9 +408,26 @@ make coverage-unit
 make coverage-all
 ```
 
-See [CONTRIBUTING.md](documents/CONTRIBUTING.md) for contribution guidance and [Project Structure](documents/PROJECT_STRUCTURE.md) for
+See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution guidance and [Project Structure](documents/PROJECT_STRUCTURE.md) for
 repository navigation and the current test-suite map. C++ reference documentation can be generated with `make docs`.
 
+CPU LLM results retain the original Mach tick boundaries and timebase for duration reconstruction.
+The build embeds compiler, flags, SDK/deployment target and Git provenance; the command hashes the
+executable once before tasks. These fields bind available artifacts and do not constitute signed
+execution attestation. See the [LLM process contract](documents/API.md) for availability and numeric rules.
+Python 3 is required to generate build provenance.
+
+Verify a saved LLM schema-2 result independently (standard library only):
+
+```bash
+python3 script-examples/verify_llm_result.py llm-result.json --binary ./memory_benchmark --require-raw-timing
+make test-llm-verifier
+```
+
+The verifier reports artifact consistency separately from run acceptance and identifies missing timing/build
+evidence. Its bounded arithmetic reconstructs logical work and checksums; it does not attest execution or measure
+physical DRAM traffic. See [the verifier contract](documents/API.md#independent-llm-artifact-verifier).
+
 ## Scope and Safety
 
 This project intentionally does not target Intel Macs or other operating systems, provide a GUI, or host a public leaderboard/backend.

build-support/generate_provenance.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python3
+"""Freeze source/build inputs at make time; update the header only on change."""
+
+import json
+import os
+from pathlib import Path
+import shlex
+import subprocess
+import sys
+
+
+def command(args):
+    try:
+        return subprocess.check_output(args, stderr=subprocess.DEVNULL, text=True).strip()
+    except (OSError, subprocess.CalledProcessError):
+        return None
+
+
+def main():
+    # A tarball placed inside another checkout must not inherit that repo's HEAD.
+    top = command(["git", "rev-parse", "--show-toplevel"])
+    own_checkout = top is not None and Path(top).resolve() == Path.cwd().resolve()
+    commit = command(["git", "rev-parse", "--verify", "HEAD"]) if own_checkout else None
+    status = command(["git", "status", "--porcelain", "--untracked-files=normal"]) if commit else None
+    compiler = shlex.split(os.environ["PROVENANCE_CXX"])
+    manifest = dict(
+        manifest_version=1,
+        status="available",
+        reason_code="valid",
+        binary_sha256=None,
+        git_commit=commit,
+        git_dirty=bool(status) if status is not None else None,
+        compiler=command(compiler + ["--version"]),
+        compile_flags={
+            key: os.environ["PROVENANCE_" + key.upper()] for key in ("cxxflags", "test_cxxflags", "asflags")
+        },
+        link_flags=os.environ["PROVENANCE_LDFLAGS"],
+        target_arch=command(compiler + ["-arch", "arm64", "-dumpmachine"]),
+        sdk=command(["xcrun", "--sdk", "macosx", "--show-sdk-version"]),
+        min_os=os.environ.get("MACOSX_DEPLOYMENT_TARGET"),
+    )
+    if any(
+        manifest[key] is None
+        for key in ("git_commit", "git_dirty", "compiler", "target_arch", "sdk", "min_os")
+    ):
+        manifest.update(status="partial", reason_code="build-fields-unavailable")
+    payload = json.dumps(manifest, sort_keys=True)
+    if len(payload) > 16384:
+        raise SystemExit("build provenance exceeds 16 KiB budget")
+    header = (
+        "// Generated build inputs; do not edit.\n#define LLM_BUILD_MANIFEST_JSON "
+        + json.dumps(payload)
+        + "\n"
+    )
+    path = Path(sys.argv[1])
+    if not path.exists() or path.read_text() != header:
+        temporary = path.with_suffix(".tmp")
+        temporary.write_text(header)
+        temporary.replace(path)
+
+
+if __name__ == "__main__":
+    main()