Version 0.64.0

CHANGELOG.md

@@ -5,6 +5,27 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.64.0] - 2026-09-06
+
+### Added
+  - **Independent LLM artifact verifier**: Added `script-examples/verify_llm_result.py` for all eight current CPU/Metal decode/prefill and contiguous/paged profiles. Independent Python arithmetic checks geometry, work, checksums, timing, rates, accepted sample populations, and statistics. Optional `--binary` checks the executable file's SHA-256, and `--require-raw-timing` requires original CPU timing evidence. Verdicts distinguish consistent accepted results, inconsistent or unaccepted results, and unsupported evidence or resource limits. Added `make test-llm-verifier` to the aggregate `make test-all` gate.
+  - **LLM build provenance and original CPU timing**: Results now retain a build manifest with available Git revision and dirty state, compiler, build flags, architecture, SDK, deployment target, and executable SHA-256. CPU measurements and excluded attempts retain original Mach start/stop/delta ticks and timebase for elapsed-time reconstruction. Python 3 is now required for build provenance generation; missing evidence remains explicitly unavailable or partial.
+  - **CPU and Metal checksum fault coverage**: Added independent arithmetic goldens, corruption and work-mutation tests, real-GPU reduction-boundary tests, and wider prefill boundary coverage. Documented profile-specific checksum collisions and final-state sampling limits, with acceptance validation across all eight profiles, file/stdout output, graceful interruption, and persistent multi-GiB Metal resources.
+
+### Changed
+  - **LLM output advances to JSON schema 2 and v2 methodologies**: All eight profiles now use `llm-memory-v2-<backend>-<phase>-<layout>`. Canonical scenario plans and expected checksums are stored once, measurements and calibration attempts reference those plans, and aggregates identify their accepted measurements. Geometry, model context, layout, resources, and component identities have explicit owners, with strict reference and null/status semantics. The API documents the schema-1-to-schema-2 field map; no compatibility aliases or fallback reader are provided.
+  - **LLM validation retains separate observations**: Named checks now preserve applicability, completion, validity, and reasons for structure, final KV writes or unchanged append state, and applicable padding checks. A checksum match cannot hide a failed final-state check, and unevaluated checks remain unresolved instead of appearing successful.
+  - **LLM file checkpoints are bounded by loop count**: Files now receive progress snapshots every `K=max(1,ceil(count/8))` completed loops, normally at most eight progress snapshots plus one terminal snapshot. Abrupt termination can lose up to `3K` completed attempts since the last successful snapshot. Exact `--output -` prepares only the terminal document, and disabled output skips JSON construction. Writer counters report actual prior persistence attempts; checkpoint failures remain terminal and are not retried.
+  - **LLM result collection avoids repeated retained data and statistics work**: Calibration keeps compact actual checksum evidence, canonical plans own expected values, and exact statistics are prepared at snapshot boundaries from accepted measurement IDs. Memory admission accounts for retained evidence, canonical-plan storage, statistics scratch, and simultaneous JSON construction and serialization peaks.
+  - **CPU task preparation validates inputs once per call**: Removed duplicate plan/resource validation within expected-checksum calculation while retaining fresh validation of borrowed plans and materialized resources before each executor call starts workers or timing.
+  - **LLM acceptance and comparison quality are separate**: `results_complete` describes the measured population, while `run_accepted` also requires valid execution evidence and no known command or checkpoint failure. Position balance, sample count, CV, duration, and environment remain separate comparison criteria. Documentation clarifies that cyclic order balances scenario positions without guaranteeing predecessor-pair balance, and that comparisons must account for conditioning and output cadence.
+  - **CLI help and reference documentation updated**: Refreshed general and LLM help, the machine API, manual, whitepaper, and supporting references for schema 2, verification, checkpoint behavior, and comparison requirements. Clarified the synthetic meaning of query heads, prefill tiles, and theoretical attention quantities.
+  - **Test coverage cleaned up**: Removed redundant contract tests and replaced brittle source-text assertions with focused semantic coverage. Hardware-dependent LLM cases are classified as integration tests, and the Makefile now builds Objective-C++ test sources with test flags and ARC.
+
+### Fixed
+  - **CPU contiguous decode validates final KV append bytes**: After timing stops and workers join, KV-bearing scenarios now verify every final K/V append byte against the expected pattern. Corruption produces an invalid measurement with retained diagnostics and cannot enter accepted aggregates, even when the timed checksum matches.
+  - **Theoretical prefill overflow no longer rejects valid memory work**: Overflowing model-context attention-pair or FMA quantities become nullable values with an explicit arithmetic-overflow reason. Exact byte, work, allocation, and execution guardrails remain enforced.
+
 ## [0.63.0] - 2026-08-23
 
 ### Added

CITATION.cff

@@ -1,16 +1,16 @@
 cff-version: 1.2.0
 message: "If you use this software in research, benchmarks, or publications, please cite it as below."
 
-title: "macOS Memory Benchmark"
+title: "macOS-memory-benchmark"
 type: software
 
 authors:
   - given-names: "Timo"
     family-names: "Heimonen"
     orcid: "https://orcid.org/0009-0004-0023-2407"
 
-version: "0.63.0"
-date-released: 2026-08-23
+version: "0.64.0"
+date-released: 2026-09-06
 
 repository-code: "https://github.com/timoheimonen/macOS-memory-benchmark"
 url: "https://github.com/timoheimonen/macOS-memory-benchmark"

Makefile

@@ -71,6 +71,19 @@ TARGET = memory_benchmark
 # Default target: build the executable
 all: $(TARGET)
 
+# Recompute provenance before considering objects. A changed source revision,
+# dirty state or build flags invalidates every object, avoiding mixed manifests.
+export PROVENANCE_CXX = $(CXX)
+export PROVENANCE_CXXFLAGS = $(CXXFLAGS)
+export PROVENANCE_TEST_CXXFLAGS = $(TEST_CXXFLAGS)
+export PROVENANCE_ASFLAGS = $(ASFLAGS)
+export PROVENANCE_LDFLAGS = $(LDFLAGS) $(APPLE_FRAMEWORKS)
+.build-provenance.h: FORCE
+	python3 build-support/generate_provenance.py $@
+
+.PHONY: FORCE
+FORCE:
+
 # Rule for linking the executable from object files
 $(TARGET): $(OBJ_FILES)
 	@echo "Linking $(TARGET)..."
@@ -80,7 +93,8 @@ $(TARGET): $(OBJ_FILES)
 # Test directory and files
 TEST_DIR = tests
 TEST_SRCS := $(sort $(wildcard $(TEST_DIR)/*.cpp))
-TEST_OBJS := $(TEST_SRCS:.cpp=.o)
+TEST_OBJCXX_SRCS := $(sort $(wildcard $(TEST_DIR)/*.mm))
+TEST_OBJS := $(TEST_SRCS:.cpp=.o) $(TEST_OBJCXX_SRCS:.mm=.o)
 
 # Dependency files generated by DEPFLAGS. Test dependencies include tests/*.h
 # helpers through the compiler's actual include graph rather than a manually
@@ -98,13 +112,18 @@ TEST_LIB_OBJS := $(filter-out main.o, $(OBJ_FILES))
 # Recompile once after a Makefile change so an existing pre-dependency-file
 # workspace cannot keep stale objects. Subsequent header changes are tracked by
 # the generated .d files.
-$(OBJ_FILES) $(TEST_OBJS): Makefile
+$(OBJ_FILES) $(TEST_OBJS): Makefile .build-provenance.h
 
 # Rule for compiling test files (must come before generic %.o rule)
 $(TEST_DIR)/%.o: $(TEST_DIR)/%.cpp
 	@echo "Compiling test $< -> $@..."
 	$(CXX) $(TEST_CXXFLAGS) $(DEPFLAGS) -c $< -o $@
 
+# Objective-C++ tests retain test flags and ARC, including generated dependencies.
+$(TEST_DIR)/%.o: $(TEST_DIR)/%.mm
+	@echo "Compiling Objective-C++ test $< -> $@..."
+	$(CXX) $(TEST_CXXFLAGS) -fobjc-arc $(DEPFLAGS) -c $< -o $@
+
 # Objective-C++ production boundaries share the normal C++ build settings.
 %.o: %.mm
 	@echo "Compiling Objective-C++ $< -> $@..."
@@ -135,11 +154,16 @@ test-integration: $(TARGET) $(TEST_TARGET)
 test-script-examples:
 	python3 -m unittest -v tests/test_script_examples.py
 
-# All tests (unit tests + integration tests + bundled script examples)
+# Independent LLM artifact contract and mutation tests.
+test-llm-verifier:
+	python3 -m unittest -v tests/test_llm_result_verifier.py
+
+# All tests (unit, integration, bundled examples, and independent LLM verifier)
 test-all: $(TARGET) $(TEST_TARGET)
-	@echo "Running all tests (unit + integration + bundled script examples)..."
+	@echo "Running all tests (unit + integration + bundled examples + LLM verifier)..."
 	./$(TEST_TARGET)
 	$(MAKE) test-script-examples
+	$(MAKE) test-llm-verifier
 
 # Reproducible production C++ source coverage in an isolated /tmp build.
 coverage-unit:
@@ -217,7 +241,7 @@ uninstall:
 	@echo "$(TARGET) uninstalled successfully."
 
 # Define targets that don't correspond to files
-.PHONY: all clean test test-integration test-script-examples test-all \
+.PHONY: all clean test test-integration test-script-examples test-llm-verifier test-all \
 	coverage-unit coverage-all clean-test docs clean-docs install uninstall
 
 # Missing dependency files are expected on a clean tree. Existing files carry