Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.63.1] - Unreleased

### Changed
- **CLI help updated**.
- **Test coverage cleaned up**: Removed redundant tests.

## [0.63.0] - 2026-08-23

### Added
Expand Down
20 changes: 11 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -305,7 +305,8 @@ Treat benchmark values as measurements of the configured workload under the obse
prefix sharing, sliding-window KV, growing context, and model loading.
- The LLM traffic classification version `llm-exact-weight-vs-kv-read-payload-v1` compares exact weight and KV-read
bytes only. `near_crossover` means exact equality and is not a measured hardware-bottleneck claim.
- The repository includes two current [Apple M5 CPU-decode working-set samples](results/0.63.0/AppleM5_LLM_working_set_scaling.md)
- The repository includes two
[Apple M5 CPU-decode working-set samples recorded with 0.63.0](results/0.63.0/AppleM5_LLM_working_set_scaling.md)
with links to their complete JSON records. They illustrate scaling across 384 MiB and 1,536 MiB data mappings.
- TLB-locality controls pointer-chain construction, not hardware TLB residency. Standard locality comparisons combine cache, locality, and translation effects; use `--analyze-tlb` for controlled translation-boundary conclusions.
- Core-to-core results are scheduler-influenced acquire/release token-protocol measurements. They do not directly observe
Expand All @@ -315,9 +316,10 @@ Treat benchmark values as measurements of the configured workload under the obse
JSON output records completion and nullable measurement state instead of using zero for unavailable results. Current
standard schema 3 requires `configuration.mode: "benchmark"`, a string `configuration.output_file` that preserves the
raw output target, plus boolean `results_complete` and `conclusions_valid` fields. The bundled standard-memory examples
track the current producer, require its exact top-level `version` (currently `0.63.0`), sanity-check the current
result locally, and read current schema-3 paths directly. They do not provide compatibility for released standard
schema 2, unversioned historical standard JSON layouts, or any other explicit standard version.
accept compatible producer releases by checking the standard mode, schema 3, the exact
`benchmark-v2-calibrated-seeded-balanced` methodology, completion state, and the shape of the fields they consume.
They retain the top-level `version` as provenance but do not require a particular software release. They do not
translate released standard schema 2, unversioned historical standard JSON layouts, or other methodology identities.
Consumers making conclusions should reject incomplete or interrupted runs according to the mode-specific status fields.
Every result-producing direct command or CPU sweep using `--output -` reserves stdout for one final JSON document and
routes its post-parse human transcript to stderr; file output is atomic. LLM file output checkpoints after each terminal
Expand All @@ -340,11 +342,11 @@ python3 script-examples/plot_cache_percentiles.py \

The sweep script prefers the repository's local `./memory_benchmark`, then falls back to `memory_benchmark` from
`PATH`; set `BENCHMARK_CMD=/path/to/memory_benchmark` to override either choice. Whichever producer is selected must
emit complete current standard schema 3. The sweep helpers return a non-zero status if a planned run fails or does not
produce a complete, parseable result.
emit a complete compatible standard schema-3 result with the expected methodology. The sweep helpers return a non-zero
status if a planned run fails or does not produce a complete, parseable result.

The two standard-result plotters require explicit current inputs; archived 0.53.x standard JSON is retained as
historical evidence and is not a valid current input:
The two standard-result plotters require explicit compatible inputs; archived 0.53.x standard JSON is retained as
historical evidence and is not a compatible input:

```bash
python3 script-examples/plot_M4vsM5_benchmark_comparison.py \
Expand All @@ -370,7 +372,7 @@ recognizes the current console labels only and is neither JSON-schema nor histor
- [LLM Memory Profile Whitepaper](documents/LLM_MEMORY_PROFILE_WHITEPAPER.md): generic schema-v1 vocabulary plus the
active CPU and Metal decode/prefill traffic, timing, checksum, and interpretation contracts.
- [Apple M5 LLM CPU-decode working-set samples](results/0.63.0/AppleM5_LLM_working_set_scaling.md): two complete
current-version JSON runs and their observed working-set scaling.
0.63.0 JSON runs and their observed working-set scaling.

Runtime behavior and `memory_benchmark -h` are the authoritative sources when documentation differs.

Expand Down
32 changes: 18 additions & 14 deletions documents/API.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Machine-Readable Command-Line API

This document defines the supported process-level integration contract for `memory_benchmark` 0.63.0. It describes how
software launches a benchmark, separates machine-readable output from the human transcript, and decides whether a JSON
result is safe to consume. The generated Doxygen pages document C++ internals; they are not this process API.
This document defines the supported process-level integration contract for the current `memory_benchmark`
implementation. It describes how software launches a benchmark, separates machine-readable output from the human
transcript, and decides whether a JSON result is safe to consume. The generated Doxygen pages document C++ internals;
they are not this process API.

Runtime behavior and executable integration tests are authoritative if this document and the implementation differ.
The documented process runtime baseline is macOS 26 or later on Apple Silicon (ARM64).
Expand Down Expand Up @@ -199,7 +200,7 @@ predicate.
Each `runs[].result` in a sweep retains its nested mode's own schema-version field and completeness contract. Nested
standard classification recognizes only current schema 3 with `configuration.mode == "benchmark"` plus typed
`results_complete`, `conclusions_valid`, and `configuration.output_file` fields; standard schema 2 and every other
standard version are unsupported. Complete, partial, interrupted, and failed current schema-3 evidence remains
standard schema version are unsupported. Complete, partial, interrupted, and failed current schema-3 evidence remains
classifiable and retained rather than being discarded by the complete-result consumer boundary. A non-zero nested
execution that initialized a result remains in the envelope: its attempt is failed, but the payload is not replaced by
a generic diagnostic. In particular, nested TLB `tlb_analysis.status == "error"` maps to a failed sweep attempt without
Expand Down Expand Up @@ -557,12 +558,12 @@ persistence and nested file writes are disabled. Schema 3 requires boolean `resu
the producer makes `conclusions_valid` true exactly when `results_complete` is true, while consumers must still check
the explicit status and both booleans shown in the table.

The bundled standard-memory example scripts are maintained in lockstep with the current producer. Each script performs
only the local version, completion, and field sanity checks needed before reading its current schema-3 metric paths. For
the current producer that includes exact top-level `version == "0.63.0"` in addition to the standard identity and
completeness fields above. The examples are not a versioned compatibility library. Released standard schema 2,
unversioned historical standard JSON layouts, and every other explicit standard version are unsupported inputs and are
not routed through a metric-shape fallback.
The bundled standard-memory example scripts accept compatible producer releases. Before reading their standard
schema-3 metric paths, they require `configuration.mode == "benchmark"`, schema 3, methodology
`benchmark-v2-calibrated-seeded-balanced`, the completion fields above, and the expected types for every consumed
field. They retain a non-empty top-level `version` string as release provenance, but exact software-version equality is
not an acceptance condition. The examples do not translate released standard schema 2, unversioned historical
standard JSON layouts, or other methodology identities through a metric-shape fallback.

Graceful interruption or runtime failure after a representable result state has been initialized emits the available
partial, interrupted, error, or failed JSON snapshot. The execution status and payload are independent: a non-zero status
Expand Down Expand Up @@ -610,6 +611,7 @@ memory_benchmark --benchmark --only-bandwidth --buffer-size 512 --count 5 --seed
>benchmark.json 2>benchmark.log
jq -e '.configuration.mode == "benchmark" and
.configuration.benchmark_schema_version == 3 and
.configuration.methodology_version == "benchmark-v2-calibrated-seeded-balanced" and
(.configuration.output_file | type) == "string" and
.status == "complete" and .results_complete == true and
.conclusions_valid == true' benchmark.json
Expand Down Expand Up @@ -667,7 +669,8 @@ jq -e '.mode == "llm_memory" and .schema_version == 1 and
## Compatibility policy

- `version`, the GPU `software_version` field, and LLM `software` identity identify the application release; none is a
result schema version.
result schema version or compatibility selector. Consumers may retain and display this provenance independently of
schema and methodology acceptance.
- Current standard schema 3, pattern schema 3, TLB schema 4, core-to-core schema 2, GPU schema 1, and LLM schema 1 remain
authoritative at their existing locations. The schema field is intentionally not normalized across these established
payloads.
Expand All @@ -676,9 +679,10 @@ jq -e '.mode == "llm_memory" and .schema_version == 1 and
consumers must use the current generic fields, tolerate unknown additive evidence fields, and validate every known
field they consume. Future removal/rename/type/meaning changes require schema-version review; software identity alone
is not a compatibility substitute.
- Bundled standard-memory examples track the current producer and read current standard schema-3 metric paths directly
after local sanity checks. They provide no compatibility layer for released standard schema 2, unversioned
historical standard JSON layouts, or any other explicit standard version.
- Bundled standard-memory examples accept compatible software releases only when standard mode, schema 3, the exact
methodology identity, completion state, and consumed field shapes match. They retain software-version provenance but
provide no translation layer for released standard schema 2, unversioned historical standard JSON layouts, or other
methodology identities.
- Both general and core-to-core sweep envelopes use `configuration.sweep_schema_version == 1`; nested results keep their
independent mode schema versions.
- Additive optional fields may remain within a schema version only when old consumers can safely ignore them.
Expand Down
7 changes: 4 additions & 3 deletions documents/CAPABILITIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,9 +223,10 @@ current support matrix and process acceptance procedure are in the [Machine-Read

Current standard results use schema 3, which requires `configuration.mode: "benchmark"`, a string
`configuration.output_file`, plus boolean `results_complete` and `conclusions_valid`. Bundled standard-memory examples
track the current producer, perform local sanity checks, and read its current schema-3 metric paths directly. They do
not support released standard schema 2, unversioned historical standard JSON layouts, or any other explicit standard
version.
accept compatible producer releases after validating standard mode, schema 3, methodology
`benchmark-v2-calibrated-seeded-balanced`, completion state, and consumed field shapes. Software version is retained
only as provenance. They do not translate released standard schema 2, unversioned historical standard JSON layouts, or
other methodology identities.

Sweep output retains completed evidence even when a later run stops. Consumers should check the mode-specific status and completeness indicators before using aggregate conclusions. Exact schemas, checkpoint behavior, and inspection examples are in the [User Manual](MANUAL.md#json-output-format) and [Technical Specification](TECHNICAL_SPECIFICATION.md#18-json-output-contract).

Expand Down
6 changes: 3 additions & 3 deletions documents/CORE_TO_CORE_WHITEPAPER.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

**memory_benchmark — Technical Whitepaper**

*Applies to version 0.63.0; archived examples may use older methodologies*
*Applies to the current repository implementation; archived examples may use older methodologies*

---

Expand All @@ -13,7 +13,7 @@ protocol between two POSIX threads. The result includes the complete polling and
scheduler effects; it is not an isolated measurement of one physical cache-line migration or the coherence fabric.
Software version 0.58.0 introduced the change from fixed work in a fixed scenario order to the auditable
v2 calibrated/balanced design. Version 0.61.1 introduced distinct 128-byte token and control blocks and the current
methodology identity; version 0.63.0 retains that design as
methodology identity. The current implementation retains that design as
`core2core-v3-calibrated-balanced-auditable-128b-isolation`:

- every scheduler-hint scenario receives its own excluded pilot after a long calibration warmup;
Expand Down Expand Up @@ -392,7 +392,7 @@ and one-way arrays are complete for the illustrated scenario.
},
"execution_time_sec": 4.2,
"timestamp": "YYYY-MM-DDTHH:MM:SSZ",
"version": "0.63.0"
"version": "<software-version>"
}
```

Expand Down
4 changes: 2 additions & 2 deletions documents/GPU_BANDWIDTH_WHITEPAPER.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Metal GPU Memory Bandwidth Whitepaper

- **Software version:** 0.63.0
- **Software version:** release provenance; not a compatibility selector
- **JSON schema:** 1
- **Methodology:** `gpu-bandwidth-v1-private-runtime-single-cmdbuf-calibrated-balanced`
- **Platform:** macOS on Apple Silicon
Expand Down Expand Up @@ -455,7 +455,7 @@ already-computed measurement state.
## 14. GPU JSON Schema 1

The top-level discriminator is independent of current standard schema 3. Released standard schema 2 is historical and
unsupported by the bundled standard-memory examples, which track the current producer:
unsupported by the bundled standard-memory examples, which accept compatible standard schema-3 producer releases:

```json
{
Expand Down
9 changes: 5 additions & 4 deletions documents/LATENCY_WHITEPAPER.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

## 1. Purpose

This document explains how `macOS-memory-benchmark` version 0.63.0 measures memory latency on Apple Silicon.
This document explains how the current `macOS-memory-benchmark` implementation measures memory latency on Apple
Silicon.

The latency path is designed to measure **load-to-use delay** (pointer chasing), not bulk throughput.
It combines:
Expand Down Expand Up @@ -254,9 +255,9 @@ analysis.
`pooled_sample_distribution`, with values and loop-boundary metadata kept separate from continuous loop headlines.
- Current standard schema 3 does not serialize the unversioned historical
`chain_diagnostics.unique_pages_touched` blocks. Do not use the old `main_memory.latency.chain_diagnostics` or
`cache.*.latency.chain_diagnostics` paths for version 0.63.0 output. Released standard schema 2 and unversioned
historical standard JSON remain archival evidence and are unsupported by the bundled examples, which track the
current producer and read current schema-3 paths directly.
`cache.*.latency.chain_diagnostics` paths for standard schema-3 output. Released standard schema 2 and unversioned
historical standard JSON remain archival evidence and are unsupported by the bundled examples, which accept the
current methodology across compatible software releases and read schema-3 paths directly.

When `--latency-tlb-locality-kb` is not explicitly supplied, standard main-memory latency also runs three paired rounds
of 16 KiB-locality and global-random chains. The first-measured layout alternates by round, and
Expand Down
2 changes: 1 addition & 1 deletion documents/LLM_MEMORY_PROFILE_WHITEPAPER.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The current contract is identified by:

| Property | Value |
|---|---|
| Software version | `0.63.0` |
| Software version | Release provenance; not a schema or methodology selector |
| Mode | `llm_memory` |
| Backend | `cpu` or `metal` |
| JSON schema | `1` |
Expand Down
Loading
Loading