Repository navigation
Research release: SDPA, KV-cache experiments and reproducible evaluation - #1
Merged
Merged
Conversation
…ctness tests
- MiniLM.set_attention_backend('auto'|'sdpa'|'reference') switches attention
math without touching weights; 'auto' preserves the original code paths.
- reference_causal_attention: explicit QK^T/sqrt(d) softmax V with a
bottom-right aligned causal mask (fp32 softmax).
- RoPE angles are now computed in float32. Previously bf16 RoPE at 2048
positions had max error ~7.3 versus a ~0.015 rounding floor.
- greedy_decode: deterministic batched decoding with or without KV cache.
- Tests: float64 oracle agreement, causality per backend, chunked cache
prefill, RoPE complex/relative-position properties, cached vs uncached
greedy equivalence (single token, full context, batch, sliding window),
gradient accumulation and bitwise checkpoint resume.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eport builder - provenance: environment/git capture, summary stats (median, p95, IQR, CV) and exclusive-create JSON records that never overwrite earlier runs. - benchmark_attention: matched forward/prefill/decode timing with rotated backend order, per-step synchronized decode timing, and per-config fresh subprocess memory probes. - compare_attention_quality: float64 oracle logit error, top-1 agreement, fp32-loss perplexity and greedy-generation agreement per device/dtype. - build_experiment_report: tables and SVG charts generated only from records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cords The MPS driver allocates ~1 GiB heaps, so driver_allocated_memory deltas were quantized and not comparable. Sample torch.mps.current_allocated_memory() after every aten op instead. Earlier MPS records are kept but superseded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Attention benchmark records for CPU fp32, MPS fp32 and MPS fp16 (first MPS records with pool-size memory kept but superseded). - Attention quality record for seeds 42/43/44 on CPU and MPS (fp32/fp16/bf16). - Training records for the three retrained seeds. - docs/EXPERIMENT_RESULTS.md: generated tables plus analysis and limitations. - docs/implementation_audit.md: October 2026 audit with evidence per claim. - Attention figures generated from records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arios Separates the cache effect from the attention kernel (uncached SDPA vs uncached default), times every step, checks token equivalence, and records cache bytes and MPS per-step allocator peaks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… checks reference_causal_attention called .float() on scores, which downcast float64 inputs to float32. The float64 'oracle' in compare_attention_quality.py therefore ran its softmax in float32; its quality record is superseded. Scores now promote to at least float32 and never below the input dtype. SDPA and reference paths now apply nn.MultiheadAttention's attention dropout in training mode (inert at the published dropout=0). Tests: per-backend gradients vs a float64 oracle (learned and RoPE), torch.autograd.gradcheck of the attention layer per backend, dropout only in training, float64 reference equals SDPA to 1e-12. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…apture
train_tiny.py --attention-backend {auto,sdpa,reference}; auto (the original
nn.MultiheadAttention path) remains the default. Checkpoints record the
backend and resume refuses a backend change. Each run writes
run_summary.json (train seconds excluding evaluation, tokens/s, per-eval
seconds, process peak RSS). evaluate_runs.py scores selected checkpoints on
named splits and writes one record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…corpus) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orrected-oracle quality Default-backend checkpoints are bitwise identical to the original v1 runs. The first auto seed-42 training record was written while unrelated doc edits were uncommitted (dirty=true); it is superseded by a clean re-record. The attention_quality record supersedes 20261022T034349Z, whose float64 oracle ran its softmax in float32. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tions benchmark_training_step.py times forward+backward+AdamW per backend with identical weights and batches, rotated order, and first-step memory in a fresh process. record_training_run.py resolves relative data paths. The report builder gains KV-cache, cache-memory, matched-training and v2 sections. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ched test book Cleaning (applied before splitting): Gutenberg body extraction verified by SHA-256; per-book exact start/end heading markers with asserted occurrence counts (drops Darwin's index, Einstein's symbol glossary, Faraday's notes and front matter); paragraph rules drop headings, fully bracketed editorial paragraphs and multi-line numeric lists/tables. Displayed equations are kept. Faraday's Chemical History of a Candle is the test book. Held-out paragraphs that are exact or >=50% word-8-gram duplicates of training text are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ol script Legacy science-v1 (data/science/) is unchanged and tested against its original manifest. v2: Darwin + Einstein train/validation (cleaned), Faraday test book (untouched by training and selection). No exact or near duplicates across train/validation/test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…valuation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- EXPERIMENT_RESULTS.md: protocol/comparability table; Part A (systems experiments and legacy science-v1, including the matched training and training-step benchmark) and Part B (science-v2 corpus and test results); SDPA recommendation with the reason the default is not switched here. - research_report.md: October 2026 update with exact commands; the original September report is kept verbatim as the legacy v1 baseline. - implementation_audit.md: addendum (float64 downcast, dropout, upstream hash change, superseded dirty record) and updates to resolved findings. - README: v2 summary, KV-cache figure, corrected reproduction steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- DEFAULT_ATTENTION_BACKEND = 'sdpa' for the model, training and evaluation. - The original nn.MultiheadAttention path (all science-v1 results) is now 'mha'; the pre-release name 'auto' is accepted as an alias, and checkpoints without a backend field are treated as mha. Checkpoints store canonical names. - 'reference' remains the explicit softmax implementation. - Every legacy reproduction script and command selects mha explicitly (sdpa_v1.sh, v2_protocol.sh); evaluate.py, generate.py, quantize.py and the benchmark scripts gain --attention-backend. benchmark_generation.py, generate.py and publish_science_results.py load checkpoints with weights_only=True. - Report builder reads historical 'auto' records as mha; regenerated EXPERIMENT_RESULTS.md differs only in labels (all 2,223 numbers identical). - New tests: default and explicit selection, legacy alias and missing-field checkpoints, resume guard, evaluate.py backend handling, state_dict compatibility, per-backend causal masking, evaluation correctness, backend/device/dtype tolerance table (CPU fp64/fp32, MPS fp32/fp16), and SDPA dropout_p == 0 in evaluation. The redundant mha+RoPE gradcheck skip is replaced by an equality assertion (mha + RoPE runs the SDPA path). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dation - prepare_science_corpus.py (v1) verifies the regenerated splits against the published hashes instead of the raw download hash, writes nothing on a mismatch and never rewrites the historical manifest. Verified: today's upstream (Darwin raw hash changed) regenerates every v1 file byte-identically. - requirements/release-lock.txt: exact package versions behind every record. - experiments/records/SUPERSEDED.json lists 8 superseded records with reasons and replacements; the report builder skips them explicitly. - validate_records.py checks naming, provenance fields, that recorded commits exist, that training/evaluation data hashes match the committed files, and that only superseded records are dirty. Tests run it and assert that rebuilding the report from records leaves docs unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
INT8 losses and stored bytes match the September record exactly; timings do not (KV speedup re-measures lower). Two records written with uncommitted doc edits are kept and listed as superseded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- README reorganized: purpose, components, five key results, reproduction (lock file, legacy v1 commands with --attention-backend mha), links, limitations. Every headline number is checked against records by tests/test_published_claims.py. - Corrected an overstated claim: the CPU training-step gain is 5-28% (5% at L=256), not 8-28%. - Legacy KV speedup now cites the recorded re-measurement (3.40x vs 3.58x). - Test-set status documented: Faraday evaluated once after all training, results published, not to guide further tuning; cleaning rules were designed while viewing its raw text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e paths validate_records.py checks that recorded commits exist, which a shallow checkout cannot satisfy. CI now installs requirements/release-lock.txt. compare_attention_quality.py records the validation path relative to the repository instead of an absolute local path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g items Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt file hash Checkpoint files embed elapsed train/eval seconds, so identical retraining gives identical tensors but a different file SHA-256 (reproduced: v1 seed 42 original vs fresh-clone retrain). minilm_lab.identity defines scheme minilm-model-identity/v1: SHA-256 over a versioned canonical stream of every MiniLMConfig field and the tokenizer (canonical JSON), then each state-dict tensor in name order (JSON header with name, dtype, shape, nbytes, followed by raw little-endian row-major bytes in the tensor's own dtype). Runtime metadata, paths, timestamps, seeds and the attention backend are excluded. train_tiny.py stores model_identity in new checkpoints (legacy checkpoints load unchanged and are hashed from their tensors); checkpoint file SHA-256 fields are kept for artifact identity. Training records (schema v2) carry best/final model identities and explicit hyperparameters; evaluation and quality rows carry the identity of the checkpoint they scored. train_tiny.py exposes build_parser(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…luation provenance minilm_lab.records is shared by the validator and the report builder: - experiment identity = protocol, canonical backend (auto -> mha before any grouping), seed, digest of model config + tokenizer + hyperparameters, and digest of the training/validation data hashes; - explicit supersession first; remaining same-identity records are replicates only if per-step losses, best step and model identity agree (earliest is canonical, counted once); otherwise validation and reporting fail; - each evaluation/quality/legacy row must resolve to exactly one current training record by checkpoint SHA-256, with matching seed, canonical backend, protocol, selected step, corpus hash and model identity where both exist; splits must match files and the protocol manifest; training commit and manifest references must exist and match; - legacy (record_version 1) hyperparameters are parsed from the recorded command with train_tiny defaults; nothing missing is fabricated. The report builder joins by checkpoint identity instead of seed/backend and refuses to build from records with validation problems. The validator takes --records-dir and no longer hard-codes the records path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…es from them Each mode now stores raw.step_seconds (one list of per-step seconds per measured repeat) plus timer, synchronization strategy and units. Summaries are computed by records.kv_summary_from_raw, which the validator reuses to check every summary against the raw samples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CPU fp32, MPS fp32 and MPS fp16 re-measured with the same grid and settings at 57be26d. The three historical KV records (summaries only) stay on disk unchanged and are listed in SUPERSEDED.json; their samples cannot be re-derived. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eckpoints A legacy 'auto' record and its canonical 'mha' re-record can reference the same checkpoint file. A checkpoint may now match several current training records only if they share one experiment identity (canonical = earliest); matches spanning identities still fail. Tests: known-answer canonical byte stream, digest stability/sensitivity (weights, config, tokenizer, precision), runtime metadata exclusion, no state mutation, legacy checkpoints, training reruns; auto+mha seed-42 counted once, conflicts fail validation and reporting, explicit supersession resolves them, historical rows reproduced; zeroed checkpoint hash, wrong training run, changed corpus/split/protocol, contradictory duplicates, invalid commit and manifest references, model identity mismatch; KV summaries vs raw samples, historical KV records marked; validator exit codes through pipelines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…for CUDA benchmark_training_step.py fell through to the host peak-RSS branch for CUDA. provenance.measure_peak_memory now dispatches explicitly (CUDA: max_memory_allocated delta; MPS: per-op live allocator peak; CPU: host peak-RSS delta; other devices rejected) and both benchmarks use it. CPU/MPS probes reproduce the recorded values (MPS exactly). The CUDA branch is tested with stubbed counters only; no CUDA hardware is available. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…them attest_model_identity.py verifies that each current training run's checkpoint files are byte-identical to the recorded SHA-256, then records their model identity in a new append-only record (and optional regeneration checks). The validator checks attestations against training records and uses attested identities in evaluation-provenance joins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All current training runs' best and final checkpoints were verified byte-identical to their recorded SHA-256 before hashing. Regeneration check: the fresh-clone retrain of v1 seed 42 has a different file SHA-256 but the same model identity. A first attestation written with an uncommitted report is kept and superseded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… script The quality oracle runs attention, matmuls and loss in float64, but RMSNorm variance and RoPE angles in float32; generated tables and future records say so. verify_release.sh runs lint, tests, record validation, report regeneration and the documentation-drift check under set -euo pipefail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- docs/RECORD_SCHEMA.md: record versions, artifact vs model identity, experiment identity, replicate/conflict/supersession rules, evaluation provenance, legacy rules, raw-sample requirements, memory methods. - EXPERIMENT_RESULTS.md regenerated from the replacement KV records (published KV numbers updated: MPS short-prompt batch-1 0.88x, CPU 2.2-15x at batch 1 and 3.7-23x at batch 8, up to 63x vs uncached MHA, MPS past-window 0.94-1.00x); oracle claim qualified; provenance method noted. - README: model identity vs checkpoint file hash (a regenerated checkpoint does not reproduce its file hash); oracle qualified; 0.88x. - research_report.md and implementation_audit.md: stale INT8 and corpus regeneration statements annotated; independent audit resolution recorded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ntities Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…umber changes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix 1 (artifact/attestation integrity): - resolve_identity gathers every source for a checkpoint's model identity: declared, attested (matched by checkpoint SHA-256), pinned audited baseline, and the artifact itself (bytes must equal the recorded SHA-256, identity recomputed from tensors). Any disagreement fails. - evidence levels: artifact_verified, trusted_baseline_verified, metadata_consistent_only, unverified; only the first two count as verification. validate_records.py --strict (release gate) and --require-artifacts; checkpoint bytes that do not match fail closed. - experiments/baselines/model_identity_baseline.json built by build_identity_baseline.py from the 24 checkpoint files (never from attestations; deterministic, --check regenerates byte-for-byte). Its SHA-256 is pinned in records.py, so changing it is a reviewable code diff. Fix 2 (duplicate integrity): training records are grouped by protocol, canonical backend, seed and dataset before comparison; configuration, outcomes and resolved checkpoint identity must agree or the slot conflicts and nothing is selected. Supersession entries must be the same kind and experiment (seed/backend or device/dtype), acyclic, ending at a current record. Fix 3 (KV derived ratios): speedups are derived as median uncached total time / median cached total time from raw samples (summary medians for the documented legacy rule); stored ratios must match (rel 1e-12); timings must be finite and positive (no ZeroDivisionError); the legacy single-prompt KV benchmark is re-derived from its samples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kv tables and figures use records.kv_speedups instead of stored ratio fields; the builder validates with strict=True. Published-claim tests use derived ratios and check the September KV speedup against its samples. The model-identity mismatch test now expects the stronger source-disagreement rejection that fires first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aseline options 36 tests: forged, contradictory and substituted fingerprints; tampered checkpoint bytes (arbitrary bytes everywhere, flipped byte where the original artifacts exist); baseline regeneration and tampering; behavior and CLI exit codes without artifacts and without a baseline; correct and contradictory modern records on a synthetic artifact; legitimate and conflicting auto/mha duplicates (fingerprint, checkpoint, losses, configuration), valid and invalid supersession (missing, wrong seed, cycle, other kind); stored KV ratios of 999/NaN/inf/0/-2, altered medians and raw samples, zero/negative/NaN timings, valid data, historical summary-only and legacy single-prompt records. validate_records.py gains --artifact-root and --no-baseline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s, derived KV ratios RECORD_SCHEMA.md documents the four model-identity sources and evidence levels, what the pinned audited baseline does and does not establish (an integrity anchor in reviewed code, not a signature), strict and require-artifacts modes, slot-based duplicate rules, supersession validity, and the exact definition and tolerance of derived KV-cache ratios including the legacy summary-only rule. README and EXPERIMENT_RESULTS updated to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dence and trust Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- README restructured: description, architecture, key results (one table, every number recomputed from records by tests/test_published_claims.py), implementation, quick start, reproducibility, documentation, limitations. - Four diagrams drawn from the implementation (architecture; attention backend dispatch; KV-cache inference; reproducibility trust boundaries), each as transparent black (-light) and white (-dark) SVG. The architecture diagram is embedded with a GitHub <picture> element; the others are linked. - docs/figures/src/make_readme_diagrams.py regenerates all eight SVGs. - Corrects the generation description: top-k/top-p apply to uncached sampling; the cached generator supports temperature sampling and greedy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ut unchanged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ication GitHub's hosted macos-14-arm64 runner reports torch.backends.mps.is_available() = True but fails the first MPS allocation (run 39751565753: four tests, 'MPS backend out of memory (MPS allocated: 0 bytes ... max allowed: 7.93 GiB)' while allocating 2 KiB-4 MiB). The MPS tests were gated on is_available() only. - tests/hardware.py probes MPS by allocating, computing and synchronizing; only a RuntimeError marks it unusable (wrong results still fail the tests). 'python tests/hardware.py' prints a hardware report. - tests/conftest.py: mps_device fixture skips with the probe's reason, or fails when MINILM_REQUIRE_MPS=1 (complete MPS suite on Apple Silicon). - test_backend_specific_tolerances is parametrized by device so the CPU rows always run and the MPS rows are reported separately; tolerances unchanged. - CI prints the hardware report and runs scripts/verify_release.sh (lint, tests with skip reasons, strict record validation, report drift). No MPS memory settings are changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cs/readme-diagrams Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The <picture> element used prefers-color-scheme, which follows the operating system, so the wrong SVG showed whenever the GitHub theme and the system theme differed. GitHub's #gh-light-mode-only / #gh-dark-mode-only image fragments are kept by its markdown renderer and hidden by github.com CSS keyed on :root[data-color-mode] (falling back to prefers-color-scheme only for 'sync with system'). Verified with GitHub-rendered HTML and live CSS: the correct single diagram shows for GitHub light/dark/auto x OS light/dark. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs: redesign README and add monochrome technical diagrams
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Release the audited research stack: selectable SDPA (default), legacy MHA and explicit reference attention; causal/cache/dropout correctness checks; matched attention, training-step and KV-cache experiments; and the cleaned science-v2 corpus with a held-out Faraday test book. Historical v1 records remain unchanged.
Evidence and reproducibility
Validation
Independent acceptance at 6748924: release verification passed in a checkpoint-free clone (184 tests passed, 2 artifact-dependent tests skipped), lint passed, strict validation passed for all 41 records, and report regeneration produced no drift. All 36 release-blocker tests passed with the original artifacts available, including both skipped tests. The published v1 losses and model tensors were independently reproduced during the audit.
Limitations
One Apple M4 laptop; three training seeds; small models trained for 1,000 steps; one held-out test book. CUDA and MPS training determinism remain unverified. v1/v2 perplexities are not directly comparable. Without distributed checkpoints, historical identity verification trusts the reviewed pinned baseline, not a cryptographic signature. Historical superseded KV records lack raw samples.
Documentation redesign is a separate stacked PR. Do not merge automatically. Preserve research history when merging if the stacked documentation PR will be retargeted; after a squash merge, recreate documentation-only commits from updated main instead.