Repository navigation
Conversation
…plan Assisted-by: Pi:gpt-6-astra
Standalone CMake experiment for Task 0 of the rank-local OpenMP plan. One process and one OpenMP team compare three ownership/index designs over the current packed OperatorIndex rows: per-thread disjoint stores, one shared store with frozen parallel lookup and caller-owned insertion, and one shared packed backing store indexed by boost::concurrent_flat_map with per-query or bulk lookup. It reuses the header-only storage/hash code without changes and does not link or import monoprop. Release-active self-tests cover exact IDs, sharded owner/local IDs, collisions, spilled rows, tails, duplicate-miss rejection, runtime-limited teams, worker participation and joined exceptions. A BOOST_UNORDERED_ENABLE_STATS diagnostic build adds Boost's own container statistics to the CSV output. Assisted-by: Pi:claude-opus-5-5
…cision Record the owner's decision after Task 0: the evidence was inconclusive for the ownership axis, so one store per rank remains the working hypothesis. Summarize the mini-app, in-library counter, one-core profile and Boost-index trial results, and record the waiver of Boost 1.85 compile checks. Move the ownership decision to Task 11's full-library comparison against the partition runtime, add a diagnostic go/no-go after Task 6 before the Task 10 API cutover, and make Task 12 conditional on confirming the single store. Replace the bounded Boost trial with a shared-serial versus shared-concurrent index A/B, and add an invariant that keeps OperatorIndex swappable. Assisted-by: Pi:claude-opus-5-5
Task 1 measurement layer for the rank-local OpenMP plan, applied identically to both arms. - benches/conftest.py records operation and outer memory-window exactness independently (all-rank reductions), imports mpi4py only for MPI-enabled extensions, records has_mpi and the extension hash, and adds --runtime-shape/--build-mode preflight while keeping the legacy multi-rank guard. - monoprop_bench_tools.preflight holds the shape/build-mode checks shared with the driver; the report and BMF renderers label or gate each memory metric by its own window's flag. - tools/benchmark-rank-local-openmp.py has exactly three modes: observe (historical pytest nodes in-process, the new bounded driver::pare_functional_construct measurement, and a separate whole-construction worker), validate (untimed numerical products with gzip JSON Lines term files), and compare (frozen-inventory joins, placement checks and the five median gates). - Docs describe the options, exactness labels, the VmHWM shortfall observed on many-core hosts, and the driver. Assisted-by: Pi:claude-opus-5-5
Record the owner's decisions after the unscored baseline pilot and freeze the campaign. - tools/rank-local-openmp-campaign.json (250 cells) and tools/rank-local-openmp-workloads.json are the owner-approved inventory: tiny/reference (+pared) across MPI 1x1, 1x96, 2x48 and MPI-off 1x96, reference routing cells at 3x32/4x24, and large (+pared) at MPI 1x96, 2x48 and MPI-off 1x96. - Term files are gzip-compressed JSON Lines instead of adding storage (4.4-16x smaller measured). - Hubbard graph cells use 2 Trotter steps within the size guard; the 29-step graph did not fit. - The measured VmHWM shortfall is accepted as documented measurement noise; gates stay <=1.00. - prek exempts the frozen workloads file, which carries full parameter vectors by design, from the large-file check. Assisted-by: Pi:claude-opus-5-5
Name the host directories that hold the uncommitted Task 0 results and the Task 1 handoff report, formal baseline evidence, baseline index, collection scripts and pilot, so Task 11 can locate the baseline half of its comparison. The artifacts stay uncommitted and will be removed before merging. Assisted-by: Pi:claude-opus-5-5
…ire OpenMP
Task 2 of the rank-local OpenMP plan: the smallest worksharing helper and the mandatory OpenMP
dependency across build routes. No engine code uses the helper yet.
- detail/parallel/Options.h holds the per-region thread budget and does not include omp.h.
detail/parallel/Workshare.h::for_blocks runs each logical block once:
- it rejects non-positive budgets and counts above PTRDIFF_MAX before any work;
- it runs serially in ascending order for a budget or count of one, or inside any OpenMP region;
- otherwise it opens one static-schedule region requesting min(threads, count) workers;
- worker exceptions stay in per-worker slots and are rethrown on the caller after the join;
- it changes no OpenMP runtime setting.
- CMake requires OpenMP CXX in every configuration, links OpenMP::OpenMP_CXX PUBLIC on both
monoprop-objs and the shared monoprop target (which is built from TARGET_OBJECTS), and the
package config calls find_dependency(OpenMP).
- openmp_workshare_tests.cpp covers:
- exactly-once visits for counts 0/1/19 and budgets 1/2/3, and more workers than blocks;
- serial and nested fallbacks, and budget and bound validation;
- unchanged runtime settings and joined worker exceptions;
- observed multi-worker participation, skipped when the runtime cannot give two workers.
- The shared link-export probe, which just test-find-package also builds, instantiates for_blocks,
requires the imported target's OpenMP flags and drives the Pauli basis as well as Majorana.
- Dependency routes:
- Homebrew libomp with OpenMP_ROOT, in the setup action, the macOS wheel config and the docs;
- a matching libomp-<ver>-dev for Linux Clang lanes, and libomp-dev in the devcontainer;
- llvmPackages.openmp for Clang-based Nix builds, monoprop-mpi in the flake checks, and a
dev-shell consumer step in nix.yml;
- clang-tidy's setup now receives libomp.
Hwloc stays until its removal task.
Assisted-by: Pi:claude-opus-5-5
Record what Task 2 delivered and what later tasks inherit: - the helper's implemented contract: exception types, serial and nested fallback conditions, and per-worker stop-on-first-failure; - RED/GREEN and participation evidence under libgomp and libomp, and the load-bearing find_dependency; - the observation that GNU ld's --as-needed drops libgomp while the engine has no OpenMP code, so wheel runtime bundling is still unverified; - which dependency routes were verified on Linux x86_64 and which await CI or a suitable host; - a build-route note: rebuild through uv sync, since the adopted tree's cached ninja is gone, and keep M and Clang trees in their own venvs; - where the uncommitted Task 2 artifacts live. Assisted-by: Pi:claude-opus-5-5
…lure boundaries
Task 3 of the rank-local OpenMP plan: configuration plumbing and safety boundaries. Kernels stay
serial.
- detail/parallel/ThreadBudget.{h,cpp} (exported, installed):
- resolve_thread_budget is a pure parser. Unset monoprop_NUM_THREADS takes a positive
omp_get_max_threads(); a set value must be ASCII digits in [1, INT_MAX], with leading zeros
accepted. Empty, signed, padded, list, zero and overflowing values are rejected. There is no
thread cap.
- capture_thread_budget reads the environment once per call.
- MonomialPropagator gains a private parallel::Options. It is serial except on the one-store
prototype (explicit partitions=1 on an ordinary communicator), which captures the budget once at
construction. Copies and retained functionals keep it; Shm/Hybrid children and facades stay
serial. An invalid budget is a PropagatorConfigError on one rank and aborts on several.
- The trailing Options reaches every path: graph-only build, stored-mask callbacks, immediate fused
application, replay, retained functionals (EvalRequest::parallel), prepare_evolved_operator, the
derivative (after record) and partial contraction. build_cos_callbacks closures capture it and the
callback types are unchanged.
- Failure boundaries:
- mpi::operation_failed reports the error and calls MPI_Abort on a multi-rank ordinary
communicator, and rethrows the original exception otherwise.
- Guards sit inside the replay Ticket and construction PendingAlltoallv lifetimes, and around
local work before each allreduce.
- Invalid owners: each operation separates validation from mutation. A failure after mutation
starts invalidates the object. Later operations, copy/clone and retained functionals then throw
InvalidPropagatorError; earlier copies keep working. Interleaved index validation now runs first.
- MPI: the requested and required level stay MPI_THREAD_SERIALIZED, and the level actually
provided is checked for host- and library-initialized MPI. require_initializing_thread
(MPI_Is_thread_main) makes prototype entries from another thread print a diagnostic and abort
locally, with no MPI call.
- Tests:
- openmp_runtime_tests.cpp, plus fresh-process openmp_env_* CTest entries;
- tests/test_openmp_config.py and a routes_pairwise cache case;
- mpi_failure_driver.cpp, outside the unit runner: eight two-rank mpi_failure_* scenarios with a
30 s timeout, covering worker throw, before exchange, both request owners, insufficient level
and wrong thread;
- the link-export probe now covers the prototype chain.
- Docs cover the prototype, the failed-operation rules, the ABI rebuild, OpenMP runtime linkage and
the new test entries.
Assisted-by: Pi:claude-opus-5-5
monoprop now links the OpenMP runtime. With OMP_PLACES set and binding on, libgomp binds the importing thread to the first place when the library loads, so the observe preflight read one CPU instead of the allocation. It then rejected the documented candidate launch on GCC builds. The allocation is now read before the import. For the baseline binary, which loads no OpenMP runtime, both orders read the same mask, so collected evidence is unaffected. Assisted-by: Pi:claude-opus-5-5
Record what Task 3 delivered and what later tasks inherit: - the implemented budget, invalidation, failure-boundary and MPI thread contracts; - RED/GREEN and mutation evidence, the R/M/Clang results and the provided MPI thread levels; - the ABI change, and the OpenMP runtime that the library now links; - the harness allocation fix, and the matching edit still due in the baseline worktrees; - where the uncommitted Task 3 artifacts live. Assisted-by: Pi:claude-opus-5-5
…check The allocation fix is now applied to the tool copy in both baseline worktrees, so both arms run the same measurement code (sha256 2207529f…). Task 11 gains a checklist item to verify that hash before measuring, because those copies are uncommitted and could drift. Assisted-by: Pi:claude-opus-5-5
Task 4 of the rank-local OpenMP plan. Two kernels now use the object's thread budget. Structure (scan,
lookup, insertion, graph packing) stays serial.
- scale_cos_lazy runs one logical range per 1024 fold words. Row parity is fetched on the caller for
each call. Each worker keeps its scratch on its own stack, never the caller's thread_local buffer.
Tail masking stays global.
- scale_cos_mask runs one range per 1024 stored mask blocks.
- apply_fused_contract keeps its phases, each joined before the next: gather insert snapshots into
the existing v_tgt field, run the optional cosine phase, then apply records in ranges of 1024.
- Every record range calls one out-of-line apply_fused_record_range, so the compiler's FMA
contraction of cos*c + sin*phi*v is the same at every budget. Serial and threaded results are
bitwise equal, and match the previous release bitwise on the paths checked.
- Debug builds assert that every coefficient slot has exactly one writing record.
- Fewer than two ranges stays serial. The thresholds are fixed until the evidence-driven tuning task.
- The kernels take a defaulted, test-only range observer; production passes an empty one.
Tests:
- openmp_kernel_tests.cpp covers exact agreement across budgets, small work staying serial, worker
exceptions joined before rethrow, and participation observed inside each kernel's own ranges.
- Synthetic multi-block lazy folds (tails, cold parity, Majorana/Pauli, empty columns, index growth)
are checked against the materialised-fold oracle.
- Records from the real build path have one add-owner per slot, in both pictures and with cross-rank
halves under MPI. Threaded prototypes match serial ones bitwise.
- Fused and two-pass application still agree with graph replay at the existing tolerances.
- openmp_env_kernels_* rerun the kernel cases under OMP_THREAD_LIMIT=2 and OMP_DYNAMIC=TRUE.
- The failure driver gains active-ticket cosine-worker: a kernel worker throws while a replay
Ticket is posted, and the job must abort rather than hang.
Docs: the one-store prototype section lists which kernels are threaded. building.mdx explains
that TSan evidence for the OpenMP kernels needs Clang + libomp + Archer, since GCC's libgomp reports
false races.
Assisted-by: Pi:claude-opus-5-5
…aluation Record what Task 4 delivered and what later tasks inherit: - the kernel range, scratch, phase and FMA-determinism contracts, and the test-only observer; - RED/GREEN, mutation, stress and sanitizer evidence, with the qualified TSan runtime; - the owner-approved replay diagnostics: one-store speedups and losses, kernel-only scaling, and the comparison with the baseline partition runtime; - where the uncommitted Task 4 artifacts live. Task 9 gains a checklist item to evaluate explicit std::fma against the current contraction-dependent arithmetic, before any numerical-policy change. Assisted-by: Pi:claude-opus-5-5
Task 5 of the rank-local OpenMP plan. The per-gate bitmap scan that finds anticommuting terms and emits their partner queries now uses the object's thread budget. Lookup, insertion, publication and graph packing stay serial. - fused_find_and_collect prepares the inverted index and row parity on the caller, then splits the words into min(threads, ceil(words/1024)) contiguous runs of whole fold blocks (scan_ranges, scan_range_words). Range IDs depend on the word count and budget only, never on the actual team. - Each range runs fused_find_and_collect_range into its own FusedScanResult. It never calls a lazy accessor, keeps its scratch private or thread_local on the executing thread, masks only the global tail word and, in the fused sweep, scales only its own sources, once, after capturing them. - merge_scan_ranges joins the pieces in range order, stream by stream: checked totals, one reservation, the first buffer moved where it can hold the stream, drained buffers released. Self stages merge through SelfQueryStage::push. Output is identical to the serial scan at every budget. - CutoffEvaluator::parallel_safe() admits only the typed length and support cutoffs. Opaque predicates, basis-change closures and scans below two ranges keep the serial scan on the caller. - The test-only range observer moves to layer_build/Common.h, gains KernelRange::scan, and build_layer forwards a defaulted one; production passes none. Tests: - ScanTestSupport.h compares threaded scans with budget one byte for byte across windows (one rank, linear zero/non-zero shifts with non-zero bases, dense splitmix at three and four ranks), pivots, Majorana and Pauli J(G), caps, cutoffs, atol boundaries, capture, the fused sweep and cold caches. - The merge is tested on hand-built pieces: variable-length wire records, both self stages, an empty piece, buffer moves and released capacity. - Full construction at budgets 1-4 gives identical rows, row IDs, graph layers and coefficients. - Opaque and basis-change cutoffs stay serial with identical results. - A scan worker failure joins before rethrow and invalidates the propagator; the failure driver gains before-exchange scan-worker, which must abort both ranks rather than hang. - openmp_env_kernels_* rerun the scan cases under OMP_THREAD_LIMIT=2 and OMP_DYNAMIC=TRUE. Docs: the one-store prototype section lists the threaded scan and its serial cutoff fallback. Assisted-by: Pi:claude-opus-5-5
Record what Task 5 delivered and what later tasks inherit: - the scan range, merge, cutoff-classification and observer contracts; - RED/GREEN, mutation, stress, sanitizer and cross-revision evidence; - the owner-approved construction diagnostics: traversal and end-to-end speedups, the losses at high budgets, the growing serial merge, and the scratch and peak-memory accounting; - where the uncommitted Task 5 artifacts live. Assisted-by: Pi:claude-opus-5-5
Task 6 of the rank-local OpenMP plan, committed as the record of the one-store experiment (the owner
decided afterwards to re-design and keep the sharding). Only the prototype selected by explicit
partitions=1 on an ordinary communicator uses these threaded paths.
- probe_frozen_positions: 256-query blocks, each one find_batch_positions call over absolute offsets
into the shared position buffer with sliced counts, outputs and hashes; span contracts checked before
any block runs; the store stays frozen until every worker joins.
- probe_incoming_queries validates every sender stream on the caller with QueryWire::checked_extent
(ternary phase, k and gap-width bounds, canonical escaped counts, whole records including the fused
value word, final offset at the buffer end), checks prefixes and allocates every array once. Workers
decode disjoint blocks; read_positions rejects size_t positions >= 2N before narrowing. Misses get
IDs serially in sender-window/query order after a row-ID ceiling check; miss_g is vector<size_t>.
- resolve_incoming scatters on workers only for sinks declaring parallel_resolve (GraphSink,
Heisenberg ContractSink); Schrodinger scoring and generic sinks stay serial. It runs inside
guard_distributed, as the peers may be in the response round.
- The self lookup probes windows of at most 4096 queries; filtering, gathering and emission stay
serial and nothing is inserted between windows. Publication order and the single-writer
grow/set_positions/bulk_insert_hashed/reindex sequence are unchanged; bulk_insert_hashed gains an
ignored trailing parallel::Options seam.
- The test-only observer gains KernelRange::{decode,incoming_probe,self_probe,scatter}.
Tests:
- Serial/threaded bitwise equality of probes, decoded positions, hashes, predicted and published IDs,
row order, responses, sink records, marks and rebuilt inverted indices, over empty/all-hit/all-miss/
mixed inputs, 255/256/257-query boundaries, empty senders, non-zero window bases, narrow and wide
positions, spilled rows, escaped counts, Plain/Fused codecs and 4096-query self windows.
- Malformed streams (including the noncanonical escape 0x8037e) rejected before decode; invalid
positions rejected in decode before probing; serial scatter for uncertified sinks; duplicate-key
bulk insertion pinned; participation observed inside each new region; four new two-rank
resolve-worker failure scenarios abort promptly.
Assisted-by: Pi:claude-opus-5-5
… sharding Record the Task 6 outcome and the post-Task-6 architecture go/no-go: - the Task 6 contracts, RED/GREEN, mutation, stress, sanitizer and cross-revision evidence; - the phase diagnostic, including the libgomp high-budget regression from small self-probe teams; - the architecture diagnostic: parity at L1, 0/16 reference cells at runtime <= 1.00 at full machine (2-62x), and no credible bounded route to parity; - the owner's decision: the one-store design is abandoned before the Task 10 cutover and the architecture is re-designed keeping the sharding. Tasks 7-13 are marked superseded. Assisted-by: Pi:claude-opus-5-5
Document owner-local sharding with operation-scoped OpenMP teams and primary-thread MPI. Supersede the one-store architecture while preserving its history and frozen evidence. Keep arithmetic ordering, shard combination, reductions and FMA policy open for separately approved post-landing optimizations, individually or together. Documentation checks pass. Full prek run reports 49 pre-existing notebook Ruff findings, waived by the owner for this documentation-only push. Assisted-by: Pi:gpt-6-astra
Task S0 of the sharded OpenMP plan: detail/sharded/Team.h, a header-only operation-scoped team and
checkpoint-stable failure protocol. Nothing in the propagator uses it yet.
- run_team opens one omp parallel num_threads(T) region, calls a noexcept orchestration body once per
worker with its fixed OpenMP index, and returns the lowest failing worker's original exception after
the join instead of rethrowing it, so future callers can handle failures while MPI requests and
buffers are still alive.
- phase catches everything into the worker's own preallocated slot and always reaches the checkpoint:
barrier, masked primary decision, barrier. A decision cannot be overwritten until every worker has
passed the next leading barrier, so a fast worker's later failure cannot change an earlier branch.
- The helper assumes exactly T workers: no team-size check, fallback, runtime-setting change, MPI or
persistent state. Options.h now documents that reduced-team correctness belongs to for_blocks.
- sharded_team_* cases run in fresh sharded_team_env_t{1,2,4} and _t4_passive launches; the installed
consumer probe instantiates Team.h through monoprop::monoprop.
Verified with GCC/libgomp and Clang 18/libomp (MPI off), ASan/UBSan, MPI on, Clang/libomp/Archer TSan
and 13 Team.h mutants. prek passes apart from the 49 pre-existing notebook Ruff findings the owner
waived for this plan.
Assisted-by: Pi:claude-opus-5-5
Mark Task S0 of the sharded OpenMP plan done and accepted by the owner, with a summary of its RED, GREEN, mutation, sanitizer and installed-consumer evidence and the platforms still pending. Record the owner's standing waiver for the whole execution of the plan: the pre-existing Ruff findings in the chemistry, fermi_hubbard and kicked_ising notebooks (49 at S0) are ignored. The waiver covers no other file, hook or new finding, and the notebooks are not to be edited to silence it. Assisted-by: Pi:claude-opus-5-5
Task S1 of the sharded OpenMP plan: detail/sharded/State.h, internal shard state that can be seeded,
copied, counted and destroyed without a worker pool or child propagators. The public runtime is
unchanged.
- ShardState holds the existing MPOperator, MPGraph and MatchedEpochSet. make_shards allocates only
the pointer vector on the caller, initializes each shard on its owner inside an S0 protected phase,
and rethrows the lowest failing owner's exception after the join, once published siblings have
been destroyed.
- Seeding is split into caller-side preparation (validate_initial_operator, paired_basis_bounds,
packed_inline_width, OperatorSeed) and the owner-local seed_operator. That function preserves input
and paired-basis order, flat owner rank * T + shard, per-owner reserve, pending entries and the
Heisenberg-sparse versus Schrodinger-dense cache warm-up. The legacy constructor now seeds through
it, so behaviour is unchanged.
- copy_shards deep-copies each shard on its owner and keeps graph layer cores shared. Per-shard counts
and accounting need no merged operator.
- MPOperator() is no longer noexcept: its store initializer allocates, and an injected allocation
failure previously terminated the process.
- sharded_state_* cases run in fresh sharded_state_env_t{1,2,4} launches, against the legacy partition
oracle and an independent reference. A test-only allocation probe fails every allocation site of the
seed, copy and legacy-constructor paths in turn. The installed consumer compiles State.h on its own.
Verified with GCC/libgomp and Clang 18/libomp (MPI off), ASan/UBSan, MPI on, Clang/libomp/Archer TSan
and 11 mutants. prek passes apart from the 49 notebook Ruff findings already waived for this plan.
Assisted-by: Pi:claude-opus-5-5
Mark Task S1 implemented, with owner acceptance still pending. Summarize its interfaces, the MPOperator noexcept fix, and its RED, GREEN, mutation, sanitizer and installed-consumer evidence. List the platforms still pending. S2 still needs its own authorization. Assisted-by: Pi:claude-opus-5-5
Task S2 of the sharded OpenMP plan: detail/sharded/Construction.h, an internal P=1, T-shard engine for
multi-gate graph construction and graph-free propagation. The public runtime is unchanged.
- build_graph and propagate run the whole gate loop in one operation-scoped team. Worker t mutates
only shard t. Other shards' published query and answer buffers are read as immutable spans after
the checkpoint that published them, in ascending flat-source order. There are no communicators, no
custom barriers and no nested worksharing, and within-shard kernels run serially.
- Per gate: traversal and leader prepare, leader resolution, leader consume and follower prepare,
follower resolution, then follower consume and finish/apply. The row order, follower filtering,
frozen probes, deferred same-shard inserts, scaled_count and every arithmetic path are preserved.
- Opaque cutoffs and basis-change closures traverse every shard on the primary, in an exclusive phase
that every worker enters.
- Phase failures are returned after the join as ConstructionOutcome{error, mutation_started}, so the
S4 caller can invalidate the owner; argument errors throw before the team.
- LayerBuildEngine gains prepare_exchange, resolve_published and consume_published. The traversal is
extracted as scan_gate. The legacy transport is now the run_exchange(comm, plan, ...) adapter over
the same phases, with unchanged request lifetimes and guards. The resolver and the sinks take
read-only views (mpi::views_of) instead of owning buffers.
- QueryWire::Writer::put/flush are no longer noexcept: they allocate, and an injected allocation
failure while encoding a query previously terminated the process.
- sharded_construction_* cases run in fresh sharded_construction_env_t{1,2,4} launches. They check
bit-identical shards and streams against the legacy partitions at (1, T). They also check
independent insertion-order and coefficient-map references, the frozen exact energy, cross-T
retained maps, participation, and failures in every phase including allocation sweeps. The
installed consumer compiles Construction.h on its own.
Verified with GCC/libgomp and Clang 18/libomp (MPI off), ASan/UBSan, MPI on (legacy compatibility),
Clang/libomp/Archer TSan and 11 mutants. prek passes apart from the 49 notebook Ruff findings already
waived for this plan.
Assisted-by: Pi:claude-opus-5-5
Mark Task S2 implemented, with owner acceptance still pending. Summarize its interfaces, the phase sequence, the QueryWire noexcept fix, and its RED, GREEN, mutation, sanitizer and installed-consumer evidence. List what S2 leaves to S3–S5 and the platforms still pending. S1's acceptance record is unchanged. S3 still needs its own authorization. Assisted-by: Pi:claude-opus-5-5
Task S3 of the sharded OpenMP plan: detail/sharded/Evaluation.h, P=1 T-shard replay, energy,
reverse-mode gradients, partial contraction and retained/pared functionals, plus coefficient-informed
graph construction. The public runtime is unchanged.
- ev_sharded and ev_and_grad_sharded run the whole forward and reverse replay in one team. The
communicator is touched only on the caller, and a phase failure goes to mpi::operation_failed
before the result reduction. evaluate_shards is the S4 seam: per-shard terms and gradients plus
the error, returned after the join.
- Each replay step publishes pre-cosine endpoint snapshots into a double buffer, so partners never
read live coefficients and each step costs one checkpoint. Self pairs, records, predivide/restore
order and repeated-parameter accumulation are unchanged. Shard terms and gradient components are
folded in ascending shard order, and the identity is added once.
- The kernels move to the library-internal detail/evolution/LayerReplay.h, shared with the rewired
legacy Evolution.cpp and MPFunctions.cpp. The legacy derivative stays inline so its contraction is
unchanged; public results are bit-identical to the previous build (GCC, Clang, MPI), and the
sharded path matches legacy bitwise.
- Library-built callbacks are marked CosCallbacks::owner_parallel and run on their owners. Opaque
callbacks run on the primary in an exclusive phase.
- build_graph_informed replays the seed and each new layer inside the construction team.
pare_graph_owner takes the flat owner instead of a communicator rank. prepare_retained snapshots,
pares and builds callbacks per shard.
- sharded_evaluation_* cases run in fresh sharded_evaluation_env_t{1,2,4} launches. They check
bitwise equality with the legacy partitions, independent map, closed-form and finite-difference
references (including the deep-circuit gradient cases), layouts, lifetimes, participation and
failures. The installed consumer calls both evaluators.
- Also reformats three S2 files that were not clang-format clean (whitespace only).
Verified with GCC/libgomp and Clang 18/libomp (MPI off), ASan/UBSan, MPI on (legacy compatibility),
Clang/libomp/Archer TSan and 14 mutants. prek passes apart from the 49 notebook Ruff findings already
waived for this plan.
Assisted-by: Pi:claude-opus-5-5
Mark Task S3 implemented, with owner acceptance still pending. Summarize its interfaces, the double-buffered snapshot protocol, the contraction-fidelity fix, and its RED, GREEN, mutation, sanitizer and installed-consumer evidence. List what S3 leaves to S4–S5 and the platforms still pending. S4 still needs its own authorization. Assisted-by: Pi:claude-opus-5-5
… protocol Update the sharded OpenMP plan with S8's optimization rounds (steps 0-2, rounds 3 and 4): what was kept, what was dropped and why, the page-cache finding behind the MPI-off code-page excess and the MPI memory drift, the owner's campaign protocol (page-cache eviction and one warm-up per arm before each batch, interleaved arms, memory against fresh baseline samples), the re-measured gates, the LTO measurement, what stays open, and the off-host move of S7's raw term files. The final parity campaign still awaits authorization. Assisted-by: Pi:claude-opus-5-5
|
Important Review skippedToo many files! This PR contains 180 files, which is 30 over the limit of 150. To get a review, reduce the PR to 150 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to Team to raise the limit. ⚙️ Run configuration
📒 Files selected for processing: 180
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
github-actions
Bot
added
documentation
Record the outcome of S8's final campaign on the final binaries (sources of d359c15): 145 of 250 cells pass all five gates (S7: 102), so strict parity is not demonstrated. Runtime is judged under option B and memory against fresh baseline samples only, with the page-cache protocol applied every 25 cells. Numerics agree in every cell. The remaining failures are the accepted MPI 1x1 runtime tail (41 cells), a few larger runtime misses (pare-construct-schr at T = 96 and 2x48, energy-hubb at 2x48, tiny propagate at 2x48), and memory excesses of median 0.56 MiB (ratio 1.003) against the fresh baseline. Assisted-by: Pi:claude-opus-5-5
Member
Author
|
🤖 AI text below 🤖 S8 final parity campaign: results145 of 250 cells pass all five gates (S7: 102), so strict parity is not demonstrated. Numerics agree with the baseline in all 250 cells. Of the 105 failing cells, 41 fail on the MPI 1x1 runtime tail, which the owner accepted before the campaign, and most of the remaining memory failures are below 1 MiB. Setup
Run
Results by geometry
Against S7, 62 cells newly pass and 19 newly fail. Most new failures are memory excesses of 0.1–1.5% against the fresh baseline; S7 judged memory against the archived samples, which ran higher. What still fails
The outcome is recorded in the sharded OpenMP plan (5ce60df). Next: discuss an optimization round on the largest remaining misses. |
added 3 commits
October 10, 2026 16:52
… at T > 1 Round 5 of S8's optimization, on the memory gates' remaining misses. Every change is bitwise neutral (differential against the cutover build: 27/27 MPI-off, 136/136 MPI). - Seed each shard in place on the heap. seed_operator() fills the operator where it lives and ShardState(bool) builds an empty one, so no MPOperator<N> (about 112 N bytes, its inverted index's columns inline) passes through a worker's frame. Worker stacks stay resident for the pool's life; at T = 96 they held 9 pages each instead of the baseline threads' 3, now 4. - Construction's physical rounds unmap their staging when a construction ends; the replay round keeps its staging. An outgrown staging block is unmapped before the larger one is mapped, so the two are never resident together. - Partner staging is no longer widened to a step's largest block: a block larger than the run is read in place from the partner's publication. The run size is unchanged. - Per-step publication and staging buffers grow by capacity (grow_to()), not from their current size, which reallocated thousands of times per evaluation at T = 96 and fragmented the owners' malloc arenas. Targeted re-measure against fresh baseline samples (3 pairs per cell): every measured tiny Pauli cell at T = 96 and the Hubbard gradient cells now pass their memory gates; Hubbard and Heisenberg energy at 1x96 still exceed the baseline by about 3.5 MiB. No runtime regression found. New tests: seeding stack use, construction staging release, staging sizing with in-place blocks, buffer growth by capacity. Assisted-by: Pi:claude-opus-5-5
Record round 5 in the sharded OpenMP plan: the owner's choice (the memory misses, options D and B), the diagnostics (worker stacks held by seeding's on-stack operators, Hubbard partner staging and incremental buffer growth, construction staging kept through evaluation at 2x48), the four kept bitwise-neutral changes of 4fdddbc and their verification, and the targeted re-measure: the measured tiny Pauli cells at T = 96 and the Hubbard gradient cells now pass their memory gates, energy at 1x96 still does not. The final campaign's verdict stands until a re-run on the round-5 binaries, which awaits the owner. Assisted-by: Pi:claude-opus-5-5
Bring main's 18 commits since 90d5717 into the branch: dependency and CI bumps, the 0.9.2 version, Pauli skip_validation, upstream nanobind 3.1.0 with per-interpreter Nix packages, read individual evolved coefficients (#337) and the N-independent emit and fold (#383). Ported onto the sharded runtime: - evolved_operator_coefficients() probes every shard's store against its contracted block; shards hold disjoint rows, so at most one finds a key (rank-local under MPI, as on main; the Heisenberg identity is the rank-level core term). Main's C++ suite runs at the launch's T and again at T = 2 and 4 (evolved_coefficients_env_t2/_t4) instead of comparing partition counts in one process. - The scan's positions emit runs in the range kernel when there is one flat owner (P = T = 1), main's single-rank condition; the prefetch-ahead loop binds the thread_local nz once. - scale_cos_lazy keeps its fold-block ranges, each now running for_each_fold_word() over its own words (the posting-driven fold where the columns are sparse enough); cos_count_lazy counts through the same fold. - Nix: main's per-interpreter checks and CI with the hwloc and pkg-config inputs still gone. Numerics are unchanged: the differential against the S8 cutover build is 27/27 MPI-off and 136/136 MPI rank documents. Assisted-by: Pi:claude-opus-5-5
|
Docs preview: https://pr-398.monoprop-docs.pages.dev |
added 7 commits
October 11, 2026 19:51
…small cosine calls The merge of main brought the posting-driven fold (#383) into every owner's per-call cosine scaling. Measured against round 5's binaries, that cost the sharded runtime at T > 1 and T = 1: - Each thread held a second 8 KiB zero block next to the word sweep's block, acquired before a fold knew whether any block took the posting path: about +1.4 MiB of thread heap per process at T = 96 during evaluation, enough to fail six memory gates round 5 passed. - for_each_fold_word()'s setup (cursors, a lower_bound per column, posting counts) dominated small folds: a T = 1 tiny energy call went from 12.4 to 17.8 µs. Now the word sweep and the posting fold share one block per thread (SharedFoldBlock): the posting fold re-zeroes it only after a word sweep dirtied it. Per-call cosine folds of one block take the direct word sweep (for_each_cos_fold_word()); longer folds keep main's kernel. Both are bitwise neutral (differential against the S8 cutover build: 27/27, 136/136). Re-measured against fresh baseline samples: five of the six cells pass again (the sixth at 1.0005), Hubbard build-graph keeps main's −61 to −72 MiB, and the tiny T = 1 energy call is back to 13.4 µs warm and 28.8 µs first (runtime ratio 1.09, campaign 1.16). Assisted-by: Pi:claude-opus-5-5
Record in the sharded OpenMP plan the merge of origin/main into the branch (94cbf92): main's scope, how evolved_operator_coefficients() and the N-independent emit and posting-driven fold were ported onto the shards, and the verification. Record step 6, the owner-approved measurement of the merged binaries against round 5: main's gains (Hubbard build-graph −60 to −72 MiB, Hubbard propagate 1x1 runtime 0.94), the per-thread fold block and small-fold setup it cost, the two bitwise-neutral fixes of 1a4573d, the TSan scenarios that time out on both sides of the change, and the re-measure. The campaign verdict stands until a re-run on the current binaries. Assisted-by: Pi:claude-opus-5-5
…penMP refactor Bring in main's two test commits since 67e7c4c: pytest-cases replaced with plain pytest (#404) and tests/cases.py renamed to tests/reference_data.py (#407). - tests/conftest.py: main's reference-data imports beside the branch's guarded mpi4py import, so an MPI-off build still never initializes MPI. - prek.toml: the name-tests-test exclusion keeps both reference-data modules and the branch's two probe scripts. - tests/sharded_openmp_probe.py loads its problems from tests.reference_data. No library source changes, so the binaries are unchanged. The full Python matrix passes with the merged tests (pytest 916 / 892 / 892 MPI-off, 1003 / 955 / 955 MPI, mpiexec -n 2 785). Assisted-by: Pi:claude-opus-5-5
…s kept for later Add a companion to the sharded OpenMP plan listing what is still open after S8, round 5, the merges of main and step 6: the remaining memory and runtime misses with their causes and possible remedies, the opportunities main's #383 opens (positions emit with several owners, sharded self-resolve prefetch, the posting fold on mid-size shards), and the items recorded for later (the post-landing numerical-performance policy, S5's flagged optimizations and their status, pending qualification). Add two items to evaluate: explicit OpenMP simd directives and OpenMP tasking, each with where it could help, the constraints it must respect (bitwise numerics, FUNNELED MPI, owner-computes memory, failure handling) and how to measure it. Assisted-by: Pi:claude-opus-5-5
`stack_bytes_touched` used Linux's `MADV_NOHUGEPAGE` and passed an `unsigned char *` residency vector to `mincore`, which macOS declares with `char *`, so every macOS job failed to compile `sharded_state_tests.cpp`. Skip the huge-page opt-out where the flag does not exist and type the vector per platform. Assisted-by: Pi:claude-opus-5-5
GCC's default `-ffp-contract=fast` fuses multiplies and adds across statements, so two code paths computing the same expression can round differently. With g++-14 this broke the bitwise cross-path checks of five C++ test cases even at one thread. `-ffp-contract=on` contracts within one expression only, as Clang does by default; with it the full suite passes on g++-14 and g++-15. Assisted-by: Pi:claude-opus-5-5
`_child_env` strips every launcher variable, including the justfile's Open MPI oversubscription settings, so the 3- and 4-rank probes failed with `prte-rmaps-base:alloc-error` on 2-slot runners. Restore both spellings in `_run_ranks`. Assisted-by: Pi:claude-opus-5-5
robertodr
marked this pull request as ready for review
October 16, 2026 11:57
robertodr
requested review from
adamglos92,
diagonal-hamiltonian,
fpietra and
ludmilaasb
as code owners
October 16, 2026 11:57
robertodr
commented
Oct 16, 2026
| set( | ||
| monoprop_CXX_FLAGS | ||
| "-Wall -Wno-unknown-pragmas -Wno-sign-compare -Woverloaded-virtual -Wwrite-strings -Wextra -Wconversion -Wnon-virtual-dtor -Wcast-align -Wunused-parameter -fdiagnostics-color=always -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer" | ||
| "-Wall -Wno-unknown-pragmas -Wno-sign-compare -Woverloaded-virtual -Wwrite-strings -Wextra -Wconversion -Wnon-virtual-dtor -Wcast-align -Wunused-parameter -fdiagnostics-color=always -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -ffp-contract=on" |
Member
Author
There was a problem hiding this comment.
I am not quite sure I agree with this one.
added 3 commits
October 16, 2026 17:32
- test-cpp-tsan now also selects every `openmp_*` case, the multi-shard coefficient launches and the threaded lazy-fold checks, which its comment already claimed to cover (142 -> 215 cases). - test-py-asan runs at T = 1 like every other ordinary suite, instead of at the runner's core count. - The sharded MPI launches, the MPI failure scenarios and the link probe allow oversubscription in both Open MPI spellings, so a direct `ctest -L mpi` fits 3- and 4-rank launches onto smaller hosts. - test_wrong_thread_entry_fails_fast fails, rather than skips, when the probe dies before reporting. - Drop the Sonar suppressions for the deleted PartitionBarrier.h and HybridComm.h. Assisted-by: Pi:claude-opus-5-5
…paths - update_initial_operator rejects a Heisenberg term the operator lacks before the epoch, the identity or any shard changes (MPOperator::require_updatable_terms), so the propagator stays usable as parallelism.mdx promises, and a mistyped term no longer aborts an MPI job. - A construction team failure on a multi-rank communicator always goes to mpi::operation_failed, including a pre-mutation frame allocation, so peers are never left blocked in the operation's rounds. - mutation_failed_ makes no MPI call on one rank, and above one rank stops a non-initializing thread locally instead of calling MPI_Abort from it. - Rename Router::partitions() to shards_per_rank() and the new kName constants to snake_case, per AGENTS.md; retire stale "partition" comments in the library, CI script and third-party bench scripts. - Document the shard-aggregating accessors, fix the EvaluationObserver doc block and a broken \throws wrap, drop includes the .inl no longer uses, and correct AGENTS.md on what build_layer is kept for. Assisted-by: Pi:claude-opus-5-5
…the sharded OpenMP refactor Bring in main since 9682c6b: term_expectation_values() (#393), the ruff 0.17 and notebook lint fixes (#416, #417) and dependency bumps. - term_expectation_values() is ported to the sharded root (ShardedPropagator.inl). Each shard's state is scattered by its owner (RootWork::adjoint_seed) and swept back through its layers, newest first, at the negated angles via replay_shards() over the new MPGraph::reverse_replay_view() -- the sharded form of ev_op_gradient(), which stays as the one-store entry. Every key's value and found flag are gathered across shards and ranks. - The Heisenberg miss check (OperatorTermNotFound) runs before mutation: the lookup reads the frozen stores and the flag allreduce is collective over replicated arguments, so the miss stays a recoverable argument error, as on main. - heisenberg_gates_absorbed_ is set by a Heisenberg propagate() and an in-place contract_partially() (contract_blocks_), and is copied with the propagator. - PartitionGroup.h and partition_group_clone_tests.cpp stay deleted: main's poison-rethrow fix belongs to the removed partition runtime. - term_expectation_values_tests.cpp drops the partition arguments, prefixes its cases and reruns at T = 2 and 4 (term_expectation_values_env_t*), against expectation_value() at the same T. - report.py: main's parenthesised header string with the branch's exactness sentence. - The two plan documents are reformatted for ruff 0.17's Markdown code blocks. MPI-off Release: ctest 515/515, pytest 904 passed; values are identical at T = 1, 2, 4 and 7. Assisted-by: Pi:claude-opus-5-5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Summary
Replace the custom partition-worker runtime with a single sharded OpenMP implementation.
Each MPI rank owns one shard per OpenMP worker, with owner-local mutation and operation-scoped execution. Only MPI’s initializing thread performs communication, reducing the required support level from
MPI_THREAD_SERIALIZEDtoMPI_THREAD_FUNNELED.This retains sharded storage and the existing numerical policy while removing custom worker management, in-process communicators, topology discovery and thread pinning.
Changes
PartitionGroup,CpuTopology,ShmComm,HybridComm, custom partition barriers and the temporary runtime selector.Changeset size by folder
Diff against the merge-base with
origin/main(90d5717→747b562):169 files changed. Counts include source, tests, documentation and configuration—not executable code alone.
.devcontainer/.github/benches/cmake/cpp/include/cpp/monoprop/cpp/tests/docs/content/docs/superpowers/nix/packages/src/tests/tools/Breaking changes and migration
partitions. Rebuild C++ consumers and bindings together with the library.monoprop_PARTITIONSis no longer read. Each propagator capturesmonoprop_NUM_THREADSat construction, falling back to the OpenMP default when unset; copies retain that budget.OMP_DYNAMIC=FALSEand sufficient allocated cores. Placement belongs to the launcher/OpenMP, not monoprop.T=1); otherwise they throwMultiShardUnsupported. Rank-local aggregate APIs cover every local shard.Recorded verification
The latest implementation records include:
Nix, macOS, Linux aarch64, CI wheel builds, minimum-version compilers and multi-node qualification remain pending. TSan coverage of Open MPI’s own MULTIPLE-level locking remains inconclusive.
Final performance campaign — running
The final campaign is currently running on a single AWS EC2
c8a.metal-24xlbare-metal node:The campaign covers the frozen 250-cell inventory, including actual MPI-OFF execution and MPI+OpenMP geometries on that node. This is single-node qualification, not evidence of multi-node or HPC-interconnect scaling.
The pre-cutover campaign completed all 250 cells with matching numerics, but only 102/250 passed all five runtime/memory gates. The owner authorized proceeding with removal without a passing pre-removal campaign; the final gates were not relaxed.
Subsequent profiling-driven changes improved targeted workloads, but those diagnostics do not establish full-library parity. The running campaign evaluates the final post-cutover binaries, with matched page-cache conditioning and fresh baseline samples for memory comparisons.
Final acceptance remains pending the campaign results: every required cell must independently satisfy runtime, operation peak sum/max and construction peak sum/max ratios ≤1.00, alongside numerical validation.
See the implementation plan for detailed evidence, decisions and outstanding qualification.
Checklist
The standing lint waiver is limited to the 49 pre-existing Ruff findings in the three documented notebooks.
AI/LLM disclosure
Important
By opening this PR I confirm that I have read CONTRIBUTING.md and I agree to the terms of the Contributor License Agreement.
Warning
If you're contributing on behalf of your employer, contact cla@algorithmiq.fi to arrange a Corporate CLA.