Skip to content

The largest simulation results are not held twice while they are built - #954

Open
jmrplens wants to merge 1 commit into
fix/regimes-derive-from-their-valuesfrom
perf/simulation-results-without-a-second-copy
Open

jmrplens wants to merge 1 commit into
fix/regimes-derive-from-their-valuesfrom
perf/simulation-results-without-a-second-copy

Conversation

@jmrplens

@jmrplens jmrplens commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Since every public record keeps a read-only copy of its own of the arrays it holds, the largest results paid for that copy in memory: while the record was being built, the solver's arrays and the record's copy were both alive. This change takes that cost away again for the four results where it was large, without weakening the rule. A record still holds read-only arrays that nothing else can reach, and a record you build by hand still copies what you give it.

simulation.fdtd_simulation and simulation.elastic_fdtd_simulation gathered their snapshots in a list, stacked the list into a second array and handed that to the record, which copied it once more, so a run held its frames three times at its peak. Each now writes every frame, and every probe sample, into one array allocated before the march with the exact number of frames. FDTD2D.run and ElasticFDTD2D.run, which held their frames as a list and its stack, write into one array the same way; what they return stays a writeable array, as before.

The two simulation functions then hand that array to their record, which seals it read-only and keeps it instead of copying it. underwater.parabolic_equation and environment.atmospheric_parabolic_equation already wrote their field range by range into one array, so for them the record's copy was the whole extra cost, and they hand their field over the same way. This goes through a private function, handed_over in phonometry._internal.frozen. The record keeps such an array only if it owns its memory and is C-ordered, and only the first record it reaches keeps it; anything else is copied as before, so every record holds the same layout however it was built.

Keeping an array without a copy is only right when nothing else can reach it, so make array-aliasing and its CI job gain a part that proves this from the code for every handed_over call, or fails, with no exemption. The call has to be an argument of a record that copies its arrays, built in the return statement. Its argument has to be a name bound once, in the function and to it alone, to np.zeros, np.empty or np.ones (or None in one branch of a conditional), in a function that is no generator and does not hand out its frame (locals(), vars(), eval, exec). Every other read of the name has to write into the array, test it against None, or pass it to an undecorated helper of the package whose parameter is used the same way, three calls deep at most. A view, a second name, a container, a closure or a lambda rules it out. The function is found however it is spelt: imported under another name, read through a dotted path to its module, or fetched by its name as a string.

Peak memory, measured with tracemalloc on the same runs before and after:

  • FDTD on 400 by 600 cells with a snapshot every ten steps (269 MB of result): 855 to 322 MB.
  • Elastic FDTD on 200 by 500 cells with its snapshots (81 MB): 287 to 128 MB.
  • Underwater parabolic equation over 20 km on 4096 depths in 5 m steps (131 MB): 333 to 202 MB.
  • Atmospheric GFPE at 1 kHz, 3 km long and 100 m high (29 MB): 139 to 111 MB.

Before results copied their arrays at all, the same runs peaked at 587, 207, 202 and 112 MB, so the two FDTD runs now need less memory than they did before that change too.

What breaks. Nothing: no public name, signature, default or value changes, every result is bit for bit what it was, and the arrays a result holds were already read-only. There is nothing to migrate. For contributors, CONTRIBUTING section 7f says when a factory may hand an array over and what the gate proves of each call, and the Makefile and CI comments describe the gate's new part.

How it was verified. Every array of fifteen runs (small and benchmark-sized FDTD and elastic FDTD runs with and without snapshots, each snapshot field, run() on both engines, both parabolic equations) was compared with the same runs before this change, dtype, shape and bytes, with np.array_equal and the writeable, C-contiguous and own-data flags: no difference. New tests pin each snapshot frame to the field of its own step: at a probe's cell, frame k equals, bit for bit, what the probe recorded k cadences in, on runs that are not a whole number of cadences long, for the acoustic FDTD and for the elastic p, vx and vy fields. A frame index one cadence off fails them. tests/test_owns_arrays.py (57 tests, 15 new) checks that an array handed over is kept and sealed, that a second record copies it, that a view (C-ordered or not) or a Fortran-ordered array is copied and the caller's view keeps its writeable flag, and with tracemalloc that none of the four functions and neither run() holds its result twice; putting back the list and its stack fails them. tests/test_check_array_aliasing.py (116 tests, 35 new) has the gate accept the factories' shape and refuse each way the array could be reached otherwise, a decorated helper and a dotted path to the function included, and the gate was seen red on a scratch copy of the tree with a snapshot array kept on the engine and with a view of a parabolic-equation field handed over. The full suite passes on Python 3.13 (27201 passed, 69 skipped), the conformance report is unchanged (1893 of 1893 checks), the API reference and llms files regenerate with no change, the related check scripts pass, and ruff, ruff format, mypy (689 files) and bandit are clean. No documentation code, figure or site page changed.

Summary by CodeRabbit

  • Performance

    • Reduced peak memory use when recording FDTD, elastic FDTD, and propagation results by filling result arrays directly during simulations.
    • Updated peak-memory figures for supported simulation types.
  • Behavior

    • Result records retain read-only arrays; arrays created by simulations can be stored without an extra copy when safe. The array returned by run() remains writable.
    • Snapshot frames remain aligned with their corresponding simulation steps, including the final frame when the run length does not match the snapshot interval.

The two FDTD simulation functions and both run() methods now write every frame and probe sample into one array allocated before the march, and the four largest results (both FDTD simulations and both parabolic equations) hand that array to their record, which seals it read-only and keeps it instead of copying it. The private handed_over keeps an array only if it owns its memory, is C-ordered and reaches its first record; anything else is copied as before. make array-aliasing gains a part that proves from the code that nothing else can reach an array handed over, with no exemption. Every result is bit for bit what it was, and peak memory falls from 855 to 322 MB, 287 to 128 MB, 333 to 202 MB and 139 to 111 MB on the measured runs.
@jmrplens
jmrplens added this pull request to stack #955 October 8, 2026 07:01
@sourcery-ai

sourcery-ai Bot commented Oct 8, 2026

Copy link
Copy Markdown

Sorry @jmrplens, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 2 days and 12 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: jmrplens/phonometry/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 212f1123-921e-4d67-ab56-3d952af6e67a
📥 Commits

Reviewing files that changed from the base of the PR and between 10ca282 and 7e418ae.

📒 Files selected for processing (14)
  • .github/workflows/python-app.yml
  • CHANGELOG.md
  • CONTRIBUTING.md
  • Makefile
  • scripts/check_array_aliasing.py
  • src/phonometry/_internal/frozen.py
  • src/phonometry/environment/propagation/refraction.py
  • src/phonometry/simulation/elastic_fdtd.py
  • src/phonometry/simulation/fdtd.py
  • src/phonometry/underwater/propagation/numerical.py
  • tests/simulation/test_elastic_fdtd.py
  • tests/simulation/test_fdtd.py
  • tests/test_check_array_aliasing.py
  • tests/test_owns_arrays.py

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.


📝 Walkthrough

Walkthrough

The change adds handed_over for eligible arrays retained by records without copying. FDTD and elastic FDTD now write into preallocated result arrays. The array-aliasing check validates handoff usage, and tests cover ownership, snapshot alignment, and memory use.

Changes

Array handoff and simulation results

Layer / File(s) Summary
Array ownership and adoption
src/phonometry/_internal/frozen.py, CONTRIBUTING.md, CHANGELOG.md, tests/test_owns_arrays.py
handed_over marks arrays for adoption. Records retain eligible owned, C-contiguous arrays as read-only; other inputs follow the existing copy path. Tests cover identity, mutability, and later record construction.
Preallocated simulation results
src/phonometry/simulation/fdtd.py, src/phonometry/simulation/elastic_fdtd.py, src/phonometry/underwater/propagation/numerical.py, src/phonometry/environment/propagation/refraction.py, tests/simulation/*, tests/test_owns_arrays.py, CHANGELOG.md
FDTD and elastic FDTD allocate result arrays before recording and fill them in place. Parabolic-equation results pass computed arrays through handed_over. Tests check snapshot alignment and result-array memory use.
Handoff proof and enforcement
scripts/check_array_aliasing.py, tests/test_check_array_aliasing.py, CONTRIBUTING.md, Makefile, .github/workflows/python-app.yml
The checker reports handed_over calls that do not meet its allocation, use, and returned-record rules. Tests cover accepted and rejected patterns. The contributor guidance and check descriptions document the rules.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Refactor

Sequence Diagram(s)

sequenceDiagram
  participant fdtd_simulation
  participant _record_run
  participant FDTDResult
  participant _owned_array
  fdtd_simulation->>fdtd_simulation: Allocate pressure history and optional snapshots
  fdtd_simulation->>_record_run: Pass arrays for in-place recording
  _record_run->>_record_run: Write initial field and scheduled frames
  fdtd_simulation->>FDTDResult: Pass arrays through handed_over
  FDTDResult->>_owned_array: Check arrays for adoption
Loading

Merge Risk: ⚪ Minimal · up to 7e418

The change cuts peak memory for large simulation results, and the author reports no public output changes. I found no concrete merge-blocking risk in the supplied context. Normal CI should still run before merge.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 7e418

The reviewed factories retain caller isolation while reducing peak memory use. No introduced security defect was established, but failure recovery and concurrent handoff guarantees are not fully demonstrated.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The demonstrated exposure is in-process numerical result ownership across four factories. The marking registry is shared by OwnsArrays consumers, so an incorrect future handoff could affect record isolation beyond those factories; the inspected production handoffs do not demonstrate such an alias.

Trust Boundaries and Controls

  • observed — Runtime adoption checks identity, one-shot eligibility, memory ownership, and layout; it does not independently establish absence of aliases. The static control requires a fresh local allocation passed directly into a returned OwnsArrays record and restricts other uses to writes, None comparisons, or bounded inspected helper calls.
  • observed — The Python CI workflow runs the ownership checker directly on pull requests. Its dedicated job uses read-only repository permission and disables persisted checkout credentials; the PR changes the job's explanatory text, not its command or permissions.

Resilience and Maintainability Implications

  • inferred — Failed marches precede marking and therefore do not publish adopted partial results through these return paths. Construction has no explicit handoff rollback: an unconsumed mark remains weak while its array is alive, and a consumed allocation remains sealed after a later hook failure. The inspected factories provide no retry or shared-buffer recovery path, but a general interruption or concurrent-adoption contract is not established.

Hardening Proposals

  • proposed — Before extending handoff to retrying or shared-buffer producers, specify and regression-test mark ownership after interrupted argument evaluation, constructor failure, and simultaneous adoption. This would make the recovery contract explicit rather than relying on current factories' local, terminal-return structure.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 69.49% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 59 functions across 10 files. (4 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main change: reducing duplicate in-memory storage while large simulation results are built.
Description check ✅ Passed The description explains the motivation, implementation, safety rules, measured memory improvements, validation results, and documentation impact. It does not reproduce the checklist headings or check…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 69.49% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 59 functions across 10 files. (4 skipped: 4 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added documentation Improvements or additions to documentation ci Workflows, linting and developer tooling area: environment Outdoor propagation, environmental sources and noise assessment area: simulation FDTD and other numerical solvers area: underwater Underwater acoustics and propagation area: core Shared internals and cross-cutting code every domain depends on labels Oct 8, 2026
@codecov

codecov Bot commented Oct 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.81%. Comparing base (10ca282) to head (7e418ae).

Additional details and impacted files
@@                          Coverage Diff                          @@
##           fix/regimes-derive-from-their-values     #954   +/-   ##
=====================================================================
  Coverage                                 96.81%   96.81%           
=====================================================================
  Files                                       443      443           
  Lines                                     89035    89043    +8     
=====================================================================
+ Hits                                      86200    86211   +11     
+ Misses                                     2835     2832    -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Numerical conformance

All 1893 conformance checks pass, across 111 domains and 523 standards

1893/1893 checks pass across 111 domains and 523 standards (227 normative designations, 125 further published sources).

Used in the tables below is how much of that clause's published tolerance the deviation consumes: 100 % sits exactly on the limit, 5 % uses a twentieth of the allowance, and a dash means the clause states no two-sided tolerance for the quantity, so there is no budget to spend. It is reported and never used to decide a verdict, which is settled at full precision before any rounding.

Nothing moved: same 1893 checks, same verdicts, same numbers.

Closest to their published limit (top 5)

The rows with the least room left, so the ones a change is most likely to push over.

Standard Quantity Deviation Used
ISO/TR 17534-3:2015 Table 3 Ground-projected path length dp, m 0.005 m 100 %
DIN 4150-2:1999-06 Annex C, Example 5 KB_FTr with hammer b) in the rest hours, Formula (5) -0.005 100 %
DIN 4150-2:1999-06 Annex C, Example 8 KB_FTm over the record with the passage maxima alone -0.0005 100 %
E DIN 4150-2:2023-08 Annex B, Table B.1 KB_FTm,Zug of the metro north by Formula (5) -0.0005 100 %
Jarvis (1996) NPL CIRA(EXT) 010 Appendix B, folio 26 Volume of the coupler of the example, v = pi r^2 L -0.00000000005 m³ 100 %

Pass Tests & coverage: 163620 tests, 0 failures (all green)
Python Version Tests Failures Coverage Status
macos-latest-3.13 27270 0 96.8% Pass Passed
macos-latest-3.14 27270 0 96.8% Pass Passed
ubuntu-latest-3.13 27270 0 96.8% Pass Passed
ubuntu-latest-3.14 27270 0 96.8% Pass Passed
windows-latest-3.13 27270 0 96.8% Pass Passed
windows-latest-3.14 27270 0 96.8% Pass Passed

Full report at this commit: docs/CONFORMANCE.md · docs/conformance.json · full CI artifacts

@sonarqubecloud

sonarqubecloud Bot commented Oct 8, 2026

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: core Shared internals and cross-cutting code every domain depends on area: environment Outdoor propagation, environmental sources and noise assessment area: simulation FDTD and other numerical solvers area: underwater Underwater acoustics and propagation ci Workflows, linting and developer tooling documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant