Refraction new

.gitignore

@@ -130,6 +130,12 @@ tests/install_scripts/t4_lf/
 !tests/wand_report_error_pdf/**
 tests/wand_report_error_pdf/**/__pycache__/
 tests/wand_report_error_pdf/data/
+# the wand tab's detection settings per camera of 13 Oct 2026: tests/wand_per_camera_detect/ (data/ holds the files
+# its Process All Frames runs write)
+!tests/wand_per_camera_detect/
+!tests/wand_per_camera_detect/**
+tests/wand_per_camera_detect/**/__pycache__/
+tests/wand_per_camera_detect/data/
 # tracking through the tube (7 Oct 2026): the library tests in tests/cylinder_stb/ and their pytest subset
 # (data/ and runs/ hold rendered images, local copies and run outputs, regenerable with its scripts)
 !tests/test_cylinder_stb.py

README.md

@@ -98,7 +98,7 @@ goes through it step by step. The figures are drawn from the two calibrations' s
 `assets/readme/make_readme_figures.py`.
 
 **The method:** S. Zhong, S. Tan, X. Zhang, Y. Li and R. Ni, "A complete forward projection algorithm through
-cylindrical and spherical refractive windows" (preprint, 2026).
+cylindrical and spherical refractive windows", [arXiv:2610.17413](https://arxiv.org/abs/2610.17413) (2026).
 
 ---
 
@@ -316,6 +316,7 @@ If you use **OpenLPT** in your research, please cite our publications:
 
 - Tan, S., Salibindla, A., Masuk, A.U.M. and Ni, R., 2020. **Introducing OpenLPT: new method of removing ghost particles and high-concentration particle shadow tracking**. *Experiments in Fluids*, 61(2), p.47.
 - Tan, S., Zhong, S. and Ni, R., 2023. **3D Lagrangian tracking of polydispersed bubbles at high image densities**. *Experiments in Fluids*, 64(4), p.85.
+- Zhong, S., Tan, S., Zhang, X., Li, Y. and Ni, R., 2026. **A complete forward projection algorithm through cylindrical and spherical refractive windows**. *arXiv preprint* [arXiv:2610.17413](https://arxiv.org/abs/2610.17413).
 
 ## License
 

cases/stb_20260727_tracer_all/2026-10-10-report.md

@@ -51,8 +51,8 @@ case; use imgFile_mask as the images for tracking."
   second row. The queue:
   - `10Hz_r_b_2` with spacing 5: tracked from 07:50 to 23:18 on 11 Oct (15.5 h), delivered at 23:27;
   - `10Hz_r_b_1` again: tracked from 23:28 to 12:32 (13.1 h), delivered at 12:38 on 12 Oct;
-  - then the ten dense runs: `15Hz_r_b_1` since 12:32, about 33 h; if the others take as long, until about
-    26 Oct (end of section 12).
+  - then the ten dense runs: `15Hz_r_b_1` tracked from 12:32 on 12 Oct to 22:05 on 13 Oct (33.6 h), delivered at
+    22:09; `15Hz_r_b_2` since 22:05. If the others take as long, until about 26 Oct (end of section 12).
 - **Still open:**
   - the masks' last frame (12,604) needs your masking tool. Until it exists, tracking stops at frame 12,603;
   - `Results_bb` is not touched, and is still one frame off from 257 on.
@@ -171,7 +171,8 @@ settings anyway (about 3 weeks per run, almost no tracks kept) and only the 10 H
     `u_quality.json`, `u_tracks.png`, `u_decision.json`, `u_queue.log`, and its console output `u_console*.txt`;
   - `step_tail.py` → `u_step_tail.txt`;
   - `config_change_check.py` → `config_change_check.txt`;
-  - `track_quality.py` on the batch's runs → `batch_quality.txt`.
+  - `track_quality.py` on the batch's runs → `batch_quality.txt`;
+  - `batch_run_summary.py` → `batch_run_summary.txt`.
 - **The smoke runs themselves (large):** `D:\Shijie Zhong\STB_staging\20260727\smoke\`.
 
 ## 8. The threshold tests (T1, T2, I5; 10 Oct)
@@ -651,3 +652,32 @@ it needs; within 8 mm, 89–97 % have.
   - **What the quality shows:** the steps are 1.5 times the 10 Hz runs', as the frequencies are. The line-fit
     residual is the dense runs' level and far below STB's limit (0.05 mm). The neighbour check is the 10 Hz runs'
     level: it is relative, so it reads higher where the flow is slower.
+
+**The first dense run, whole (13 Oct, 22:20):**
+- **15Hz_r_b_1** was tracked from 12:32 on 12 Oct to 22:05 on 13 Oct (33.6 h) and delivered at 22:09 (137 files,
+  24.2 GB in 4 min). Every file arrived on X: with its size. At its last frame it had 13,356 active long tracks.
+- **It held steady from start to end** (`tests/batch_run_summary.txt`, from its log; means per frame, saves apart).
+  The 10 Hz runs did too (same file).
+
+| frames | frame time | particles found | long tracks at the end |
+|---|---|---|---|
+| 0–1,999 | 8.22 s | 77,525 | 13,924 |
+| 2,000–3,999 | 8.38 s | 77,805 | 14,236 |
+| 4,000–5,999 | 8.40 s | 77,996 | 14,034 |
+| 6,000–7,999 | 8.42 s | 77,922 | 14,067 |
+| 8,000–9,999 | 8.45 s | 77,698 | 13,024 |
+| 10,000–11,999 | 8.23 s | 77,735 | 12,906 |
+| 12,000–12,603 | 7.98 s | 77,750 | 13,338 |
+
+- **Where the time went:** STB's own frame times add up to 29.6 h (8.45 s per frame), with its 64 saves (32 s each,
+  0.56 h together). The other 4.0 h, 1.1 s per frame, fall outside its frame timer, as in the 10 Hz runs (0.9 s).
+- **Quality at the end** (`tests/batch_quality.txt`: frame 12,400, rebuilt from the saves at 12,600 and 12,603) is
+  the same as at frame 200:
+
+| 15Hz_r_b_1 at frame | long tracks | step median | line-fit median, p90 | neighbour median, share > 0.5 |
+|---|---|---|---|---|
+| 200 | 13,474 | 0.146 mm | 0.0088 mm, 0.026 mm | 0.14, 8.8 % |
+| 12,400 | 13,362 | 0.144 mm | 0.0091 mm, 0.026 mm | 0.15, 9.5 % |
+
+- **Next:** `15Hz_r_b_2` has tracked since 22:05. If the nine runs left take 33.6 h each, the queue ends about
+  midday on 26 Oct. D: has 444 GB free, and nine more runs like this one need about 220 GB.

cases/stb_20260727_tracer_all/README.md

@@ -4,20 +4,21 @@ Your request (10 Oct 2026): "use the new calibrated cam file like what we use fo
 10Hz to 35Hz. save interval should be set to 200 frames; create a subfolder called Results_tracer inside each test
 case; use imgFile_mask as the images for tracking."
 
-**Status (12 Oct, 14:15):**
+**Status (13 Oct, 22:20):**
 - **Your choice of 11 Oct (report section 12):** every run links short tracks within 5 voxels instead of 30
   (20–32 % more long tracks in the tests), and the dense runs have their own finder thresholds.
-- **The queue:** both 10 Hz runs are tracked with spacing 5 and delivered. `chain_batch.py` started `run_all.py`
-  (pid 6780) at 23:28 on 11 Oct. `15Hz_r_b_1`, the first dense run, has tracked since 12:32 on 12 Oct and ends
-  about 22:00 on 13 Oct; at about 33 h per run the other nine take until about 26 Oct.
+- **The queue:** `chain_batch.py` started `run_all.py` (pid 6780) at 23:28 on 11 Oct. Both 10 Hz runs and
+  `15Hz_r_b_1`, the first dense run, are tracked with spacing 5 and delivered. `15Hz_r_b_1` took 33.6 h (12:32 on
+  12 Oct to 22:05 on 13 Oct, 24.2 GB), steady from start to end; `15Hz_r_b_2` has tracked since 22:05. At that
+  pace the nine runs left take until about 26 Oct.
 - **10 Hz runs:**
   - **On the renumbered images** (frames 0–12,603), with spacing 5, in `<run>\Results_tracer`: `10Hz_r_b_2`
     (07:50–23:18 on 11 Oct, 60.0 GB) and `10Hz_r_b_1` (23:28–12:32, 36.1 GB). `10Hz_r_b_1`'s spacing-30 run
     (18:22–07:50 on 10–11 Oct) is kept as `10Hz_r_b_1\Results_tracer_spacing30` on X:.
   - **R1:** frames 0–255 repeat the old run exactly, and the drop at frame 256 is gone.
   - **The run on the old images** is kept as `10Hz_r_b_1\Results_tracer_old_images` on X:.
   - **Settings check:** the batch's settings reproduce the tested run byte for byte (S1).
-- **15–35 Hz runs:** queued (above). Their images are 6–15 times denser, and their particles move 3.3 times faster.
+- **15–35 Hz runs:** tracking (above). Their images are 6–15 times denser, and their particles move 3.3 times faster.
   - **With the 10 Hz settings,** IPR finds about 390,000 mostly ghost particles per frame, 2.5 min per frame, and
     almost no tracks survive (S2).
   - **With a higher threshold** (T1, T3; report sections 8–9), they take 25–30 h per run. After 255 frames they keep
@@ -58,8 +59,7 @@ Every run is tracked as `10Hz_r_b_1\stb_new_calib\new_cams` was (case `stb_20260
 
 ## What is written where
 
-**In each run's folder on X:** (`X:\Shijie Zhong\Bubble Shear Project\Processed\20260727\Results\<run>\`; so far
-for the two 10 Hz runs)
+**In each run's folder on X:** (`X:\Shijie Zhong\Bubble Shear Project\Processed\20260727\Results\<run>\`)
 - `config.json`: the run's tracking settings, as the GUI's Save writes them. The GUI opens it with the project, and
   can run it again. In `10Hz_r_b_1` it takes over from the old `config.txt`, as after any GUI Save (the old file stays).
 - `imgFile_mask\cam0..3ImageNames.txt`: the image lists, as the GUI's preprocessing writes them. `10Hz_r_b_1` keeps
@@ -83,7 +83,7 @@ run is copied to `Results_tracer` on X:.
 | `run_all.py` | the batch: every run in turn, delivered to X:; resumes an interrupted run from its last save; `--status` |
 | `chain_batch.py` | starts the next batch when the running one has ended with every run delivered and every config checked |
 | `renumber/` | the renumbering of the images on X: (B2): `renumber.py` (`--check`, `--undo`), its logs, checks and summaries; `mock_check.py` (R0) and `rerun_identical.py` (R1) with their results |
-| `tests/` | the checks before the batch (I1, I2, I4, S1, S2, B1), the tests of the dense runs (T1–T4, I5), of the copied frame 256 (D1–D3) and the tuning tests (U0–U6), each with its results beside it |
+| `tests/` | the checks before the batch (I1, I2, I4, S1, S2, B1), the tests of the dense runs (T1–T4, I5), of the copied frame 256 (D1–D3) and the tuning tests (U0–U6), and the checks of the batch's runs (`batch_run_summary.py`; `track_quality.py` → `batch_quality.txt`), each with its results beside it |
 
 ## Running
 
@@ -105,6 +105,8 @@ Use the `refract` environment (`C:\Users\zcloc\.conda\envs\refract\python.exe`)
     python tests/tune_tests.py run|summary        # U0-U6: tuning the tracking (beside the batch)
     python tests/step_tail.py                     # U1-U6: the fastest steps against the baselines
     python tests/config_change_check.py RUN ...   # a run's configs on disk against what make_configs.py writes
+    python tests/track_quality.py RUN_DIR F [--max-save S]   # the long tracks' quality at a save frame F
+    python tests/batch_run_summary.py RUN ...     # finished batch runs, whole, from their logs
     python chain_batch.py --after PID RUN ...     # the next batch after the running one ([--dry-run])
 
 After a reboot, `python run_all.py` with all twelve runs resumes the unfinished run from its last save and tracks

cases/stb_20260727_tracer_all/tests/batch_quality.txt

@@ -1,5 +1,5 @@
-Batch runs: quality of the long tracks at frame 200 (track_quality.py), read while the batch runs, just after a
-save (report section 12, end). Measures as in u_quality.txt.
+Batch runs: quality of the long tracks (track_quality.py), at frame 200 while a run tracks (read just after a save),
+and at frame 12,400 once it is finished (report section 12, end). Measures as in u_quality.txt.
 
 15Hz_r_b_1 (spacing 5, threshold 45), 12 Oct 2026 at 14:08, after its save at frame 600:
     python tests/track_quality.py "D:\Shijie Zhong\STB_staging\20260727\15Hz_r_b_1\Results_tracer" 200 --max-save 600
@@ -15,3 +15,18 @@ nb_p90         0.4719463589958337
 nb_over_half   0.08806902707527521
 frame          200
 files          ['ExitTrack_400.csv', 'LongTrackInactive_400.csv', 'ExitTrack_600.csv', 'LongTrackActive_600.csv', 'LongTrackInactive_600.csv']
+
+15Hz_r_b_1 at its end, 13 Oct 2026 at 22:15, after the run finished (rebuilt from the saves at 12,600 and 12,603):
+    python tests/track_quality.py "D:\Shijie Zhong\STB_staging\20260727\15Hz_r_b_1\Results_tracer" 12400
+tracks         13362
+step_med       0.14382774818163693
+step_p90       0.18245470651587098
+lf_med         0.009092118454036603
+lf_p90         0.025765276179949007
+nb_radius_mm   8.0
+nb_coverage    0.9966322406825325
+nb_med         0.14970904966496307
+nb_p90         0.48560074880169046
+nb_over_half   0.0948411804460464
+frame          12400
+files          ['ExitTrack_12600.csv', 'LongTrackInactive_12600.csv', 'ExitTrack_12603.csv', 'LongTrackActive_12603.csv', 'LongTrackInactive_12603.csv']

cases/stb_20260727_tracer_all/tests/batch_run_summary.py

@@ -0,0 +1,72 @@
+"""The batch's finished runs, whole, from their logs (report section 12, end).  Read only.
+
+For each run given that has DONE.json in its staging folder:
+  - when it was tracked and delivered (batch_status.json, DONE.json);
+  - its time in all, the sum of STB's own frame times and its saves' (count, median, sum), and the time outside them;
+  - per 2,000 frames: the mean frame time (saves apart), the particles IPR found, the long tracks promoted, lost
+    (to the inactive ones) and exited per frame, and the active long tracks at the block's end;
+  - the active long tracks at the last frame (`la_end`, as run_all.py records it).
+
+    python tests/batch_run_summary.py RUN ... > tests/batch_run_summary.txt
+"""
+from __future__ import annotations
+
+import datetime
+import json
+import os
+import statistics
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.dirname(HERE))
+import case  # noqa: E402
+import parse_log  # noqa: E402
+
+BLOCK = 2000
+
+
+def read_json(path):
+    with open(path, encoding="utf-8") as f:
+        return json.load(f)
+
+
+def summary(cfg, status, run):
+    sd = case.stage_dir(cfg, run)
+    done_path = os.path.join(sd, "DONE.json")
+    if not os.path.exists(done_path):
+        print(f"{run}: not finished (no DONE.json)\n")
+        return
+    done, st = read_json(done_path), status[run]
+    recs = parse_log.parse(os.path.join(sd, "log.txt"))
+    n, last = len(recs), recs[-1]["frame"]
+    tot = sum(r["t_total"] for r in recs)
+    sv = [r["t_total"] for r in recs if r["saved"]]
+    wall = st["wall_s"]
+    print(f"{run}: tracked {st['start']} to {st['end']}, {n} frames (0..{last}); delivered {done['delivered']}, "
+          f"{done['files']} files, {done['GB']:.1f} GB")
+    print(f"  in all {wall / 3600:.2f} h = {wall / n:.2f} s per frame; STB's frame times {tot / 3600:.2f} h = "
+          f"{tot / n:.2f} s per frame, with {len(sv)} saves (median {statistics.median(sv):.1f} s, together "
+          f"{sum(sv) / 3600:.2f} h); outside them {(wall - tot) / n:.2f} s per frame")
+    print("  frames         frame time  particles  promoted  lost  exited  long tracks")
+    print("                 (no save)   found      ------ per frame -----  at the end")
+    for a in range(0, last + 1, BLOCK):
+        w = [r for r in recs if a <= r["frame"] < a + BLOCK and "la_n" in r]
+        if not w:
+            continue
+        m = lambda k: statistics.mean(r.get(k, 0) for r in w)  # noqa: E731
+        t = statistics.mean(r["t_total"] for r in w if not r["saved"])
+        print(f"  {a:5d}-{min(a + BLOCK - 1, last):<5d}    {t:6.2f} s  {m('ipr_found'):9.0f}  {m('la_add'):8.0f}  "
+              f"{m('li_add'):4.0f}  {m('ex_add'):6.1f}  {w[-1]['la_new']:11d}")
+    print(f"  active long tracks at the last frame: {st['la_end']}\n")
+
+
+def main():
+    cfg = case.load_config()
+    status = read_json(os.path.join(cfg["staging_root"], "batch_status.json"))
+    print(f"The batch's finished runs, whole ({datetime.datetime.now():%d %b %Y, %H:%M}); report section 12, end.\n")
+    for run in sys.argv[1:]:
+        summary(cfg, status, run)
+
+
+if __name__ == "__main__":
+    main()

cases/stb_20260727_tracer_all/tests/batch_run_summary.txt

@@ -0,0 +1,41 @@
+The batch's finished runs, whole (13 Oct 2026, 22:18); report section 12, end.
+
+10Hz_r_b_1: tracked 2026-10-11T23:27:52 to 2026-10-12T12:31:42, 12604 frames (0..12603); delivered 2026-10-12T12:38:15, 137 files, 36.1 GB
+  in all 13.06 h = 3.73 s per frame; STB's frame times 9.94 h = 2.84 s per frame, with 64 saves (median 46.1 s, together 0.81 h); outside them 0.89 s per frame
+  frames         frame time  particles  promoted  lost  exited  long tracks
+                 (no save)   found      ------ per frame -----  at the end
+      0-1999       2.58 s      11449       269    96    17.2        21472
+   2000-3999       2.62 s      11261       262   100    20.3        21071
+   4000-5999       2.63 s      11104       256    99    19.9        20708
+   6000-7999       2.62 s      11226       257    98    19.5        21377
+   8000-9999       2.62 s      11292       262   100    19.7        20469
+  10000-11999      2.65 s      11208       261    98    19.6        21815
+  12000-12603      2.64 s      11088       257    99    20.0        21525
+  active long tracks at the last frame: 21517
+
+10Hz_r_b_2: tracked 2026-10-11T07:49:52 to 2026-10-11T23:17:53, 12604 frames (0..12603); delivered 2026-10-11T23:26:45, 137 files, 60.0 GB
+  in all 15.47 h = 4.42 s per frame; STB's frame times 12.18 h = 3.48 s per frame, with 64 saves (median 92.8 s, together 1.60 h); outside them 0.94 s per frame
+  frames         frame time  particles  promoted  lost  exited  long tracks
+                 (no save)   found      ------ per frame -----  at the end
+      0-1999       3.00 s      13213       323    94    28.7        35515
+   2000-3999       3.03 s      12787       305    95    32.0        34573
+   4000-5999       3.03 s      12767       300    92    31.4        35150
+   6000-7999       3.04 s      12704       301    93    32.1        35849
+   8000-9999       3.06 s      12827       307    97    32.0        34934
+  10000-11999      3.06 s      12838       303    95    32.2        35597
+  12000-12603      3.06 s      12845       300    95    32.2        35480
+  active long tracks at the last frame: 35511
+
+15Hz_r_b_1: tracked 2026-10-12T12:31:42 to 2026-10-13T22:04:50, 12604 frames (0..12603); delivered 2026-10-13T22:08:38, 137 files, 24.2 GB
+  in all 33.55 h = 9.58 s per frame; STB's frame times 29.59 h = 8.45 s per frame, with 64 saves (median 32.1 s, together 0.56 h); outside them 1.13 s per frame
+  frames         frame time  particles  promoted  lost  exited  long tracks
+                 (no save)   found      ------ per frame -----  at the end
+      0-1999       8.22 s      77525      2615   193    23.1        13924
+   2000-3999       8.38 s      77805      2623   194    23.6        14236
+   4000-5999       8.40 s      77996      2620   192    23.5        14034
+   6000-7999       8.42 s      77922      2628   194    23.8        14067
+   8000-9999       8.45 s      77698      2620   195    24.1        13024
+  10000-11999      8.23 s      77735      2616   194    23.7        12906
+  12000-12603      7.98 s      77750      2627   193    22.9        13338
+  active long tracks at the last frame: 13356
+

modules/camera_calibration/curved_calibration/cylinder_wand_gui.py

@@ -118,6 +118,7 @@ class Job:
     pixel_mm: Optional[object] = None      # pixel size (mm), one number or one per camera: only to print f in mm
     detect_mode: Optional[str] = None      # how the points were found ("ellipse", "fast", "reliable"; None: unknown)
     source: str = ""                       # where the points came from (for the report)
+    detection: Optional[Dict] = None       # the points' detection record (wand_calibrator.save_detection_record)
 
     @property
     def tube(self) -> cm.Tube:
@@ -137,7 +138,7 @@ def settings(self) -> Dict:
                 "n": [float(v) for v in self.n], "fit_k1": bool(self.fit_k1), "fit_L": bool(self.fit_L),
                 "noise_weights": self.noise_weights, "n_final": int(self.n_final),
                 "refit_from_last_result": self.start is not None, "pixel_mm": self.pixel_mm,
-                "detect_mode": self.detect_mode, "source": self.source,
+                "detect_mode": self.detect_mode, "source": self.source, "detection": self.detection,
                 "frame_ids": [int(f) for f in self.frame_ids], "cam_ids": [int(c) for c in self.cam_ids],
                 "sizes": [[int(w), int(h)] for (w, h) in self.sizes],
                 **({"fix_f": [int(c) for c in self.fix_f]} if self.fix_f else {})}