Repository navigation
Conversation
19b5c6f to
1a6f5dc
Compare
vjaganat90
left a comment
There was a problem hiding this comment.
I reviewed the binning changes, the 3D background handling in RAM and out of core, the CLI and Python validation, the tests and registry rows, and the interaction with #485 and #488. What I checked and found right:
- Binning formula:
floor(n (x − min) / (max − min)) + 1with the max clamped is exact. It can't go a bin high for any realistic bin count (up to about 2²⁰). - CLI: the
strtolhandling is correct. - NGLDM: positive depths are unchanged, and the removed helpers were unused.
- 3D GLRLM:
RPnow divides by the ROI voxel count. - In RAM vs out of core: the two paths agree for GLSZM, GLDM and GLRLM.
- Histograms: the header and value counts match in every writer, Arrow included.
- Regression tables: the regenerated 3D GLRLM grey64 table is right. I wrote the 64-level values into the image and ran main's unbinned path on it, and all 16 first-direction values and 16 means match within 7.7e-10. The GLSZM and GLDM tables correctly don't move, because the phantom ROI's lowest MATLAB level is 22.
Blocking
- The default-output change and the
set_paramsrefusal aren't marked as breaking (inline onenv_features.cpp). The commit is a plainfix(binning):, so release-please files it under Bug Fixes as a patch.
Nits are inline. Two more without an anchor in the diff:
- 3D NGTDM moves to 64 MATLAB levels before its background fix.
3d_ngtdm.cpp:231still setszeroI = 1and skips every level-1 voxel as a centre. The description says NGTDM "gets the same fix in its own PR"; that is #484, but it isn't linked. Please link #484 and land this after it, or say in the description that 3D NGTDM's new default stays wrong on level-1 voxels until #484 merges. #484 also addsroi_mask_3d()next tobin_roi_3d()in3d_ooc_volume.h, so expect a small conflict there. e754bb76 ci: re-run checks after a conda-forge outageis an empty commit. Please drop it before merge; re-running the failed job does the same thing.
Merge order with #485 and #488
-
#485 adds
run_3d_feature_on_volumeintests/test_main_nyxus.hwith a default, emptyAABB. With this PR, 3D GLSZM and GLRLMcalculate()go throughbin_roi_3d→require_contains, which throws on that box. After both merge, #485'sTEST_3D_GLSZM_LARGE_ZONE_SIZE_EMPHASIS_ANALYTICandTEST_3D_GLRLM_LONG_RUN_EMPHASIS_ANALYTICfail. Git conflicts only intest_3d_glszm_common.h,test_all.ccandreport_output.*. Whichever lands second has to move the box set-up this PR adds attest_3d_glszm_common.h:234-238into that shared runner, and regenerate the report. -
#488 conflicts with this PR in
README.md,cmdline_and_examples.rstandnyxus.py. It also documents, and itstest_2d_docs_examples_mechanics.pyruns, behaviour this PR changes:directory_and_params.py:28callsset_params(coarse_gray_depth=256), which now raises.glrlm_3d_worked_example.py:78-86pinsRP1.182 andRP_AVE1.130.glszm_3d_worked_example.py:84-85andgldm_3d_worked_example.py:66-73assert that the default equals IBSI and thatcoarse_gray_depthdoesn't change those families.
Beyond the failing examples, these go stale:
- the
--coarseGrayDepthentry incli_options.tomlandcli_reference.rst; conventions.rst's binning section;- the 3D defaults in
featurelist.rstandf_{glrlm,glszm,gldm,ngtdm}_3d.rst; - the
nyxus.pydocstrings, which saycoarse_gray_depthmust be positive and that a changed depth only updatesget_params.
#488's
cli_options.tomlcites defaults by line number, and the three includes this PR adds toenvironment.cppshift six of them, so its--checkfails on the merged tree. I'm asking on #488 for symbol-based citations. Either way, the second PR to land owns the update.
| // The other per-family grey depths start at the common one too, so the coarse grey depth | ||
| // bins every texture family; a family's own metaparameter ("3glrlm/greydepth=...") then | ||
| // overrides it for that family alone. | ||
| s[(int)NyxSetting::GLDM_GREYDEPTH].ival = get_coarse_gray_depth(); | ||
| s[(int)NyxSetting::GLRLM_GREYDEPTH].ival = get_coarse_gray_depth(); | ||
| s[(int)NyxSetting::GLSZM_GREYDEPTH].ival = get_coarse_gray_depth(); | ||
| s[(int)NyxSetting::NGTDM_GREYDEPTH].ival = get_coarse_gray_depth(); |
There was a problem hiding this comment.
Blocking: These lines move the default output of 3D GLRLM, GLSZM, GLDM and NGTDM from raw levels to 64 MATLAB-style levels, so every existing 3D run now returns different numbers for those four families. On top of that, set_params(coarse_gray_depth=...) and set_environment_params(coarse_gray_depth=...) used to return and now raise ValueError (nyxus.py _without_coarse_gray_depth). Both changes break existing users. The commit is still fix(binning): ..., with no ! and no BREAKING CHANGE: footer (git log -1 --format=%B 1a6f5dcd), so release-please files it under Bug Fixes and bumps a patch version. The repo already marks breaking changes this way (feat(omezarr)!: with a BREAKING CHANGE: footer).
Please make the commit and the PR title fix(binning)!: ... and add a BREAKING CHANGE: footer that states three things:
- the new 3D default;
- that
--coarseGrayDepth=0or3<family>/greydepth=0gives the old output; - that
set_params/set_environment_paramsnow refuse a differentcoarse_gray_depth.
|
|
||
| FAMILIES = {"3GLSZM_": "*3D_GLSZM*", "3GLDM_": "*3D_GLDM*", "3GLRLM_": "*3D_GLRLM*"} | ||
|
|
||
| # 8 x 90 x 90 voxels: a footprint well over 1 MB, so ram_limit=1 streams the ROI out of core |
There was a problem hiding this comment.
Nit: The comment says "a footprint well over 1 MB, so ram_limit=1 streams the ROI out of core". For the ellipsoid that is not true on every platform. LR::get_ram_footprint_estimate_3D (roi_cache.cpp:33) counts (box cells + ROI voxels) × sizeof(Pixel2) plus 758×10×8 bytes. On LP64, Pixel2 is 24 bytes, so the total is 28,800 + 15,168 = 43,968 cells × 24 + 60,640 = 1,115,872 bytes, which clears 1 MiB by only 6%. On MSVC, long is 32-bit, Pixel2 is 12 bytes, and the total is 588,256 bytes. That is under the limit, so on Windows CI test_3d_texture_background_by_roi_mask_mechanics runs its "out of core" leg in RAM. Nothing asserts which path ran. That leg is the only out-of-core case with background inside the box, so it is the case this PR's level-0 change is about. Please use ram_limit=0 for that side, which every footprint reaches, as test_ooc_mechanics.py already does for its whole-volume case. Alternatively, assert the path from the verbose log.
| if (ibsi || ibsi_grey_binning(greybin_info)) | ||
| return x; | ||
| if (radiomics_grey_binning(greybin_info)) | ||
| return to_grayscale_radiomix (x, min_I_inten, max_I_inten, std::abs(greybin_info)); |
There was a problem hiding this comment.
Nit: The CLI now refuses INT_MIN and anything past the int range "rather than wrapped". The per-family metaparameter feeds the same std::abs(greybin_info) here and in bin_pixel/bin_intensities*, and it still goes through Nyxus::parse_as_int (helpers.h:115, result = (int)res from a long), which does not check the range. Two results:
set_metaparam("3glszm/greydepth=-2147483648")storesINT_MIN, andstd::absof it is UB.set_metaparam("3glrlm/greydepth=4294967276")is accepted and becomes -20, i.e. 20 PyRadiomics bins.
Now that a family's greydepth means the same thing as the coarse depth, please give parse_as_int (or the greydepth branches in env_metaparams.cpp) the same range check as parse_cmdline.
| // GLSZM_GREYDEPTH=64, whose positive sign selects MATLAB-style binning into that many levels, so bin 1 | ||
| // is the background level and a voxel binned there starts no zone of its own. | ||
| // | ||
| // Recipe glszm3d.regression_ut_phantom_nobinning: the same phantom at GLSZM_GREYDEPTH=0, which is the | ||
| // value a run that passes no --3glszm/greydepth reaches the feature with. It is a third binning | ||
| // scheme rather than a coarser version of the second -- the background is intensity 0 there and bin 1 | ||
| // under MATLAB binning -- so it gets its own table. Nothing vets it: PyRadiomics has no counterpart | ||
| // for reading raw levels off an unbinned volume, so these are Nyxus' own numbers and catch movement | ||
| // only. What the family's default DOES have an oracle for is the same setting on a fixture small | ||
| // enough for one: test_3d_glszm_ibsi_gapped_pyradiomics. | ||
| // Recipe glszm3d.regression_ut_phantom_nobinning: the same phantom at GLSZM_GREYDEPTH=0, no binning, | ||
| // which --coarseGrayDepth=0 or 3glszm/greydepth=0 selects. A run that names neither bins at the coarse | ||
| // grey depth, 64, which is the recipe above. It is a third binning scheme rather than a coarser version | ||
| // of the second -- the background is intensity 0 there and bin 1 under MATLAB binning -- so it gets |
There was a problem hiding this comment.
Nit: After this change, the background is level 0 under every scheme and a ROI voxel at level 1 does start a zone. These comments still describe the old behaviour:
- lines 7-8 say GLSZM_GREYDEPTH=64 makes "bin 1 the background level and a voxel binned there starts no zone of its own";
- line 13 says the background is "bin 1 under MATLAB binning";
tests/vetting/config_recipes.md:1013-1019(glszm3d.regression_ut_phantom) repeats the bin-1 claim and still calls 0 "this family's documented 'no binning' default";tests/vetting/audit/glszm_3d_pyradiomics_vetting_report.md:330-333still callsGLSZM_GREYDEPTH=0"this family's documented 'no binning' default".
The table happens not to move because the ut_ ROI's lowest MATLAB level is 22. Please update the text.
There was a problem hiding this comment.
The header, config_recipes.md and the audit report are fixed. The 16 glszm3d.regression_ut_phantom_nobinning rows in oracle_coverage.csv (957, 961, ... 1017), which this PR rewrites, still say "the background is intensity 0 here and bin 1 under MATLAB binning". With the level-0 background that is no longer true. Please use the header's new wording there ("its levels are the raw intensities, not 64 levels over [0, max]") and regenerate report_output.csv.
There was a problem hiding this comment.
Fixed The 16 glszm3d.regression_ut_phantom_nobinning rows now describe no binning as the raw intensities against 64 levels over [0, max], with no "bin 1" background, and report_output.csv is regenerated, in c42d6fa.
| /// mod 37), 36 distinct over 100-208. At 20 PyRadiomics-style bins five of its pixels sit exactly | ||
| /// on a bin edge, so it separates IBSI's fixed bin number from a formula that rounds the edge |
There was a problem hiding this comment.
Nit: At 20 bins over 100..208 the edges fall at 127, 154 and 181, and 14 of the 144 pixels lie exactly on one (5 at 127, 4 at 154, 5 at 181). The rounded-bin-width formula gets 127 and 154 right. Only the 5 pixels at 181 went one bin low (15 instead of 16). The fixture is fine; the wording isn't. The same "five ... on a bin edge" wording is in test_2d_gldzm_mirp.h:214, test_2d_ngldm_mirp.h:83, the registry notes and gen_negative_greydepth_mirp.py:20-22. The generator's "The 3D phantom has no such pixel" is also wrong: 2736 of its voxels sit on an edge. None of them is misbinned only because the bin width there, 2000/20 = 100, is exact. Please say "14 pixels sit on a bin edge, and the old formula sent the 5 at 181 one bin low".
| std::cerr << "Error: " << clo_COARSEGRAYDEPTH << "=" << raw_coarse_grayscale_depth << ": expecting an integer constant\n"; | ||
| return false; | ||
| } | ||
| if (errno == ERANGE || v <= INT_MIN || v > INT_MAX) |
There was a problem hiding this comment.
Nit: The CLI test and the Python check advertise ±(2³¹−1) as valid, but nothing in the pipeline can run there:
histogram_bin_countgives the HISTOGRAM writers 2³¹−1HISTOGRAM_BIN_*columns (output_2_csv.cpp:197);TrivialHistogramallocates |n|+1 bins per ROI, andval_HISTOGRAMn doubles, about 17 GB at that depth;- 2D GLCM at a positive depth allocates depth² cells (glcm.cpp:405).
The positive side was already like this, and this PR opens the same range for negative depths. Please set a documented upper bound on |depth|, or at least don't present the extremes as supported values in the docstring and tests.
| # pixel count. MEAN is requested beside it because HISTOGRAM on its own comes back as zeros: it belongs to | ||
| # no featureset that triggers the intensity family's calculation. |
There was a problem hiding this comment.
Nit: The comment says HISTOGRAM requested on its own "comes back as zeros". That is a silent wrong-output bug. I reproduced it on main: Nyxus(["HISTOGRAM"]) returns 64 columns that sum to 0, while ["MEAN","HISTOGRAM"] returns counts summing to the ROI's pixel count. The test works around it by adding MEAN, so nothing tracks it. Please file it and cite the ID here, so the workaround can be removed when it is fixed.
| Grey binning of the texture features, selected by the sign: a positive value bins into | ||
| that many MATLAB-style levels, 0 bins nothing (the raw levels, as IBSI reads them), and a | ||
| negative value bins into abs(value) PyRadiomics-style bins over each ROI's range. Every 2D | ||
| and 3D texture family uses it unless a family's own metaparameter overrides it. The | ||
| first-order histogram statistics use abs(value) bins, or 24 at 0. |
There was a problem hiding this comment.
Nit: This text is copied from Nyxus/Nyxus3D. ImageQuality computes no texture families and no first-order histogram, and none of focus_score/power_spectrum/saturation/sharpness reads the grey depth. Please say the parameter is accepted and validated but has no effect on image-quality features.
| if (GLDZMFeature::required(env.theFeatureSet)) | ||
| { | ||
| Fsettings& s = env.fsett_D3_GLDZM; | ||
| Fsettings& s = env.fsett_GLDZM; |
There was a problem hiding this comment.
Nit: The three swaps at 551 (2D GLDZM read fsett_D3_GLDZM), 699 (3D GLSZM read fsett_D3_GLDZM) and 855 (enclosing circle read the hexagonality vector) are defect fixes, and CLAUDE.md asks for a short comment on the changed line saying what was wrong. The enclosing-circle swap also has no visible effect today, since neither vector has a metaparameter. If it is only consistency, say that in the commit message.
| @@ -0,0 +1,185 @@ | |||
| """OFFLINE MIRP oracle for the negative coarse grey depth: NGLDM and GLDZM, 2D and 3D, at 20 bins. | |||
There was a problem hiding this comment.
Nit: SPEC §4 puts each oracle generator at tests/vetting/oracles/gen_<family>_<token>.*. This one is named after a setting and covers four families (ngldm, gldzm, ngldm3d, gldzm3d), each of which already has its own MIRP generator that this file imports. Please add the four --negdepth tables to gen_{ngldm,gldzm,ngldm3d,gldzm3d}_mirp.py, so that each family's MIRP pins are produced and checked in one place.
e754bb7 to
9092da6
Compare
Every defect the pages describe now says so and links its issue (PolusAI#489-PolusAI#528), and every "a fix is coming" with an issue behind it is replaced by the link; packaging gaps are called known issues. The worked-example scripts name the issue next to each assertion that pins a defect's current value. New Possible issues entries, with measured values: 3D shape under single_roi=True (the merged mask's bounding box), morphology codes that need companion families, the whole-slide extrema, the whole-slide radial centre, and the 3D GLCM metaparameter getter. The "is coming" lines left describe behaviour that PolusAI#486 changes.
vjaganat90
left a comment
There was a problem hiding this comment.
LGTM. I checked the rewrite against every thread:
- The breaking change is now marked in the commit (
!and aBREAKING CHANGEfooter), the title and the description. parse_grey_depth()covers--coarseGrayDepthand all sixgreydepthmetaparameters.- The background test really streams out of core at
ram_limit=0. bin_roi_3dsizes its cube from the box it checks against.- #484 is linked.
I rebuilt 9092da6 and ran:
- gtest and the whole
pytest tests/python. Everything passed apart from the Arrow tests, since my build has no Arrow. - the four MIRP generators with
--check, which pass. - a trial merge with #485's current head, which passes 1190 gtest cases.
Nits, none blocking:
- The registry rows (thread reply) and the
Nyxus3Ddocstring (inline). - Merge order, #485: this note is stale. #485's
run_3d_feature_on_volumenow setsr.aabbitself (itstests/test_main_nyxus.h:253-255). The merge then only drops this PR's copies intest_3d_glszm_common.handtest_3d_gldm_common.hin favour of the shared runner. - Merge order, #488: this note says #488's
environment.cppcitations are unaffected. The--coarseGrayDepthhunk at line 665 still adds 6 lines, so itsevidenceranges past that point (709-720, 723-731, ... 1040-1054) land 6 lines early. #488's--checkdoesn't notice, because it only checks that a range exists. #488 can re-cite them when it rebases. - History: 8b3233c adds
gen_negative_greydepth_mirp.pyand 9092da6 deletes it. Folding the generator move into 8b3233c would keep a file that lives for one commit out of main's history.
| first-order histogram statistics use abs(value) bins, or 24 at 0. Any 32-bit integer but | ||
| -2**31 is accepted, but the magnitude sizes per-ROI buffers and output columns (HISTOGRAM | ||
| writes one column per bin, 2D GLCM allocates abs(value)**2 cells), so a depth beyond the |
There was a problem hiding this comment.
Nit: This is copied from Nyxus, but Nyxus3D computes neither HISTOGRAM nor 2D GLCM. There is no 3D histogram feature, and the HISTOGRAM column writer keys on Feature2D::HISTOGRAM (output_2_csv.cpp:195). Here the cost is 3D GLCM's matrix, which grows with the square of the depth (3d_glcm.cpp:311). Please name that instead, or drop the examples.
There was a problem hiding this comment.
Fixed The Nyxus3D docstring now names 3D GLCM's matrix, abs(value)**2 cells per direction, as what a large depth costs, and says every 3D texture family reads the depth, in c42d6fa.
…est build Merge main-binning-plumbing (PolusAI#486, tip 3f654e2) into the test-build branch that already carries PolusAI#484 and PolusAI#485, so the wheel also bins a 3D ROI voxel at the volume minimum into level 1. Conflicts resolved: - 3d_ngldm.cpp, 3d_ooc_volume.h: both sides kept. 3D NGLDM takes PolusAI#484's roi_mask_3d() and PolusAI#486's ngldm_roi_grey_level(); the header carries roi_mask_3d() and bin_roi_3d(). - test_all.cc, scan_glrlm3d_coverage.py, the first config_recipes.md hunk: both sides kept. - test_3d_glszm_common.h: PolusAI#485's run_3d_feature_on_volume() call; the helper builds the same voxel cloud and box PolusAI#486 builds inline. - test_3d_ngtdm_regression.h, matrix/ngtdm3d.md, the NGTDM config_recipes.md hunk: PolusAI#484's side, which describes the masked 3D NGTDM this tree runs. - oracle_coverage.csv, report_output.csv, report_output.md: PolusAI#486's side. The registry on this branch is not regenerated and is not authoritative. Test-build branch only; not for merging.
vjaganat90
left a comment
There was a problem hiding this comment.
I re-reviewed 3f654e2. The binning change itself is right; one thing has to change before my approval covers this commit.
What I checked:
roi_voxel_binning()/bin_roi_voxel()match the stated rule under all three schemes, including:- a voxel at the ROI's minimum or maximum;
- a constant ROI;
- n = 1;
- a stored 0.
- In RAM and out of core agree bit for bit for the seven 3D families at depths 64, 0, −8 and −20 on four phantoms, on whole volumes and with anisotropy. 2D output doesn't move.
- The 13 new volume-minimum tests fail on 9092da6 and pass here. The 12 control cases and the full-box case pass both ways, as they should.
- The six generators'
--checkmodes pass with PyRadiomics 3.0.1 and MIRP 2.6.0, andvolmin_phantom.pyrebuilds the three NIfTI files byte for byte. runAllTestsandpytest tests/python/pass, apart from the Arrow tests my build doesn't include.
Blocking
- 3D GLCM's default output changes, and no commit marks it as breaking.
- What moves: at depth 64, 3D GLCM no longer pairs the box's background (read as level 1) with the ROI. So its output moves on every ROI that doesn't fill its box, and on every whole-volume run, because
AABB::init_from_whdmakes the box one voxel larger than the volume on each axis (aabb.h:69-77, withget_width()ataabb.h:50). Run as a whole volume,volmin_intengoes from3GLCM_CONTRAST_AVE943 to 459. - Why the PR body isn't enough: it lists the ROI case under Breaking changes, but main takes a PR's commits as they are and release-please builds the changelog from them. 8b3233c's footer names only 3D GLRLM, GLSZM, GLDM and NGTDM, and 3f654e2 has neither
!nor a footer, so the release notes would show this as a plain fix about the volume minimum. - Fix: please make 3f654e2
fix(binning)!:, with aBREAKING CHANGE:footer that names 3D GLCM at the default depth, ROIs that don't fill their box, and whole-volume runs. Please also add whole volumes to the PR body's bullet.
- What moves: at depth 64, 3D GLCM no longer pairs the box's background (read as level 1) with the ROI. So its output moves on every ROI that doesn't fill its box, and on every whole-volume run, because
Nits
- The commit body says "3D NGTDM only changes where it bins", but its default output moves too: the five ut57 pins in
test_3d_ngtdm_regression.hmove by 1.3–1.6%. NGTDM is already in 8b3233c's footer, so only that sentence needs fixing. - No test pins 3D NGTDM's change under PyRadiomics-style binning. The volume-minimum set leaves NGTDM out, and on that phantom
3NGTDM_BUSYNESSmoves 29% at −20. When this rebases onto #484, please addtest_3d_ngtdm_volmin_pyradiomics, since the merged tree is the one that matches PyRadiomics. - Two more inline.
| if (ibsi_grey_binning(greyInfo)) | ||
| if (lvl_a == 0 || lvl_b == 0) | ||
| continue; | ||
| // skip pairs with a background cell, which holds level 0 under every binning scheme |
There was a problem hiding this comment.
Nit: This comment and the one at 3d_glcm_nontriv.cpp:92 say what the skip does now, but not what was wrong. Under MATLAB-style binning the background used to sit at level 1 and pair with the ROI, and the zero skip ran only with no binning. Please name that on the changed line, e.g. // MATLAB-style binning put the background at level 1 and paired it with the ROI; bin_roi_3d leaves it at 0, skipped under every scheme.
There was a problem hiding this comment.
Diverted The comments state what the code does now, by this repo's convention for changed lines; what was wrong is recorded in c42d6fa's message, which names the level-1 background pairing under MATLAB-style binning.
| @@ -190,14 +190,13 @@ void D3_NGTDM_feature::calculate (LR& r, const Fsettings& s) | |||
| h = r.aux_image_cube.height(), | |||
There was a problem hiding this comment.
Nit: With D.allocate (w, h, d) gone, w, h and d (lines 189-191) and the // grey-bin intensities comment above them are unused. Please delete them.
The coarse grey depth now reaches every texture family, and each of its
three signs (positive MATLAB-style levels, 0 no binning, negative
PyRadiomics-style bins) gives defined values.
- 3D GLRLM, GLSZM, GLDM and NGTDM start their own grey depth at the coarse
grey depth, as 3D GLCM already did, so --coarseGrayDepth / coarse_gray_depth
bins them. Their default output moves from unbinned to 64 levels. A
family's own metaparameter (3glszm/greydepth=...) still overrides it.
- MATLAB-style binning sends intensity 0 to level 1, the level a ROI's
darkest voxels land in too, so 3D GLSZM, GLDM and GLRLM tell background
from the ROI by the voxel cloud: bin_roi_3d() bins only the ROI's voxels
into a zero-filled cube in RAM, sized from the box every voxel is checked
against and refusing a voxel outside it, and the streamed volume gives
the background level 0 out of core. Before, GLSZM and GLDM dropped the
ROI's level-1 voxels and GLRLM counted the background as level-1 runs,
which put 3GLRLM_RP above 1 at positive depths.
- 2D GLCM builds its co-occurrence matrix at glcm/greydepth, the depth its
blank-ROI check already read, instead of the common grey depth.
- The whole-slide 2D GLDZM and the whole-volume 3D GLSZM reduce hand each
family its own settings vector, as does the enclosing-circle reduce. The
whole-volume 3D GLSZM had been ignoring 3glszm/greydepth. The 2D GLDZM
and enclosing-circle swaps are consistency only: neither family, nor the
vector each read before, has a metaparameter, so the values they read
are the same.
- PyRadiomics-style bins are IBSI's fixed bin number,
floor(n (x - min) / (max - min)) + 1, and a ROI of one intensity is one
bin. Dividing by the bin width put a level on a bin edge one bin low
wherever that width is inexact, and a zero-width range divided 0 by 0.
The unused copy in 3d_intensity.cpp is removed.
- 2D and 3D NGLDM bin through the shared sign dispatch: 0 keeps the raw
levels (before, every pixel was put at level 0), and a negative depth
takes PyRadiomics-style bins (before, the count wrapped to about 4.3e9
levels). The positive scheme is unchanged.
- 2D GLDZM skips background pixels under PyRadiomics-style binning as it
already did with no binning. Before, the background formed zones of a
level missing from the matrix and every feature came back 0.
- The intensity histograms take abs(depth) bins, and 24 at depth 0, which
used to write one element before the buffer. The HISTOGRAM output columns
use the same count, so a negative depth no longer misaligns them.
- Python's coarse_gray_depth accepts any 32-bit integer but -2**31 and
documents what its sign selects and what a large magnitude costs; the
binding is signed; ImageQuality accepts the depth but no image-quality
feature reads it. set_params() and set_environment_params() accept only
the object's current depth, since they do not rebuild the per-family
settings; the README and docs example set gabor_f0 instead.
- One parser, Nyxus::parse_grey_depth(), reads --coarseGrayDepth and every
<family>/greydepth metaparameter (glcm, 3glcm, 3gldm, 3ngtdm, 3glrlm,
3glszm): text that is no integer is refused as such, and INT_MIN or a
value past the int range as out of range. The metaparameters went
through parse_as_int(), which narrows a long to int unchecked, so
3glszm/greydepth=-2147483648 reached std::abs() as UB and
3glrlm/greydepth=4294967276 became -20.
Tests, each failing without the fix (the 3D GLDZM pins are new coverage of
a path that was already right):
- the coarse grey depth reaches every family at -20, 0 and 20, and a
family's metaparameter overrides it (feature settings mechanics)
- 2D GLCM at glcm/greydepth=4 equals the run with both depths at 4
- 2D and 3D NGLDM at depth 0 equal their IBSI runs
- 2D NGLDM, 2D GLDZM (six features), 3D NGLDM and 3D GLDZM at depth -20
against MIRP fixed_bin_number 20; 14 of the 2D fixture's 144 pixels sit
on a bin edge, and the 5 at 181 are the ones a division by the bin
width sends one bin low
- 3D GLSZM, GLDM and GLRLM at the default depth equal the same ROI
pre-binned with no binning, in RAM and out of core (ram_limit=0, which
every footprint reaches), on a ROI with level-1 voxels and one with
background in its box
- 3D GLRLM's RP, SRE, RLNN, GLNN and LRE hold their bounds on all 13
angles at -100, -20, 0, 32, 64, 128 and 256; the grey64 regression table
is regenerated
- the shared binning: a zero-width range is one bin, bin_roi_3d places
voxels at their box offsets, takes the box's shape whatever
aux_image_cube holds, and refuses a voxel outside the box
- the CLI accepts depths at every sign and refuses the rest with the
reason; the parser's own range edges; all six greydepth metaparameters
refuse -2147483648, 4294967276, 2147483648 and "abc" and keep their depth
- Python: every sign accepted and non-integers refused, set_params refuses
a different depth and round-trips get_params(), the flag equals the four
3D metaparameters, first-order statistics at 0 equal 24 bins, HISTOGRAM
columns at -20, whole-volume 3D GLSZM equals segmented under
3glszm/greydepth, and glcm/greydepth joins the out-of-core GLCM invariant
Vetting: each family's MIRP generator (gen_{ngldm,gldzm,ngldm3d,gldzm3d}
_mirp.py) prints and checks its negative-depth table, the 2D fixture's
builder shared from oracles/square12_graded.py; the registry records the
54 negative-depth rows, the no-binning point is described as such rather
than as the default, and report_output.csv/.md are regenerated.
BREAKING CHANGE: 3D GLRLM, GLSZM, GLDM and NGTDM now bin at the coarse
grey depth, so their default output moves from unbinned raw levels to 64
MATLAB-style levels. Pass --coarseGrayDepth=0 (coarse_gray_depth=0 in
Python), or set 3<family>/greydepth=0 for one family, to get the previous
unbinned output. Python's set_params() and set_environment_params() now
raise ValueError for a coarse_gray_depth other than the object's current
one; pass a different depth to the constructor.
Nyxus stores a volume offset by its lowest intensity, so a ROI that holds the volume minimum has voxels stored as 0, the value the background cells of its bounding box hold. to_grayscale_radiomix() keeps 0 at 0, so under PyRadiomics-style binning those ROI voxels went to level 0 instead of floor(n (x - min) / (max - min)) + 1 = 1. 3D GLCM, GLDM, GLRLM and GLSZM then read them as background and dropped them; 3D GLDZM lifted every level by one to make room for them; 3D NGLDM gave them level 0; 3D NGTDM lifted every level by one and so carried an extra grey level. On a CT whose ROI holds the volume minimum (140 of 10.5M voxels at -1024 HU) at binCount 20, 3NGTDM_BUSYNESS was 32% off PyRadiomics and 3NGTDM_CONTRAST 9.5%; GLRLM, GLDM and GLCM by up to 4.5e-4, 1.7e-4 and 4.5e-5. The fix fills each family's binned cube from the voxel cloud with one rule: Nyxus::roi_voxel_binning() / TextureFeature::bin_roi_voxel() bins a voxel known to be the ROI's, so PyRadiomics-style binning takes IBSI's fixed bin number with no special case for 0 and MATLAB-style binning its level 1..n. Every ROI voxel gets a level of 1 or more, so level 0 marks the cells that are not the ROI's again, and the families' existing zeroI / U.erase(0) tests read it as background without a mask of their own. - bin_roi_3d() (in RAM) and every OocBinnedVolume (out of core) bin through that rule, for 3D GLCM, GLDM, GLRLM, GLSZM, GLDZM and NGTDM; their out-of-core background is level 0 in every scheme. - 3D NGLDM keeps its own positive scheme and takes the same fixed bin number under PyRadiomics-style binning (ngldm_roi_grey_level()); 2D NGLDM still calls ngldm_grey_level(). - 3D GLCM builds its matrix from bin_roi_3d() and skips pairs with a level-0 cell under every scheme. Under MATLAB-style binning it had read the background as level 1 and paired it with the ROI, so its default output moves on every ROI that does not fill its bounding box, and on every whole-volume run, whose box is one voxel larger than the volume on each axis. This is the 3D GLCM part of the background-as-level-1 issue; 3D GLRLM, GLSZM and GLDM were already masked. - 3D GLDZM lifts its levels only with no binning, the one scheme that still hands it a ROI voxel at level 0. - 3D NGTDM bins through the same rule but still takes the whole bounding box as its levels and neighbourhoods; the ROI mask of 3D NGTDM comes from its own fix. Its background now sits at level 0, which its zero-lift moves to 1 together with every ROI level, so its default output moves on every ROI that does not fill its box (the five regression pins on the segmented phantom by 1.3-1.6%). Merged with the mask, PyRadiomics agrees to ~1e-15 on the CT above. Unchanged: with no binning (greydepth 0 / IBSI) a ROI voxel of raw stored value 0 is still indistinguishable from background and dropped by 3D GLCM, GLDM, GLRLM and GLSZM. 2D texture families still bin a stored 0 to 0 under PyRadiomics-style binning (measured: on a slice whose ROI holds the image minimum all seven 2D families move). Both stay open on the board. Tests, each failing on the previous commit: - bench_volmin_3d (tests/vetting/oracles/volmin_phantom.py): an int16 CT-like volume whose ellipsoid ROI holds the volume minimum and leaves background in its box; its control moves the minimum outside the box. 3D GLCM, GLDM, GLRLM and GLSZM against PyRadiomics 3.0.1 at binCount 8, 3D GLDZM and NGLDM against MIRP fixed_bin_number 8, rel 1e-9, on both phantoms, in RAM and out of core (24 cases; the 12 volmin cases fail before, the 12 control cases pass before and after). Logarithmic features keep the fast_log10 residual and are left out. - 3D GLCM at the default depth 64 against its co-occurrence definition (ASM, CONTRAST) on an ellipsoid with background in its box, in RAM and out of core; a ROI that fills its box is the negative case. - bin_roi_3d() puts a ROI voxel stored as 0 at level 1 under both binning schemes, keeps it at 0 with no binning, and leaves background at 0. - 3D GLDZM at bin count 5 on its zero-level fixture now expects levels 1 and 5, IBSI's fixed bin number; the previous expectation of 1 and 6 was the lift. - The 3D GLCM grey64 regression table is regenerated (background no longer pairs; a dump helper regenerates it), and the 3D NGTDM regression pins move with the background level. - The registry records the new recipes, the fixture and the rows. - The Nyxus3D docstring names 3D GLCM's matrix as what a large grey depth costs. BREAKING CHANGE: at the default grey depth (64, MATLAB-style binning) 3D GLCM no longer pairs the bounding box's background with the ROI, so its output changes on every ROI that does not fill its bounding box and on every whole-volume run. 3D NGTDM's default output changes on the same ROIs, whose grey levels it now lifts by one. Under PyRadiomics-style binning (negative depth) every 3D texture family changes on a ROI that holds the volume's lowest intensity, whose voxels there now take bin 1.
3f654e2 to
c42d6fa
Compare
|
Fixed The level-0 commit is now |
|
Fixed The #485 merge note now says #485's runner sets the box itself, so the merge drops this PR's set-up in |
97f688e to
8890f51
Compare
Summary
The coarse grey depth (
--coarseGrayDepth/coarse_gray_depth) chooses the texture binning by its sign:abs()PyRadiomics-style bins over the ROI's range.Before this PR it reached only some families, and every sign other than the positive one broke something:
glcm/greydepthglszm/greydepthHDHGLE1.6e19)HISTOGRAM_BIN_*header had 0 columns while the values hadabs(n)Breaking changes
Both commits carry
!and aBREAKING CHANGE:footer, so release-please treats them as breaking.--coarseGrayDepth=0(coarse_gray_depth=0in Python) gives the previous unbinned output, and3<family>/greydepth=0does so for one family.set_params()andset_environment_params()now raiseValueErrorfor acoarse_gray_depthother than the object's current one. Pass a different depth to the constructor.Merge order
3d_ngtdm.cppcombines cleanly (grey levels frombin_roi_3d, membership from fix(3d ngtdm): confine the neighbourhood to the ROI #484's mask) and 3D NGTDM matches PyRadiomics to ~1e-15 on a CT whose ROI holds the volume minimum, where fix(3d ngtdm): confine the neighbourhood to the ROI #484 alone is 32% off. Measured conflicts:3d_ooc_volume.h(both PRs add a helper at the same place; keep both),3d_ngldm.cpp(both change NGLDM's grey-level set; keep fix(3d ngtdm): confine the neighbourhood to the ROI #484'sroi_mask_3d()and this PR'sngldm_roi_grey_level()),tests/test_3d_ngtdm_regression.h(both regenerate the same five pins; take fix(3d ngtdm): confine the neighbourhood to the ROI #484's, which hold on the merged tree), and the regeneratedreport_output.csv/.md.run_3d_feature_on_volumeintests/test_main_nyxus.hsets the ROI'sAABBitself, so the merge drops this PR's box set-up intest_3d_glszm_common.handtest_3d_gldm_common.hin favour of that runner and regenerates the report.set_params(coarse_gray_depth=256), the 3D GLRLMRPpins, the GLSZM/GLDM "default equals IBSI" assertions, and the--coarseGrayDepth, binning and 3D-default text. The second PR to land owns the update. The--coarseGrayDepthhunk inenvironment.cppadds 6 lines, so docs: bring the documentation in line with the code #488's line-number citations into that file past it land 6 lines early and need re-citing when it rebases.Change
What moves for existing users: see Breaking changes above. A family's own
3<family>/greydepthmetaparameter still overrides the coarse depth.MATLAB-style binning sends intensity 0 to level 1, which is also where a ROI's darkest voxels land, so the binned value can't tell background from ROI. Under it, 3D GLSZM and GLDM used to drop the ROI's level-1 voxels as background, and 3D GLRLM counted the bounding box's background as level-1 runs. 3D GLRLM's run percentage
3GLRLM_RPexceeded 1 for that reason (4 of 13 angles at +64). Those three families now take the background from the ROI's voxel cloud and give it level 0, in RAM and out of core. 3D NGTDM gets the same fix in #484.set_params()andset_environment_params()now accept only the object's currentcoarse_gray_depth; pass a different one to the constructor.Level 0 is the background, and only the background. Nyxus stores a volume offset by its lowest intensity, so a ROI that holds the volume minimum has voxels stored as 0, the value of the bounding box's background cells. Every 3D texture family now bins its ROI voxels through one rule,
Nyxus::roi_voxel_binning()/TextureFeature::bin_roi_voxel(), in RAM (bin_roi_3d) and out of core (OocBinnedVolume): PyRadiomics-style binning takes IBSI's fixed bin number with no special case for 0, MATLAB-style binning its level 1..n. Every ROI voxel gets a level of 1 or more, so the families' existing level-0 tests read only the background. Before, such a voxel went to level 0: 3D GLCM, GLDM, GLRLM and GLSZM dropped it, 3D GLDZM lifted every level by one, 3D NGLDM kept it at 0, and 3D NGTDM lifted every level and carried an extra one (32% off PyRadiomics onBUSYNESSon a CT with 140 such voxels in 10.5M). With no binning a ROI voxel of raw stored value 0 still has level 0; see Not in this PR.env_features.cppGLCM_GREYDEPTHglcm.cppreduce_trivial_rois.cppfloor(n (x − min) / (max − min)) + 1, and a ROI of one intensity is one bintexture_feature.h; an unused copy in3d_intensity.cppis removedngldm.*,3d_ngldm.*gldzm.cppabs(depth)bins, 24 at 0, and so do theHISTOGRAM_BIN_*writersconstants.h, intensity / output writerscoarse_gray_depthaccepts any 32-bit integer but-2**31; the docstrings say what the sign selects and what a large magnitude costs (HISTOGRAM columns, 2D GLCM's depth² cells);ImageQuality's says no image-quality feature reads it; the binding is signednyxus.py,new_bindings_py.cppparse_grey_depth(), for--coarseGrayDepthand every<family>/greydepthmetaparameter: it refuses non-integers, and refusesINT_MINand values past the int range as out of range. The CLI help text describes the signhelpers.h,environment.cpp,env_metaparams.cppbin_roi_3d, sized from the box every voxel is checked against, refusing a voxel outside it), and give the background level 0 out of core3d_glszm.cpp,3d_gldm.cpp,3d_glrlm.cpp,3d_glrlm_nontriv.cpp,3d_ooc_volume.hset_params()/set_environment_params()accept only the currentcoarse_gray_depth; the setter binding drops itnyxus.py,new_bindings_py.cpp, README andcmdline_and_examples.rstexamplestexture_feature.h,3d_ooc_volume.h,3d_*.cppbin_roi_3dand skips pairs with a level-0 cell under every scheme3d_glcm.cpp,3d_glcm_nontriv.cpp3d_gldzm.cppTests
Every new test except one fails on
mainwith this PR's tests and passes with the fix. The exception is the 3D GLDZM pins, which are new coverage: 3D GLDZM was already right at a negative depth.C++:
TEST_FEATURE_SETTINGS_COARSE_GRAY_DEPTH_REACHES_EVERY_TEXTURE_FAMILY_MECHANICS: the depth reaches all 14 texture vectors at −20, 0 and 20.TEST_FEATURE_SETTINGS_FAMILY_GREYDEPTH_OVERRIDES_COARSE_MECHANICS(negative): a family's metaparameter overrides it for that family only.TEST_2D_GLCM_OWN_GREYDEPTH_BINS_THE_MATRIX_MECHANICS: 2D GLCM's matrix followsglcm/greydepth.TEST_2D_NGLDM_ZERO_GREYDEPTH_EQUALS_IBSI_MECHANICSand the 3D twin: at depth 0, NGLDM equals its IBSI run.TEST_{2D_NGLDM,2D_GLDZM,3D_NGLDM,3D_GLDZM}_NEGATIVE_GREYDEPTH_MIRP: 54 pins against MIRPfixed_bin_number20, which bins for itself, so the binning is compared too. Worst residual 6.2e-16. 14 of the 2D fixture's 144 pixels sit exactly on a bin edge, and the 5 at 181 are the ones a division by the bin width sends one bin low; the 3D phantom's bin width, 100, is exact, so its 2736 edge voxels cannot separate the two formulas. Each table is produced and checked by its family's MIRP generator. 2D GLDZM pins only its six features that read no border distance; on a ROI that fills its bounding box, border distance can't tell the 2D distance map from IBSI's.TEST_3D_GLSZM_DEFAULT_GREYDEPTH_MECHANICSnow asserts the default is 64.TEST_TEXTURE_BINNING_ZERO_WIDTH_RANGE_MECHANICS: a ROI of one intensity bins to 1 at a negative depth instead of computing 0/0 (3D GLCM and NGTDM reach it with no constant-ROI guard). Negative test: a nonzero range still bins by the fixed bin number.TEST_TEXTURE_BINNING_ROI_CUBE_{PLACES_VOXELS,REFUSES_VOXEL_OUTSIDE_BOX}_MECHANICS: the shared 3D binning writes each voxel at its offset from a box corner away from the origin, leaves the rest at 0, and refuses a voxel one past the box. The cube takes the box's shape even whenaux_image_cubeholds a different one.TEST_FEATURE_SETTINGS_COARSE_GRAY_DEPTH_CLI_{ACCEPTED,REFUSED}_MECHANICS:--coarseGrayDepthtakes −256, −20, 0, 64 and 4096. It refuses64abc,abcand1.5as no integer, andINT_MIN,2147483648and ±99999999999 as out of range, each with its own message.TEST_FEATURE_SETTINGS_GREYDEPTH_METAPARAM_REFUSED_MECHANICS: all six<family>/greydepthmetaparameters refuse-2147483648,4294967276(−20 once wrapped) and2147483648as out of range andabcas no integer, leaving the depth unchanged; −20 and 32 are still taken.TEST_FEATURE_SETTINGS_GREY_DEPTH_PARSE_RANGE_MECHANICS: the parser's own edges. ±(2³¹−1) parse; that is a type check, not a claim that a depth that large fits in memory.TEST_3D_GLRLM_BOUNDED_FEATURES_INVARIANT:RP,SRE,RLNN,GLNNin [0,1] andLRE≥ 1 on all 13 angles of theut_phantom at −100, −20, 0, 32, 64, 128 and 256.TEST_TEXTURE_BINNING_ROI_CUBE_VOXEL_AT_ZERO_MECHANICS:bin_roi_3dputs a ROI voxel stored as 0 at level 1 under both binning schemes, keeps it at 0 with no binning, and leaves background at 0.TEST_3D_GLDZM_ZERO_LEVEL_VOXELS_ARE_ZONED_RADIOMICS_MECHANICSexpects levels 1 and 5 at bin count 5, IBSI's fixed bin number, where the lift gave 1 and 6.TEST_3D_GLCM_GREY64_DUMP_REGRESSIONregenerates it), and the five 3D NGTDM regression pins move with the background level until fix(3d ngtdm): confine the neighbourhood to the ROI #484's mask lands.The volume-minimum phantom (
bench_volmin_3d,tests/vetting/oracles/volmin_phantom.py): an int16 CT-like volume whose ellipsoid ROI holds the volume minimum on 31 of 560 voxels and leaves background in its box; its control moves the minimum outside the box and leaves the ROI unchanged, so one oracle table pins both.test_3d_{glcm,gldm,glrlm,glszm}_volmin_pyradiomics: PyRadiomics 3.0.1 atbinCount=8;test_3d_{gldzm,ngldm}_volmin_mirp: MIRPfixed_bin_number8. Each on both phantoms, in RAM and out of core, rel 1e-9, 24 cases. The 12 phantom cases fail on the previous commit (GLDMLGLE30% off, GLRLMLGLRE29%, GLSZMLALGLE16%, GLDZM 33%, NGLDM 34%, GLCM up to 6.5×); the 12 control cases pass before and after. The logarithmic features keep thefast_log10residual and are pinned elsewhere.test_3d_glcm_matlab_background_analytic: at the default depth 64,3GLCM_ASM_AVEand3GLCM_CONTRAST_AVEagainst the co-occurrence matrix built from the definition, on an ellipsoid with background in its box, in RAM and out of core (before:ASM_AVE0.089 against 0.011). Negative case: a ROI that fills its box.Python (
test_coarse_gray_depth_mechanics.py,test_3d_glszm_mechanics.py,test_3d_texture_background_mechanics.py):ram_limit=0, which every footprint reaches on every platform). One ROI fills its box and has level-1 voxels; the other leaves background in its box. On the old code GLSZM differs on 12/16 features, GLDM on 10/14 and GLRLM on 32/32.set_params(coarse_gray_depth=...)andset_environment_params(...)raise for a depth other than the object's (−20, 0, 32) on all three classes, and apply nothing else from the call. Negative tests: the current depth is accepted beside parameters that still apply, andset_params(**get_params())round-trips.Noneand ±2³¹ are refused (negative).HISTOGRAMat −20 has 20 aligned columns.MEANis requested beside it becauseHISTOGRAMon its own returns zeros, a separate bug tracked as PN-165.3glszm/greydepth.glcm/greydepthjoins the out-of-core GLCM invariant.Verification
pytest tests/python: all pass. The 7 Arrow tests need an Arrow build.tests/python171 passed, 0 diagnostics, sources checksum-verified before and after. That run covers the depth-0 first-order path that used to write before its buffer, and the background masks in RAM and out of core.gen_{ngldm,gldzm,ngldm3d,gldzm3d}_mirp.py --checkpass.coarse_gray_depth=-20against PyRadiomicsbinCount=20, through the flag alone (before this PR only the per-family metaparameters reached these families): 3D GLRLM 16/16 and 3D GLDM 14/14 match to ≤1e-15. 3D GLSZM matches 14/16; itsSAEandLAEneed the zone-size overflow fix inmain-texture-int-overflow, with which they match too. The log-basedRE/ZE/DEare within the knownfast_log10residual (~4e-4).Not in this PR
set_params()doesn't recompile the per-family settings, so it accepts only the currentcoarse_gray_depthuntil it does.neighbor_distance,ibsiand the float-image range have the same limitation. All of this is tracked separately.README.md,cmdline_and_examples.rst) still give--coarseGrayDeptha default of 256 and no sign semantics. They are rewritten from the CLI reference in the docs update.