Skip to content

fix(ooc): take the out-of-core tile scan's column from the tile index modulo - #529

Open
darkclad wants to merge 8 commits into
PolusAI:mainfrom
darkclad:main-ooc-tile-column
Open

darkclad wants to merge 8 commits into
PolusAI:mainfrom
darkclad:main-ooc-tile-column

Conversation

@darkclad

@darkclad darkclad commented Oct 20, 2026 •

Copy link
Copy Markdown

Supersedes #487. It carries that PR's first three commits unchanged and addresses its review: the load_tile() commit is reworded, the trailing close() commit is dropped, and the contour fix and the test the review asked for are added.

What changes

The out-of-core 2D pass streams an oversized ROI's pixels tile by tile and derives each pixel's slide coordinates from the tile index. It computed the tile's column as tileIdx / num_tiles_hor, the same expression as the tile's row, so a pixel's x was right only in a tile on the grid's diagonal, and the ROI's pixel cloud had wrong x coordinates anywhere else. This is the defect #483 mentions; with #483 it surfaces as an error instead of wrong values.

Once pixels reach the right coordinates, a ROI that spans tiles reaches the out-of-core contour, which made its result depend on how the file is tiled. That is fixed here too, since this PR is what makes it reachable.

One commit per change:

  1. The column is tileIdx % num_tiles_hor, matching how the image loaders decompose a tile index (phase3.cpp).
  2. Rows are bounded by the vertical tile count and columns by the horizontal one, and a tile is indexed row * ntHor + col, as in the other tile scans. The old bounds and stride were transposed but cancelled out, so no tile was skipped.
  3. The inner row/col no longer shadow the loop variables.
  4. load_tile()'s status is checked, as in the other tile scans. This reports nothing today: load_tile(size_t) refuses only an index outside the grid, which this scan never builds, and a tile the TIFF reader fails on arrives zero-filled with a true status, so its pixels are still dropped without a message.
  5. The scan places pixels by the loop's own row and col instead of decomposing the index per pixel, reads the tile size once per ROI, and uses size_t tile indices and StatsInt pixel coordinates, the type Pixel2 stores.
  6. buildRegularContour_nontriv allocates borderImage where it called paddedImage.allocate() a second time (contour.cpp). allocate() writes from the file's current position, which set_at() leaves just after the last pixel written, so every slot after that pixel was zeroed before the trace. Pixels streamed in raster order lose nothing; pixels streamed tile by tile lose those of an earlier tile that lie lower in the box, so a ROI spanning tiles got a contour, PERIMETER and EDGE_* values that depended on the tiling.
  7. A test, below.

Tests

test_2d_ooc_2d_tiled_matches_untiled_invariant (tests/python/test_2d_ooc_invariant.py) featurizes the whole 2D feature surface out-of-core on a tiled TIFF and on a strip copy of the same arrays, and requires every column of both ROIs to agree. Both sides run out-of-core, so the comparison isolates the tile scan from the out-of-core vs in-RAM differences in #509.

  • The image is 160 x 96 at 32 x 32 tiles, 5 across and 3 down, so the two tile counts differ.

  • Label 1 lies inside the single tile at grid row 0, column 3. With change 1 reverted the run stops with the fix(roi buffers): refuse a pixel outside the box that sizes its buffer #483 "lies outside the bounding box" error.

  • Label 2 spans tiles: a column reaching down to y 60 and an arm to its right ending at y 50, so the arm's last pixel is the last one streamed. With change 6 reverted, PERIMETER, EDGE_*, CIRCULARITY and the enclosing-circle diameter of label 2 diverge (PERIMETER 259 vs 278).

  • With change 2 reverted the test fails too: since change 5 places pixels by the loop's row and col, the old bounds and stride put label 1 outside its box on the 5 x 3 grid.

Change 4 has no reachable failure to test.

The full gtest suite passes, pytest tests/python/ passes apart from the 7 Arrow tests a build without Arrow cannot run, and an ASan+UBSan run of both is clean.

sameeul and others added 7 commits October 14, 2026 16:10
… modulo, not its quotient

The scan computed a tile's column as tileIdx / tiles_hor, the same expression as its row, so a
pixel's x was right only in a tile on the grid's diagonal.
… count and index a tile by its row stride

The scan bounded rows by the horizontal count and columns by the vertical one, and indexed a tile as
row * tiles_vert + col, so its walk of the grid was the transpose of every other tile scan's. It still
visited every tile once, but only because the two mistakes cancel.
…shadowing its loop variables

The inner row and col hid the loop's own, which is how the column's wrong operator went unseen.
…n, as the other tile scans do

The scan discarded the status. load_tile(size_t) refuses only an index outside the tile grid, which
this scan never builds, so the check reports nothing today; it is there for parity with the other tile
scans. A tile the TIFF reader fails on still arrives zero-filled with a true status, so its pixels are
still dropped without a message.
… row and column

With the tile index row * ntHor + col, its quotient and remainder by ntHor are the loop's own row and
col, so the scan uses those instead of decomposing the index again for every pixel, and reads the tile
size once per ROI. The tile indices are size_t, the type of the tile counts and of load_tile()'s
argument, and the pixel coordinates are StatsInt, the type Pixel2 stores.
… of re-allocating its padded image

buildRegularContour_nontriv called allocate() on paddedImage a second time where borderImage was
meant. allocate() writes from the file's current position, which set_at() leaves just after the last
pixel written, so every slot after that pixel was zeroed before the trace. Pixels streamed in raster
order lose nothing, because the last one streamed is also the last in the ROI's box. Pixels streamed
tile by tile lose those of an earlier tile that lie lower in the box, so a ROI spanning tiles of a
tiled TIFF got a contour, PERIMETER and EDGE_* values that depended on the tiling.
…e of its untiled copy

Every out-of-core fixture was a strip TIFF, so no test put a ROI in a tile off the grid's diagonal or
streamed a ROI tile by tile. The new invariant runs the whole 2D feature surface out-of-core on a
160 x 96 image at 32 x 32 tiles (5 x 3, so the two tile counts differ) and on a strip copy of the same
arrays, and requires every column of both ROIs to agree. Label 1 lies inside one off-diagonal tile and
fails with the tile's column taken from the index quotient; label 2 spans tiles with a column reaching
below its arm and fails with the padded image re-allocated in the out-of-core contour.

@vjaganat90 vjaganat90 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. This settles everything I raised on #487:

  • The contour line is fixed, with its defect comment.
  • The load_tile commit no longer claims a fix.
  • The redundant close() is gone.
  • The new invariant compares a tiled file with an untiled copy on a grid that isn't square.

I built the head and ran mutations against test_2d_ooc_2d_tiled_matches_untiled_invariant. It fails on each of them:

  • Column taken from the row (the original bug): label 1 stops with the #483 bounding-box error.
  • paddedImage.allocate restored at contour.cpp:710: label 2 diverges. PERIMETER is 259 vs 278, and EDGE_*, CIRCULARITY, FRACT_DIM_PERIMETER and the enclosing-circle diameter differ too.
  • The old transposed bounds and stride.
  • The wrong stride alone.

test_2d_ooc_invariant.py passes, and so does the full pytest tests/python/, apart from the Arrow tests my build doesn't include.

Nits

  • The description says change 2 can't be pinned by a test. Since 5fcd5b5 places pixels by the loop's row and col, it can: on the 5 × 3 grid the old bounds and stride put label 1 outside its box, and this test fails. Please drop that sentence.
  • 3299303 is an empty ci: commit. Main lands PRs by rebase, so it would reach main as is. Please drop it before merge.

@darkclad
darkclad force-pushed the main-ooc-tile-column branch from 3299303 to e2895e9 Compare October 21, 2026 15:09
@darkclad

Copy link
Copy Markdown
Author

Fixed The description no longer says change 2 can't be pinned; it now states the 5 × 3 grid test fails with change 2 reverted. The empty ci: commit is dropped, so the branch ends at e2895e9.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants