Skip to content

Refactor big-data processing trait, add chunkById(), remove Fiber variants - #343

Merged
techmahedy merged 1 commit into
doppar:4.xfrom
techmahedy:techmahedy-4.x
Oct 1, 2026
Merged

techmahedy merged 1 commit into
doppar:4.xfrom
techmahedy:techmahedy-4.x

Conversation

@techmahedy

@techmahedy techmahedy commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Pull Request Checklist

Q A
Branch? 4.x
Bug fix? yes
New feature? Yes
Deprecations? Yes
Issues -
License MIT

Summary

Fixes data-loss and performance problems in InteractsWithBigDataProcessing (chunk, cursor, stream, batch), adds chunkById(), and removes fchunk(), fstream() and fcursor().

Bugs fixed

  • fstream() dropped the first item of every chunk and yielded null after each chunk.
  • fcursor() never passed the first $bufferSize rows to the processor and raised a foreach(null) warning at the end.
  • chunk(), stream() and batch() used OFFSET with no ORDER BY, so pages could overlap or skip rows. They now order by the primary key when the query has no ordering and no GROUP BY.
  • An existing limit() or offset() on the query was overwritten, so ->limit(100)->chunk(10, ...) scanned the whole table. Both are now respected.
  • $processed in chunk() started at $chunkSize, even when the first page was smaller. It is now the running count including the current chunk.
  • A chunk size of 0 or less looped forever. It now throws InvalidArgumentException.
  • cursor() rethrew database errors without the original PDOException. It now keeps it as the previous exception.

Performance

  • Iteration stops on a short page, which saves one empty query at the end of every run.
  • cursor() no longer calls gc_collect_cycles() on every row.
  • cursor() now hydrates rows through fetchLazy(), so it matches get() (connection name, encrypted attributes).

New

  • chunkById($chunkSize, $processor, $column = null, $total = null) pages with WHERE id > :last ORDER BY id. It stays fast on very large tables, and it is safe when the processor updates or deletes rows that match the WHERE clause. OFFSET paging skips rows in that case.
  • cursor(..., bool $unbuffered = false) streams rows from the server on MySQL. By default the MySQL PDO driver loads the whole result set into client memory. It is opt-in because an unbuffered connection cannot run other queries while the cursor is open.

Removed (breaking)

fchunk(), fstream() and fcursor() are removed. Fibers are cooperative and everything ran on one thread and one PDO connection, so these methods never ran in parallel, and the buffering gave no backpressure. Migration:

Removed Use instead
fchunk($size, $processor, $concurrency) chunk($size, $processor)
fstream($size, $transform, $bufferSize) stream($size, $transform)
fcursor($processor, $bufferSize) cursor($processor)

Behavior changes to note

  • chunk() and stream() may now add ORDER BY <primary key> to the generated SQL.
  • chunk(), chunkById(), stream() and batch() throw InvalidArgumentException for a chunk size below 1.
  • cursor() processor index is the 1-based row number.

Docs

entity-orm.md in the docs repo is updated: chunkById(), buffered vs. unbuffered cursor(), the removal table above, and the behavior changes. That change is in a separate repo and needs its own PR.

Checklist

  • Tests have been added or updated
  • Documentation has been updated if necessary
  • Code follows the project coding standards
  • All tests pass locally

@techmahedy techmahedy self-assigned this Oct 1, 2026
@techmahedy techmahedy added bug Something isn't working enhancement New feature or request feat new feature labels Oct 1, 2026
@techmahedy
techmahedy merged commit 9c97c7b into doppar:4.x Oct 1, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request feat new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant