fix(low-code): stop `OffsetIncrement`/`PageIncrement` re-reading pages by bazarnov · Pull Request #1202 · airbytehq/airbyte-python-cdk

Baz (bazarnov) · 2026-10-06T07:59:00Z

Overview

👉 TL;DR: Paginating by offset or by page number no longer reads every page a second time just to count its records, and no longer stops after the first page when the response is streamed. The count still covers the records a filter drops, so filtered `SimpleRetriever` streams declaring `OffsetIncrement` or `PageIncrement` paginate exactly as before. Specifically, `RecordSelector` counts the records its extractor yields and stores the count on the response, and `OffsetIncrement` and `PageIncrement` read it through the new `extracted_record_count` instead of the extractor copies `_create_record_counting_extractor` built, which is removed.

Changes

RecordSelector.select_records counts every item its extractor yields, before the record filter and including the empty records a CombinedExtractor drops, and stores the count on the response once the page has been read. If a custom record filter stops reading early, the selector reads the rest of the page to count it. extracted_record_count(response, default) reads the count back.
OffsetIncrement and PageIncrement use that count and fall back to last_page_size when no RecordSelector read the response. An extractor passed to either constructor still counts the page, and OffsetIncrement.extractor now defaults to None.
OffsetIncrement decodes the response only to evaluate a string page_size, which after the factory's digit-string conversion means a template. An integer or absent page size no longer parses the page again.
ModelToComponentFactory no longer builds extractor copies: _create_record_counting_extractor is removed and create_page_increment is a @staticmethod again. create_record_selector and the AsyncRetriever download retriever set count_dropped_empty_records on the selector's own extractor instead. The union rejection for OffsetIncrement is kept.
CombinedExtractor always logs the empty records it drops, since a single extractor now reads the page, and its docstrings describe the new counting.
Tests:
- Two factory tests read real pages through a SimpleRetriever: one extraction per page with a record filter, and pagination over a gzip-streamed body.
- Unit tests cover the selector count, also behind a filter that stops reading the page early, and both strategies.
- The tests that pinned the copy now read the page through the record selector, and the CombinedExtractor tests test_the_counting_copy_* are renamed test_a_drop_counting_extractor_*.

Review Spotlight

Reviewers with limited time, please review first:

record_selector.py#L103-L120: the count is taken during the only extraction and kept on the response, because one retriever reads all partitions of a stream concurrently; the page is read to its end even when a record filter stops early
record_selector.py#L24-L35: extracted_record_count; the isinstance check keeps unread responses and mocks on the fallback
offset_increment.py#L83-L104: count precedence and the decode skip
model_to_component_factory.py#L3319-L3330: dropped-record counting moved to the record selector's extractor, called from create_record_selector and the download retriever

Risks

Streamed responses now paginate. With a streamed decoder (a GzipDecoder over JSON outside the Connector Builder is the one pagination accepts), the copy got an empty document from the pagination decoder, counted 0 records and stopped after the first page, silently. Every page is now read. 0 monorepo connectors combine a streamed decoder with these paginators: after resolving $refs, the only non-JSON decoder on such retrievers is the buffered XmlDecoder of source-workflowmax.
Custom components now get the pre-filter count. main built no counting copy for custom retrievers or custom pagination strategies, so they used last_page_size (records emitted after filtering), and a record filter that shortened a full page ended their pagination there. A custom retriever reading through RecordSelector.select_records, and a subclass of OffsetIncrement or PageIncrement calling super().next_page_token, now get the pre-filter count and read the following pages. A CustomPaginationStrategy subclassing OffsetIncrement also builds now that extractor has a default; on main the factory raised a TypeError. 0 monorepo connectors are affected: no custom retriever holds these paginators, and the PageIncrement subclasses in source-mixpanel and source-monday override next_page_token without calling super.
No count, previous fallback. A custom retriever that bypasses RecordSelector.select_records leaves no count, so the strategy uses last_page_size, as it did on main.
count_dropped_empty_records on every record selector. The factory now sets it on any CombinedExtractor of a record selector, not only on the paginator copy. It only matters with skip_empty_records, which no monorepo manifest sets, and RecordSelector discards the placeholders, so the emitted records do not change.
Python API. extracted_record_count is new and OffsetIncrement.extractor gains a default. Only the private _create_record_counting_extractor is removed, and create_page_increment still accepts decoder and extractor_model as keyword arguments, through **kwargs.
Blast radius. 269 connectors reference these paginators (114 OffsetIncrement, 162 PageIncrement), counted with git grep -l -E 'type: *"?(OffsetIncrement|PageIncrement)' origin/master -- 'airbyte-integrations/connectors/*.yaml' on airbyte master at fc800755 and deduplicated by connector.
- After resolving $refs, 266 use them on stream retrievers. Each saves one extraction per page, and every OffsetIncrement also saves one decode per page: its 878 definitions have 858 integer page sizes, 20 absent and 0 templates.
- 15 rely on the pre-filter count (record_filter in 11, client-side incremental in 5, source-tiktok-marketing in both) and paginate exactly as before.
- The 8 custom extractors are pure functions of the response, so the count equals what the copy returned.
- 0 combine these paginators with property chunking, a CombinedExtractor, a custom retriever or an AsyncRetriever download paginator.

Open PR #1074 can reuse this count

#1074 adds a third extractor copy, in CursorPaginationStrategy, to get the same pre-filter count. It can call extracted_record_count(response, default=last_page_size) instead.

Test plan

New and updated tests:

poetry run pytest \
  "unit_tests/sources/declarative/parsers/test_model_to_component_factory.py::test_a_record_counting_paginator_extracts_each_page_once" \
  "unit_tests/sources/declarative/parsers/test_model_to_component_factory.py::test_a_record_counting_paginator_paginates_a_streamed_response" \
  "unit_tests/sources/declarative/parsers/test_model_to_component_factory.py::test_create_offset_increment" \
  "unit_tests/sources/declarative/extractors/test_record_selector.py::test_select_records_counts_the_records_of_the_page_before_filtering_them" \
  "unit_tests/sources/declarative/requesters/paginators/test_offset_increment.py::test_given_a_page_counted_by_the_record_selector_then_do_not_read_the_response_again" \
  "unit_tests/sources/declarative/requesters/paginators/test_page_increment.py::test_given_a_page_counted_by_the_record_selector_then_use_its_count" \
  "unit_tests/sources/declarative/extractors/test_combined_extractor.py::test_a_drop_counting_extractor_warns_about_the_records_it_drops"

Source files from main (git checkout origin/main -- the 5 changed files under airbyte_cdk/) with this branch's tests: test_record_selector.py fails to import extracted_record_count, which stops collection, and the other 8 tests fail when run without it:
- Extracts once: assert 4 == 2 for both strategies (each page extracted twice).
- Streamed body: [1, 3] == [1, 3, 4] for both strategies (stopped after the first page).
- Strategy unit tests: None == 6 and None == 4 (the record selector's count is ignored).
- test_a_drop_counting_extractor_warns_about_the_records_it_drops: 0 == 1; test_create_offset_increment: the strategy still holds a copy.
Without the loop that ends select_records, the filter_that_stops_reading_the_page case of the record selector test fails with -1 == 3: the page is left uncounted.
This branch: 10 passed.
poetry run pytest unit_tests/sources/declarative unit_tests/connector_builder: 2388 passed. test_read_with_concurrent_and_synchronous_streams* (3 tests) was deselected because it can crash locally on macOS (Cached parent stream with more than one partition corrupts the shared sqlite request cache under concurrent reads (no such table: responses, file is not a database, no more rows available) #1150).
poetry run pytest unit_tests/legacy unit_tests/manifest_migrations unit_tests/source_declarative_manifest: 161 passed.
Local runs added -p no:cacheprovider -p no:memray; the memory-limited tests in test_decoders_memory_usage.py and test_response_to_file_extractor.py passed separately with both plugins on.
Cost on one 100k-row page (28.7 MB of JSON, DpathExtractor): on main, OffsetIncrement.next_page_token takes 5.95 s and grows max RSS by 413 MiB, on top of the record selector's own 5.36 s pass. On this branch it takes 0.005 ms and 0 MiB. Reading 2 pages parses JSON 6 times with OffsetIncrement and 4 times with PageIncrement on main, 2 times with either on this branch.
main vs this branch through ConcurrentDeclarativeSource, same requests and records on both:
- property chunking with OffsetIncrement, with and without a record filter: two chunk requests per page, at no offset, then offsets 2 and 4;
- LazySimpleRetriever with OffsetIncrement and PageIncrement and a record filter: the filtered full embedded page still requests the second page.
main vs this branch with retrievers built by ModelToComponentFactory:
- a custom record filter that stops reading at the first record it rejects, with either strategy: same records and the same second-page request on both;
- a custom retriever with OffsetIncrement, and a PageIncrement subclass calling super, each with a record filter: main stops after the first page ([1]), this branch reads all 3 pages ([1, 3, 4, 5]).
ruff format, ruff check on the 10 changed files and mypy --config-file mypy.ini on the 5 source files: clean.

Fixes #1136

Both strategies stop on the number of records the API returned for a page, counted before the record filter. The factory gave each of them a second copy of the stream's extractor, so every page was decoded and extracted twice, and `OffsetIncrement` decoded it once more to evaluate a constant `page_size`. A streamed response cannot be read twice, so the copy counted no records and pagination stopped after the first page. `RecordSelector` now counts every item its extractor yields, including the empty records a `CombinedExtractor` drops, and stores the count on the response. The strategies read it through `extracted_record_count` and fall back to `last_page_size` when no selector read the response. An extractor passed to either strategy directly still counts the page.

github-actions · 2026-10-06T07:59:12Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@bazarnov/fix-paginators-single-page-extraction#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch bazarnov/fix-paginators-single-page-extraction

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

/autofix - Fixes most formatting and linting issues
/poetry-lock - Updates poetry.lock file
/test - Runs connector tests with the updated CDK
/prerelease - Triggers a prerelease publish with default arguments
/poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
/poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

CDK API Reference

📝 Edit this welcome message.

github-actions · 2026-10-06T08:11:10Z

PyTest Results (Fast)

4 990 tests +8 4 978 ✅ +8 11m 2s ⏱️ + 1m 29s
1 suites ±0 12 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 57b58bc. ± Comparison against base commit d214b0c.

This pull request removes 5 and adds 13 tests. Note that renamed tests count towards both.

unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_the_counting_copy_counts_dropped_empty_records_under_union
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_the_counting_copy_counts_nothing_when_every_record_is_empty
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_the_counting_copy_counts_the_drops_of_a_nested_combined_extractor
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_the_counting_copy_counts_the_same_winner_the_record_stream_reads
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_the_counting_copy_does_not_warn

unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_a_drop_counting_extractor_counts_dropped_empty_records_under_union
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_a_drop_counting_extractor_counts_nothing_when_every_record_is_empty
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_a_drop_counting_extractor_counts_the_drops_of_a_nested_combined_extractor
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_a_drop_counting_extractor_counts_the_same_winner_the_record_stream_reads
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_a_drop_counting_extractor_warns_about_the_records_it_drops
unit_tests.sources.declarative.extractors.test_record_selector ‑ test_select_records_counts_the_records_of_the_page_before_filtering_them[filter_that_stops_reading_the_page]
unit_tests.sources.declarative.extractors.test_record_selector ‑ test_select_records_counts_the_records_of_the_page_before_filtering_them[record_filter]
unit_tests.sources.declarative.parsers.test_model_to_component_factory ‑ test_a_record_counting_paginator_extracts_each_page_once[OffsetIncrement-2]
unit_tests.sources.declarative.parsers.test_model_to_component_factory ‑ test_a_record_counting_paginator_extracts_each_page_once[PageIncrement-1]
unit_tests.sources.declarative.parsers.test_model_to_component_factory ‑ test_a_record_counting_paginator_paginates_a_streamed_response[OffsetIncrement]
…

github-actions · 2026-10-06T08:15:45Z

PyTest Results (Full)

4 993 tests +8 4 981 ✅ +8 15m 41s ⏱️ -23s
1 suites ±0 12 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 57b58bc. ± Comparison against base commit d214b0c.

This pull request removes 5 and adds 13 tests. Note that renamed tests count towards both.

unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_the_counting_copy_counts_dropped_empty_records_under_union
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_the_counting_copy_counts_nothing_when_every_record_is_empty
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_the_counting_copy_counts_the_drops_of_a_nested_combined_extractor
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_the_counting_copy_counts_the_same_winner_the_record_stream_reads
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_the_counting_copy_does_not_warn

unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_a_drop_counting_extractor_counts_dropped_empty_records_under_union
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_a_drop_counting_extractor_counts_nothing_when_every_record_is_empty
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_a_drop_counting_extractor_counts_the_drops_of_a_nested_combined_extractor
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_a_drop_counting_extractor_counts_the_same_winner_the_record_stream_reads
unit_tests.sources.declarative.extractors.test_combined_extractor ‑ test_a_drop_counting_extractor_warns_about_the_records_it_drops
unit_tests.sources.declarative.extractors.test_record_selector ‑ test_select_records_counts_the_records_of_the_page_before_filtering_them[filter_that_stops_reading_the_page]
unit_tests.sources.declarative.extractors.test_record_selector ‑ test_select_records_counts_the_records_of_the_page_before_filtering_them[record_filter]
unit_tests.sources.declarative.parsers.test_model_to_component_factory ‑ test_a_record_counting_paginator_extracts_each_page_once[OffsetIncrement-2]
unit_tests.sources.declarative.parsers.test_model_to_component_factory ‑ test_a_record_counting_paginator_extracts_each_page_once[PageIncrement-1]
unit_tests.sources.declarative.parsers.test_model_to_component_factory ‑ test_a_record_counting_paginator_paginates_a_streamed_response[OffsetIncrement]
…

Baz (bazarnov) requested a review from a team as a code owner October 6, 2026 07:59

Daryna Ishchenko (darynaishchenko) approved these changes Oct 6, 2026

View reviewed changes

Baz (bazarnov) merged commit 6673dd6 into main Oct 6, 2026
29 of 30 checks passed

Baz (bazarnov) deleted the bazarnov/fix-paginators-single-page-extraction branch October 6, 2026 09:34

Navigation Menu

Repository navigation

fix(low-code): stop `OffsetIncrement`/`PageIncrement` re-reading pages - #1202

fix(low-code): stop `OffsetIncrement`/`PageIncrement` re-reading pages#1202
Baz (bazarnov) merged 1 commit into
mainfrom
bazarnov/fix-paginators-single-page-extraction

Baz (bazarnov) commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

Testing This CDK Version

PR Slash Commands

Helpful Resources

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Baz (bazarnov) commented Oct 6, 2026

Overview

Changes

Review Spotlight

Risks

Open PR #1074 can reuse this count

Test plan

Uh oh!

github-actions Bot commented Oct 6, 2026

👋 Greetings, Airbyte Team Member!

Testing This CDK Version

PR Slash Commands

Helpful Resources

Uh oh!

github-actions Bot commented Oct 6, 2026

PyTest Results (Fast)

Uh oh!

github-actions Bot commented Oct 6, 2026

PyTest Results (Full)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants