Repository navigation
feat(page size reduction): rewrite the page size inside a RequestPath page token URL - #1169
Draft
Anatolii Yatsuk (tolik0) wants to merge 15 commits into
Draft
Anatolii Yatsuk (tolik0) wants to merge 15 commits into
Anatolii Yatsuk (tolik0) wants to merge 15 commits into
Conversation
…ction An HttpClient retry re-sends the same PreparedRequest, so an error handler that mutates the stream's page size can never affect the request being retried: the oversized page size is already serialized into the URL or the body. This adds a `REDUCE_PAGE_SIZE` response action and a `page_size_reduction` block on `SimpleRetriever` so a manifest can say "re-issue this page smaller" instead. On a matching response, `SimpleRetriever._read_pages` shrinks the page size and re-issues the same page. All reduction state lives in a `PageSizeReducer` created per `_read_pages` call, mirroring `PaginationTracker`, because one retriever and one paginator are shared by every partition of a stream and partitions are read concurrently. The reduced page size reaches the paginator and the pagination strategy as an appended `page_size_override: Optional[int] = None` keyword argument, built by `page_size_override_kwargs` so it is omitted when there is no reduction. A paginator or strategy defined outside the CDK keeps working unchanged. Stop conditions see the size that was actually requested: - `OffsetIncrement` compares `last_page_size` against the requested size. - `CursorPagination` exposes it to `stop_condition` and `cursor_value` as `page_size`, and the factory rejects a `stop_condition` comparing `last_page_size` against anything else, because a full page at the reduced size would otherwise read as a short page and end the pagination silently. - `PageIncrement` is rejected, including through a subclass: pages are addressed as page number * page size, so a smaller page shifts every following boundary. Termination is bounded by `max_attempts` reductions. That budget restarts after every successful page under `reset_policy: AFTER_SUCCESSFUL_PAGE`, which is the policy for an API that rejects the configured page size on every page - without the restart such a stream would fail at page `max_attempts + 1` however healthy the reads are. `PageSizeReducer.MAX_TOTAL_REDUCTIONS` bounds the partition either way, and each reduction waits a short, growing amount of time before re-issuing. The action is rejected at config time everywhere it cannot be honored: query properties, `file_uploader`, `lazy_read_pointer`, all six `AsyncRetriever` sub-requesters and `login_requester`. A `CustomPaginationStrategy` is accepted only when its `next_page_token` can receive the override. When the action reaches a retriever the factory could not inspect, the reduction fails with a config error naming the stream rather than a bare crash. `HttpClient` logs a response resolving to `REDUCE_PAGE_SIZE` as an auxiliary request, so a Connector Builder test read still shows the failed attempt without counting it against `max_pages_per_slice`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… failing healthy long partitions
Three round-2 review findings, two of them introduced by the round-1 fixes.
The `stop_condition` gate matched the expression as a string, and a regex
cannot do that job in either direction. `\bpage_size\b` matches inside
`config['page_size']` because `'` is a non-word character, so
`{{ last_page_size < config['page_size'] }}` passed the gate and still
silently truncated - the exact bug the gate exists to prevent. And
`{{ last_page_size == 0 }}` was rejected although no reduction can make a
full page empty; that is 11 of the 14 `last_page_size` stop conditions in
the monorepo, including all 6 in source-zendesk-support, which this PR
names as an adopter. Patching the regex was not the fix: two bugs in one
pattern meant the mechanism was wrong.
The condition is now parsed as a Jinja expression and the comparisons
`last_page_size` takes part in are classified on the AST, where a bare
`page_size` Name is distinguishable from a Getitem on `config`. The rule is
whether the condition can be true for a page that is full at the size that
was requested: an emptiness test and a threshold at or below
`minimum_page_size` cannot be, a lower bound can only stop being satisfied
as the page shrinks, and an upper bound is safe only when it follows the
reduction. A shape the analysis cannot classify is warned about rather than
rejected, because this runs at stream construction and a false rejection
takes `check` and `discover` down with `read`. Reverting to the round-1
regex fails 12 of the new factory tests, in both directions.
`MAX_TOTAL_REDUCTIONS = 1000` guaranteed termination but moved the
`AFTER_SUCCESSFUL_PAGE` cliff from page 6 to page 1001 instead of removing
it, and the failure it produced was wrong three ways: `transient_error` on a
job that could never succeed, "the source kept failing" on a partition where
every page succeeded, and a page size that was restored rather than
requested. A stream that gets every page through after one reduction is
healthy and has to complete, so the cap is gone. `max_attempts` now bounds
only the reductions made in a row *without* a page succeeding, which is what
separates a stuck partition from an expensive one. Termination still holds:
only a successful page restarts the budget and `_read_pages` calls that once
per page it consumed, so every restart costs a page of progress. The
schema's `max_attempts` and `reset_policy` descriptions say so, and two
tests pin the healthy long stream and the stuck partition - the first one
fails against the previous revision.
Also:
- `PageSizeReductionRequiredException` goes back to `config_error`. On the
SimpleRetriever path it is always caught and the type is inert; the only
way it escapes is a retriever that never re-issues the page, where a
job-level retry would fail identically forever. The escape path is kept
deliberately - it is the only signal such a stream gets - and the message
no longer claims a retry that will not happen.
- Two assertions passed vacuously. `"test" in internal_message` also matched
the stream name, so it certified nothing about the error handler's
`error_message` reaching the exception; it now asserts the mapping's text.
`test_given_no_page_size_then_page_size_interpolates_to_none` held equally
if `page_size` were never bound at all; it is now parametrized with two
positive cases that pin the variable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…odel Reverts 09882ec ("Auto-committed changes from Poe command `build`"). Running the codegen pipeline produced a +259/-265 diff against the committed generated model and turned the MyPy Check red with 8 `valid-type` errors. Every one is a `conint(...)` or `confloat(...)` used as an annotation, which mypy rejects; the committed model on `main` contains none of these forms, so `main` is green and only a regenerated file fails. Three of the eight come from this PR (`PageSizeReduction.reduction_factor`, `.minimum_page_size`, `.max_attempts`). The other five are pre-existing and unrelated to this change: `DynamicStreamCheckConfig.stream_count`, `ConstantBackoffStrategy` (x2), `ExponentialBackoffStrategy.jitter_range_in_seconds`, and `AsyncRetriever.failed_retry_wait_time_in_seconds`. They are latent on every branch and surface the moment anyone regenerates. The root cause is in bin/generate_component_manifest_files.py: the datamodel-codegen invocation does not pass `--field-constraints`, so numeric `minimum` / `exclusiveMinimum` constraints become `conint()` / `confloat()` instead of `Field(ge=...)`. Fixing that regenerates the whole file and should be its own PR against main. Reverting loses no validation. The manifest is checked against declarative_component_schema.yaml by `jsonschema.validators.validate` in `_validate_source()`, so the YAML bounds are enforced regardless of what the generated model expresses - the same mechanism that enforces `minItems` on other components, which codegen also drops. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stop-condition analysis read each comparison on its own, which is only
sound while that comparison decides the condition in its own polarity.
`{{ not (last_page_size >= 100) }}` and `{{ last_page_size - 100 < 0 }}` both
mean `last_page_size < 100`, yet both were accepted as SAFE, so a full page at
a reduced size read as a short page and the partition was truncated silently -
the exact failure the check exists to prevent.
A comparison is now classified only when it is reached from the root of the
expression through `and`/`or` alone, and the `lt`/`lteq` readings require
`last_page_size` to be compared bare rather than after arithmetic. Anything
else is UNKNOWN, which warns instead of rejecting.
Also brings the reducer's user-facing messages in line with the error-message
guideline: every message names the stream, and the two `transient_error`
messages drop the remediation the user cannot act on. The `config_error`
messages keep theirs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`{{ last_page_size < [page_size, 100] | max }}` was SAFE because the threshold
only had to *mention* `page_size`, while `max(50, 100)` is the configured size
again; `{{ last_page_size < page_size + 50 }}` had the same hole. The `lt`
branch now requires the threshold to be the `page_size` variable itself,
filters aside, so an expression built from it falls to UNKNOWN.
`{% if last_page_size < 100 %}true{% endif %}` had dropped from TRUNCATES to
UNKNOWN when the traversal started tracking whether a comparison decides the
condition. A `{% if %}` renders truthy text exactly when its test holds, so the
test is a decider - but only while the branch it guards is the whole story, so
an `else`, an `elif`, or a body the CDK reads as false keeps it unclassified.
Text rendered next to a comparison is unclassifiable for the same reason: the
condition is then truthy whatever the comparison decided.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…terminal failure message Two fixes from the architecture review of this PR. A `page_token_option` of type `RequestPath` makes the next page a URL the API built, and that URL already carries the page size the API echoed back. The reduced page size is injected as a request option on top of it, so the request goes out with the page size twice and which one the API honors is up to the API: every page after the first would keep asking for the size that just failed. `ModelToComponentFactory` now rejects that combination at stream construction and points at a `RequestOption` page token instead. When the reduction bottoms out, the CDK only knows that the API rejected every page size it asked for. What narrows a query down is API-specific, so `PageSizeReduction` now takes an optional `failure_message` that is appended to the two `transient_error` messages. It is deliberately not appended to the `config_error` ones: those are about the manifest, not about the API rejecting a page size, and they carry their own remediation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…fe, and make the reduction budget mean what it says
Seven findings from the code review of this PR.
F1, the blocker. `classify_stop_condition` returned SAFE for every
`stop_condition` that did not name `last_page_size`, but the response body
carries the same count: `{{ response['data'] | length < 100 }}` is
`{{ last_page_size < 100 }}` counted one layer out, and it truncates in exactly
the same way. Which of a connector's own response fields holds a page length is
not knowable at config time, so a condition that bounds any value from above is
now reported as UNKNOWN - a warning, which is what this analysis already does
for every shape it cannot read. A condition that makes no upper-bound
comparison, the common `{{ not response.next }}` shape, cannot observe a page
length by comparison and stays SAFE. Measured over the monorepo at
`09124c5aabc`: of 1505 conditions, 1466 stay accepted, 3 rejected and 36 now
warn - the 36 are the response-derived page lengths in source-pardot,
source-zendesk-chat and source-zendesk-talk, which is the shape this is about.
All three of source-github's conditions stay accepted.
F2. `max_attempts` counts the reductions made in a row without a page getting
through, which is what the terminal error claims happened - but
`on_successful_page` only restarted the budget under `AFTER_SUCCESSFUL_PAGE`, so
under the default `NEVER` a partition in which every reduction was followed by a
successful page still failed at the `max_attempts + 1`-th reduction, blaming the
API for rejecting every page size it had in fact served. That is the same defect
as the `MAX_TOTAL_REDUCTIONS` cap this PR removed, with the cliff at the 6th
reduction instead of the 1001st page, and it is reachable exactly when per-page
cost varies - the GraphQL case the feature exists for. The budget now restarts
under both policies; `reset_policy` only decides whether the page size is
restored. Termination still holds: under `NEVER` the page size strictly
decreases, so `minimum_page_size` bounds the partition on its own.
F3. Each reduction divides the page size by `reduction_factor` and spends one
attempt, so an unbroken run of failures bottoms out at
`page_size / reduction_factor ** max_attempts` whatever `minimum_page_size`
says: with the defaults a floor of 10 on a page size of 1000 is never reached
and its error branch never fires. The factory now warns when the floor is out of
reach of the budget, and says how large `max_attempts` would have to be. It only
warns when the floor was set explicitly - the default of 1 is out of reach of
the default budget on any page size above 32 - and it does not raise, because
pages that succeed in between restart the budget, which makes the floor
reachable over a partition.
F4. The `PageSizeReduction` and `max_attempts` descriptions now say that the
wait between reductions is the CDK's own and does not consult the error
handler's `backoff_strategies` or a `Retry-After` header.
F5. `bin/generate_component_manifest_files.py` now records why
`declarative_component_schema.py` is edited by hand for bounded numeric fields:
without `--field-constraints` the generator emits `conint`/`confloat`
annotations that mypy rejects, on five fields that predate this PR.
F6. A rejected page appeared in the Connector Builder auxiliary panel titled
like an ordinary page fetch. `_as_auxiliary_request_log` now takes a title and
description, and the reduction path passes its own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…only warn about a floor worth reaching
Two findings from the second round of review.
R2-F1. `_bounded_from_above` read `lt`/`lteq` in full - any non-literal left
operand counts as bounded - but read `gt`/`gteq` only when the threshold was a
literal, so `{{ config['page_size'] > response['data'] | length }}` stayed
classified as safe while `{{ response['data'] | length < config['page_size'] }}`
was warned about. Same comparison, same silent truncation, and the module
docstring promises that any comparison bounding a value from above is reported.
Every one of the four ordering operators bounds one of its operands from above
and only a literal is certain not to be a page length, so both directions are
now mirrored, as `_MIRRORED_OPERATORS` already does on the `last_page_size`
path. `{{ response.page >= response.total_pages }}` warns as a result: in
`a >= b` there is an upper bound on `b`, and the earlier reading of that shape
as "no upper bound, so nothing here can be a page length" was wrong.
Measured over the monorepo at `09124c5aabc`: of 1505 conditions, 1459 stay
accepted, 3 rejected and 43 warn, up from 36. The 7 newly warned are the two
shapes the mirror adds, source-serpstat x6 and source-jira x1; neither bounds a
page length, and neither connector opts into `page_size_reduction`. All three of
source-github's conditions, on the adopter branch, stay accepted.
R2-F2. The floor-reachability warning gated on `__fields_set__`, which records
that a value was supplied and not that it differs from the default, so spelling
`minimum_page_size: 1` out longhand earned a warning advising `max_attempts: 10`
on any page size above 32. It now gates on the floor being above 1, which is the
condition that makes the floor worth reaching in the first place.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ries at the minimum page size Two gaps found while adopting `page_size_reduction` in source-github, where GitHub answers 502 both to a GraphQL page that is too expensive and to a transient hiccup: - The wait between attempts was a hard-coded 0.5s times the attempt count, so a run of five reductions spread six requests over ~7s. `backoff_seconds` makes it a knob; the default keeps today's behaviour. - Once the page size reached `minimum_page_size`, the first response asking for a smaller page ended the stream. `REDUCE_PAGE_SIZE` bypasses the HTTP retry budget, so at the floor a stream got fewer attempts than it had before adopting the reduction. `retries_at_minimum_page_size` re-issues the same page unchanged, after the backoff, in a budget of its own, separate from `max_attempts` (which only counts reductions) and restarted by every successful page. The default of 0 keeps today's behaviour. The `config_error` raised when the page size can never be reduced is now limited to the case where `minimum_page_size` is what blocks it, which is the only one the user can act on. A stream whose configured page size is already 1 gets the transient error instead: one record per page is as small as a page gets, so the API rejecting it says nothing about the configuration. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`PageSizeReducer` took `sleep: Callable = time.sleep`, which binds the function object when the module is imported. A connector test that patches `time.sleep` to keep a run of reductions from taking its backoff for real therefore had no effect, and a suite covering a reduction down to the minimum page size paid the whole wait. The override is now stored and `time.sleep` is looked up per call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…page size got there Two findings from the fourth round of review. R4-F1. `_retry_at_minimum_page_size` tested the `config_error` branch first, and that branch never consulted `retries_at_minimum_page_size`. So the same manifest gave a partition its retries when the reduction walked down to the floor and none at all when the user's `page_size` was already there: the first response ended the stream, telling the user to raise `page_size` or lower `minimum_page_size` - a manifest field they cannot reach, for a case the connector author had already answered by configuring retries. Whether a transient error was retried therefore depended on how the page size arrived at the floor rather than on anything about the error. The retries now come first and the misconfiguration is reported once they are spent, so it is still reported and still names both numbers to change. Measured over the eight configurations that reach the floor: `configured=20, min=10, retries=3` and `configured=10, min=10, retries=3` now take the same three waits and differ only in the failure type, which is the one thing that should differ - a floor that blocks every reduction is the connector's to fix, a floor the reduction reached means the API kept rejecting every size. R4-F2. The sleep-resolution fix of `01add3fa` was not pinned by any test: every test here passes its own `sleep`, so none exercised the default path. `test_given_no_sleep_override_when_reduce_then_wait_through_time_sleep` patches `page_size_reducer.time.sleep` and asserts the wait lands there. Verified against the defect: restoring the bound default makes that test fail, and the file's run takes 7.5s of real sleeping instead of 0.45s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@tolik0/cdk/reduce-page-size-request-path#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch tolik0/cdk/reduce-page-size-request-pathPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
…nd page_size
`page_size` is bound to the page size that was actually requested, which is what makes
`{{ last_page_size < page_size }}` the one comparison that survives a reduction. But it is
only a number when the CursorPagination strategy declares a `page_size`. Without one it
binds to `None`, and the comparison does not fail: Jinja raises, `JinjaInterpolation._eval`
treats the TypeError as "not a template" and returns the raw template string, and
`InterpolatedBoolean` reads a non-empty string as `True`. The stop condition is then
satisfied on page 1 and the rest of the partition is dropped without anything failing.
This mattered beyond the streams that reduce: the example and the advice to prefer
`page_size` over a hardcoded number sit on `CursorPagination.stop_condition`, a field 710
pagination blocks across 97 connectors already use without declaring a `page_size`.
`CursorPaginationStrategy.__post_init__` now rejects a `stop_condition` or `cursor_value`
that reads `page_size` while the strategy declares none. The reference is read off the
Jinja AST, so the bare variable is told apart from `config['page_size']`, which is a
lookup on the config and is bound either way. Rejecting at construction rather than
failing per page keeps a sync from emitting a truncated partition first.
The schema now says the variable needs a declared `page_size`, and points at
`{{ last_page_size == 0 }}` for the case where there is none.
No manifest in the monorepo is affected: 0 of 535 declare a CursorPagination without a
`page_size` whose `stop_condition` or `cursor_value` reads the variable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… page token URL `page_size_reduction` rejected a `page_token_option` of type RequestPath: the next page is then a full URL built by the API which already carries the page size it echoed back, so the reduced page size was injected next to the original one and the request went out asking for both. `HttpClient._dedupe_query_params` only drops the injected parameter when the two values agree, which is exactly when there is nothing to reduce. `DefaultPaginator.path` now takes the reduced page size and rewrites that parameter inside the token URL, leaving a single page size in the request. Only the parameter named by `page_size_option` is touched and only when the URL already carries it; every other pair is kept byte for byte rather than parsed and re-encoded, so a token URL the API built is handed back as it was written. Whether a page token URL can be re-requested at a smaller page size at all is something only the connector knows - it holds for a URL addressing records by cursor or timestamp, and not for one carrying a page number, where a smaller page size moves every following boundary and skips records the way PageIncrement does. A RequestPath token is opaque to the CDK, so this is opted into with `rewrite_page_size_in_page_token_url` rather than assumed, and the manifest is rejected when the field is set without a RequestPath token or with a page size that does not inject into a request parameter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Anatolii Yatsuk (tolik0)
force-pushed
the
tolik0/cdk/reduce-page-size-request-path
branch
from
September 18, 2026 15:28
9909693 to
86afff0
Compare
…ion for RequestPath The PageSizeReduction description was updated when rewrite_page_size_in_page_token_url was added, but the field description on SimpleRetriever still said a RequestPath page token is rejected outright. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1149 — review that one first; this branch targets
tolik0/cdk/reduce-page-sizeand its diff is the last commit only.What
Lets
page_size_reductionbe used on a paginator whosepage_token_optionis aRequestPath, by rewriting the page size inside the URL the API returned for the next page instead of sending the reduced page size next to the one that URL already carries. Opted into with a newrewrite_page_size_in_page_token_urlflag onPageSizeReduction.Why
#1149 rejects a
RequestPathpage token at config time. The reason is real, and reproducible onmaintoday:HttpClient._dedupe_query_paramsdrops an injected parameter only when the URL already carries the same value — which is exactly the case where there is nothing to reduce. So with aRequestPathtoken the request goes out asking for both page sizes and the API picks; every page after the first keeps asking for the size that just failed.27 connectors in the monorepo pair a
RequestPathpage token with apage_size_option. The near-term adopter is source-zendesk-supportticket_comments(manifest.yaml:984-995):CursorPaginationonnext_page,page_token_option: RequestPath,page_size_option: per_page, and a504 → RETRYfilter that retries at an unchangedper_pageso the retries cannot succeed.One correction to the #1149 description, which said 3 of those 27 (mendeley, sendgrid, zendesk-support) already handle 502/504: grepping the current manifests, only zendesk-support has any 5xx response filter — mendeley and sendgrid have none. Structurally this unlocks 27 connectors; today exactly one of them has an error path to adopt it for, and that path is a gateway timeout rather than an explicit "page too large" signal. Worth weighing before merging.
How
DefaultPaginator.pathtakespage_size_overrideand rewrites the parameter named bypage_size_optioninside the token URL. Only that pair is edited; the rest of the query is kept byte for byte rather than parsed and re-encoded, so a URL the API built is handed back as it was written (percent-encoding, blank values and bare flags survive).get_request_paramsthen adds the reduced page size the usual way, and appending it to the path would send it twice.paththroughSimpleRetriever._paginator_path, passed withpage_size_override_kwargsso a paginator defined outside the CDK that does not accept the argument keeps working.PaginatorTestReadDecoratorforwards it the same way._dedupe_query_paramsdrops the injected one and a single page size goes out. There is a test asserting exactly that, next to one asserting the duplicated URL without the rewrite.Why it is opt-in
Whether a page token URL can be re-requested at a smaller page size is a property of the API, and a
RequestPathtoken is opaque to the CDK. It holds for a URL addressing records by cursor or timestamp; it does not hold for one carrying a page number, where a smaller page size moves every following page boundary and skips records — the same hazard that makes #1149 rejectPageIncrement. So the connector asserts it rather than the CDK assuming it.Rejected at config time:
rewrite_page_size_in_page_token_urlwithout aRequestPathpage token — there is no URL to rewrite in.rewrite_page_size_in_page_token_urlwith apage_size_optionthat does not inject intorequest_parameter— only a query parameter of that URL can be rewritten.RequestPathpage token without the flag — the existing feat: add REDUCE_PAGE_SIZE response action for dynamic page-size reduction #1149 error, now naming the flag and what has to be true about the API to set it.Scope
Nothing changes for a stream without
page_size_reduction:page_size_overrideisNonefor every other stream, andpathreturns the token unchanged as before. The URL rewrite is reachable only from a reduced page.Testing
DefaultPaginator.path: rewrite, absent parameter, no query, no override, non-request_parameterinjection, repeated parameter, a parameter whose name is a prefix of the page size field, an interpolatedfield_name, a fragment, other parameters returned byte for byte.HttpClient._create_prepared_request: one page size out after the rewrite, two without it.SimpleRetriever: the override reachespath, is omitted when there is none, and_fetch_next_pagesends the rewritten URL.unit_tests/sources/declarativeandunit_tests/sources/streams/httpgreen (2468 passed); the two memory-usage tests were skipped locally, they SIGILL on this machine independently of this branch.No connector adopts this yet; source-zendesk-support would need its
504filter switched toREDUCE_PAGE_SIZEand an SDM prerelease image, since it is manifest-only.