fix: honor the configured lookback_window when filtering client-side incremental records

[Baz (bazarnov)]

Summary

ConcurrentCursor.should_be_synced filtered records from self.start, ignoring the configured lookback_window, while stream_slices requests records from the state minus the lookback window. So for client-side incremental streams (is_client_side_incremental: true) the CDK asked for the lookback window and then dropped it: records whose cursor value is behind the state but inside the lookback window were silently lost.

The effect is worst with global_substream_cursor: true: the global cursor is the max across all partitions, so records of a partition that sit slightly behind it are dropped, which is the missing conversation_parts reported for source-intercom in #1014. The same gap also affects plain and per-partition client-side streams, and parent streams with incremental_dependency: true, where a dropped parent record means its child partition is never requested.

This is a regression from #679 (released in 6.60.4). Before it, the legacy DatetimeBasedCursor.should_be_synced used max(start_datetime, state - lookback_window).

Changes:

• ConcurrentCursor.should_be_synced now keeps records from the state minus the lookback window, never before the configured start date, which is the same lower bound stream_slices requests. Nothing changes when lookback_window is unset or zero.
• The is_client_side_incremental and is_data_feed descriptions in declarative_component_schema.yaml (and the two matching strings in the generated models) now describe the lookback window.

Behaviour change for connectors

Only streams that combine lookback_window with is_client_side_incremental or is_data_feed are affected. They emit the records inside the lookback window again, as they did before 6.60.4. State never moves backwards: the emitted state is still the max synced cursor value.

• Record counts per sync go up by the records in the lookback window, which means more duplicates for append-mode destinations. Dedup destinations are unaffected. Streams with a hard-coded, always-on lookback are source-snapchat-marketing (P2D) and source-gorgias events (PT5M).
• Parent streams with incremental_dependency: true and a lookback produce child partitions for parent records inside the window, so they make more child requests (source-intercom conversation_parts and company_segments when lookback_window is set).
• Data feeds with a lookback paginate until they pass the state minus the lookback window (no such stream in the monorepo today).

Tests

• test_cursor.py: records inside the lookback kept and older ones dropped; the lookback never keeps records before the configured start; state below the configured start keeps the previous floor; the start=None and OverflowError branches.
• test_client_side_incremental_lookback.py (new, end to end): plain, per-partition and global substream cursors with state, with and without a newer record, asserting emitted records and state (including per-partition states); the no-state case; an Intercom-shaped parent with incremental_dependency and a global child cursor.
• test_data_feed_integration.py: a data feed with a lookback paginates through the window and emits its records.
• Reverting the cursor change turns every lookback test red. Replacing the clamp with max(lower, self._start) or removing it turns the matching guard tests red.

Thanks to Praveen Kumar Mittal (@mittalpk), whose analysis and branch in #1014 pointed at the same method. This version adds the start-date clamp (without it, records older than the start date were emitted on the first sync) and the end-to-end coverage.

Fixes #1014

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@bazarnov/fix-lookback-client-side-filter#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch bazarnov/fix-lookback-client-side-filter

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

4 708 tests  +20   4 696 ✅ +20   8m 32s ⏱️ + 1m 56s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit c5da119. ± Comparison against base commit 652c0fe.

[github-actions]

PyTest Results (Full)

4 711 tests  +20   4 699 ✅ +20   14m 27s ⏱️ -1s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit c5da119. ± Comparison against base commit 652c0fe.

[Anatolii Yatsuk (tolik0)]

Approving. The fix is correct. should_be_synced now uses the same lower bound that stream_slices requests, and streams without a lookback stay on the old return self.start path. The start-date clamp and the OverflowError branch are both covered by tests. I also reverted the fix locally, and the unit, plain-stream, and data-feed lookback tests went red.

One request before merge: please update the "Behaviour change for connectors" section of the description. Release notes are built from it, and it currently leaves out some affected connectors. I scanned every manifest.yaml in the monorepo for a DatetimeBasedCursor that sets both lookback_window and is_client_side_incremental (with $refs resolved). No data feed sets a lookback, so that part of the description is right. These are the client-side streams:

Connector	Streams	Lookback
source-snapchat-marketing	8 streams on their own cursor, plus the parent cursors of 19 stats/child streams	P2D, hard-coded (already listed)
source-app-store-connect	customer_reviews_per_app	P7D, hard-coded, so it is always affected and gets the largest jump per sync. Not listed.
source-gorgias	events	PT5M, hard-coded (already listed)
source-intercom	conversation_parts, company_segments (listed), plus companies, contacts, conversations, segments	from config, default 0
source-gong	callTranscripts (global_substream_cursor: true)	from config (lookback_window_days), default 0. Same shape as #1014, so this PR fixes it too. Not listed.
source-linkedin-ads	account_users, campaign_groups, campaigns, conversions, creatives, plus the parent cursors of 11 analytics streams and videos	from config, default 0. Not listed.

When the config-driven windows are left at their default, they evaluate to P0D, which gives a falsy timedelta(0). Those streams only change for users who set a lookback.

Two optional nits:

• _lowest_cursor_value_to_sync() runs once per record, but all of its inputs are fixed after __init__, so it could be computed once.
• In test_client_side_incremental_lookback.py, the incremental-parent test checks state with >=. Exact values (_STATE_VALUE + 10 or _STATE_VALUE) would also catch a state that moves too far forward.