feat(declarative): add OpaqueTokenCursor for opaque server-issued sync tokens (1/2: capture and commit)

[devin-ai-integration]

Summary

Adds OpaqueTokenCursor, a declarative incremental sync type for APIs that return an opaque server-issued sync cursor in the response envelope (Ashby syncToken, Google/Todoist sync tokens, Box stream positions, Graph/Dataverse delta tokens). This is PR 1 of 2 (capture and commit). PR 2, stacked on this one, adds RESET_OPAQUE_TOKEN recovery.

Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/17213:

• https://github.com/airbytehq/airbyte-internal-issues/issues/17213

Close on merge: yes — internally filed, no customer to confirm with. (Close after PR 2 merges as well.)

Changes

• Schema / models: new OpaqueTokenCursor with five settings: token_path, token_option, completion_condition, initial_token, cursor_field. It is added to DeclarativeStream.incremental_sync.anyOf. The definition uses additionalProperties: true so that stream-level $parameters still validate.
• OpaqueTokenCursor (incremental/opaque_token_cursor.py):
  • Keeps per-partition committed and pending token maps, keyed with PerPartitionKeySerializer.
  • A token becomes pending only on a terminal page that satisfies completion_condition and carries a valid string or int token. It is committed in close_partition, so state is queued after every record of the traversal.
  • Failed or partial traversals never change committed state.
  • Partition handling: 25,000-partition cap with no eviction. Tokens of vanished partitions are dropped only after the router enumerates successfully. Partitioned emission is throttled at 600 s, with a final unthrottled emission.
  • Foreign or mis-shaped state is ignored (INFO) and the stream bootstraps.
  • Log messages follow the issue's observability table.
• OpaqueTokenRequestOptionsProvider: sends the stored token, otherwise initial_token, otherwise nothing, through token_option. For partitioned streams it is wrapped in PerPartitionRequestOptionsProvider.
• SimpleRetriever: new page_observer hook (PageCompleted(stream_slice, response, pagination_exhausted)). It is called after a page's records are yielded and after the next page token is computed. It is bound only for OpaqueTokenCursor and is None (inert) for every other stream.
• DefaultStream: new source_defined_incremental flag. A token stream advertises [full_refresh, incremental], source_defined_cursor: true, is_resumable: true, and default_cursor_field: [] (or [cursor_field]). It never advertises null.
• Factory: cursor and provider wiring, plus every validation in the issue's table:
  • datetime-only keys
  • GroupingPartitionRouter
  • AsyncRetriever
  • lazy parents
  • incremental_dependency in either direction
  • a partition field named token
  • pagination_reset.limits
  • property chunking
  • a CustomRetriever warning

Example (Ashby):

incremental_sync:
  type: OpaqueTokenCursor
  token_path: ["syncToken"]
  token_option: {type: RequestOption, inject_into: body_json, field_name: syncToken}
  cursor_field: updatedAt

Reproduction (before the change)

On main (v7.37.0), a manifest whose stream uses incremental_sync: {type: OpaqueTokenCursor, ...} fails ConcurrentDeclarativeSource(source_config=manifest) with jsonschema.exceptions.ValidationError: Validation against json schema defined in declarative_component_schema.yaml schema failed. incremental_sync only accepts DatetimeBasedCursor and IncrementingCountCursor.

Declarative-First Evaluation

This is a CDK feature, not a connector change. DatetimeBasedCursor and IncrementingCountCursor both derive state from record values, and no existing declarative component can capture a value from the response envelope and commit it only when the traversal completes. The new component is built-in and declarative, so adopters need no custom Python components.

Test Coverage

• unit_tests/sources/declarative/incremental/test_opaque_token_cursor.py: unit tests for T-TOKEN, T-CAP, T-FOREIGN, T-SCHEMA, T-DISC and T-VALID, plus the cursor state rules.
• unit_tests/sources/declarative/incremental/test_opaque_token_cursor_integration.py: end-to-end tests through ConcurrentDeclarativeSource.read with HttpMocker, asserting exact request bodies. Covers T-BOOT, T-INC, T-NOECHO, T-EMPTY, T-FAIL, T-BUILDER, T-INIT, T-PAR (3 partitions), T-RESUME and T-INERT.
• The new e2e tests fail on unmodified main (schema validation error) and pass with this change.
• Full unit suite: all tests pass except a few that also fail on unmodified main: two date-only parsing tests in file-based, and a schema-test fixture race under -n auto.

Breaking change

None. This is an additive, opt-in component. Existing cursors, streams and the retriever are unchanged when the cursor is not used: page_observer is None, and source_defined_incremental defaults to False.

Link to Devin session: https://app.devin.ai/sessions/94b41efe75db43e78eb5a4f8d5f58484
Open in Devin Desktop: https://app.devin.ai/desktop/session/94b41efe75db43e78eb5a4f8d5f58484?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1792189821-opaque-token-cursor#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1792189821-opaque-token-cursor

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 798 tests  +44   5 767 ✅ +44   12m 26s ⏱️ +47s
    1 suites ± 0      31 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 9808890. ± Comparison against base commit 73ccaa9.

[github-actions]

PyTest Results (Full)

5 801 tests  +44   5 770 ✅ +44   17m 9s ⏱️ + 7m 1s
    1 suites ± 0      31 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 9808890. ± Comparison against base commit 73ccaa9.