Repository navigation
feat: build catalog-only streams for config-free discover (create_catalog_stream) - #1237
Draft
Aaron ("AJ") Steers (aaronsteers) wants to merge 3 commits into
Conversation
Co-Authored-By: AJ Steers <aj@airbyte.io>
Contributor
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksTesting This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1792446479-catalog-only-discover#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1792446479-catalog-only-discoverPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
Airbyte Support Bot (airbyte-support-bot)
requested a balanced review from Copilot
October 19, 2026 22:14
Copilot started reviewing on behalf of
Airbyte Support Bot (airbyte-support-bot)
October 19, 2026 22:14
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Missing config values in cursor-field templates can silently produce incorrect catalogs instead of configuration errors.
1 open issue
What changed in this PR
Extends config-free discovery with catalog-only streams, avoiding construction of credential-dependent runtime components.
Changes:
- Routes eligible config-free discovery through catalog-only stream construction.
- Shares schema loading and cursor-field resolution with full streams.
- Adds catalog parity tests and a guard against reading catalog-only streams.
| File | Description |
|---|---|
| unit_tests/sources/declarative/test_concurrent_declarative_source_config_free_discover.py | Tests catalog parity and read prevention. |
| airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py | Adds catalog-only construction and shared metadata helpers. |
| airbyte_cdk/sources/declarative/concurrent_declarative_source.py | Routes eligible discovery to catalog-only streams. |
Review effort: Balanced
…onfig value Co-Authored-By: AJ Steers <aj@airbyte.io>
Co-Authored-By: AJ Steers <aj@airbyte.io>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
👉 TL;DR: Lets sources like GitHub return their catalog without a config, even when building their full streams needs credentials or a start date.
Without a config,
ConcurrentDeclarativeSource.discovernow builds streams through a new catalog-only vector,ModelToComponentFactory.create_catalog_stream. It shares_build_schema_loader,_resolve_cursor_fieldandDefaultStream.as_airbyte_stream()with the full vector increate_default_stream.Stacked on #559:
Changes
_build_schema_loader: the schema-loader block moves out ofcreate_default_stream, unchanged. Both vectors call it._resolve_cursor_field: the cursor-field resolution, which was copied into the datetime, incrementing-count and per-partition cursor builders, now lives in one helper (catalog-defined override first, otherwise the manifestcursor_fieldinterpolated with$parameters).create_catalog_stream: returns aDefaultStreamwith the same name, schema loader, primary key, cursor field andsupports_file_transferascreate_default_stream. It never builds requesters, authenticators, partition routers or real cursors. Its_CatalogOnlyPartitionGeneratorraises if anything tries to read it.StateDelegatingStream: uses the full refresh stream when there is no state, the same ascreate_state_delegating_stream.incremental_synctypes raise, the same as the full vector.ConcurrentDeclarativeSource.discover: uses_catalog_streams()only whenconfigis empty andcheck_config_during_discoverisFalse, which is feat: Add config-free discover for declarative sources with static catalogs #559's config-free path. Configured discover,checkandreadstill go throughstreams().Review Spotlight
Reviewers with limited time, please review first:
create_catalog_stream: the catalog-only vector_resolve_cursor_field: shared by the three read-path cursor buildersdiscover/_catalog_streams: when vector 2 is usedRisks
create_default_streamalters a catalog field (name, schema, primary key, cursor field, file transfer),create_catalog_streamhas to change too. The parity tests are the guard.config, config-free discover still fails with feat: Add config-free discover for declarative sources with static catalogs #559'sconfig_error. It never returns a partial catalog.Evidence
AirbyteStreamequalStateDelegatingStream; catalog-only streams can't be readThe 2 remaining build failures are imports in each connector's own
components.py: source-bigcommerce (pendulum) and source-braze (DatetimeBasedCursor, which the CDK no longer exports).Only source-github was compared against a configured discover. For the other 198 newly covered sources, the evidence is that they return a catalog.
Newly covered sources
Show/Hide Content
Link to Devin session: https://app.devin.ai/sessions/ae9d8c8aceea4199ac2609308e973916
Open in Devin Desktop: https://app.devin.ai/desktop/session/ae9d8c8aceea4199ac2609308e973916?variant=devin
Requested by: Aaron ("AJ") Steers (@aaronsteers)