feat(standard-tests): run the Docker read test by default and check its records

[Anatolii Yatsuk (tolik0)]

Part of airbytehq/airbyte-internal-issues#17137 (P1.4 "Docker read test", under airbytehq/airbyte-internal-issues#17129). Companion monorepo PR: airbytehq/airbyte#88939.

Fleet impact: connector CI installs the CDK CLI unpinned. With the next release, airbyte-cdk image test reads from the connector image for every Python and manifest-only connector. On the 58 Python sources, 8 go red (2 certified), all for real reasons. Manifest-only connectors already run the same record checks in-process. The release waits for P0.7 (airbytehq/airbyte-internal-issues#17133) to fix the 8 Python connectors, or to accept the rest as known reds.

What

test_docker_image_build_and_read runs read inside the connector image, and it never ran. The cause:

• --read-from-streams was registered without a default.
• pytest ignores a getoption(default=...) fallback for a registered option.
• The fixture treated the missing value as none.

Neither the CDK's connector matrix nor the connector monorepo passed the option. Python connectors' in-process tests are deselected in the monorepo until P0.7, so the standard tests never run a read for a Python connector. Only source-s3 and source-hardcoded-records read, through their own integration tests.

This PR:

1. Turns the read on by default.
   • --read-from-streams and --read-scenarios get a real default, default.
   • --read-from-streams=false (or none, from this release on) stays the opt-out.
   • PYTEST_ADDOPTS=--read-from-streams=false works for airbyte-cdk image test, which has no pytest passthrough.
   • Use false in CI files: CDK 7.36.0 and earlier read none as a stream name and run the read.
2. Checks the records like test_basic_read. A config under basic_read goes through the same assert_read_records call:
   • Every stream it reads must return records. empty_streams are not read.
   • Every record must match its stream's schema. validate_schema: false opts out, and is rejected at test_strictness_level: high before any Docker command.
   • A config only under full_refresh must return some records.
   • Failure messages name streams, record numbers and schema paths, never record or config values.
3. Runs the Docker check, discover and read in a writable /source working directory (an in-memory tmpfs), as the platform does. The image WORKDIR is not writable by the airbyte user. Because of that, source-shopify and source-salesforce failed the read with PermissionError on their bulk-export temp files. A sync never hits that error.
4. Fixes how the two options work together.
   • The default --read-scenarios read only configs named config, valid_config or default. 12 source connectors name their read configs differently and would have skipped the read once it was enabled. They now read their first read config.
   • --read-scenarios=none raised a ValueError for every config, and a list raised one for each config it left out. Both now skip.
5. Reads once per connector in this repo's matrix. The FAST steps of connector-tests.yml also collect the Docker tests, so they pass --read-from-streams=false. The image test step keeps the read.

How

Option	Default	Other values
--read-scenarios (which configs)	default: configs named config, valid_config or default; else the first read config expected to succeed	all, a comma list of config names, none (skip)
--read-from-streams (which streams)	default: the connector's suggestedStreams, or every read stream when it declares none or none of them is read	all, a comma list of stream names, false / none (skip; only false skips on CDK 7.36.0 and earlier)

The test calls only assert_read_records and assert_schema_validation_opt_out_allowed from _read_assertions.py, both on main. P1.2's follow-up checks (primary keys, state) can be added to both reads later without touching this test's control flow.

Files

• airbyte_cdk/test/standard_tests/pytest_hooks.py: real defaults for both options, fixture parsing, docstrings.
• airbyte_cdk/test/standard_tests/docker_base.py: default scenario selection, stream selection, record checks, /source working directory for check, discover and read.
• unit_tests/test/test_docker_read.py (new, 28 tests).
• .github/workflows/connector-tests.yml: --read-from-streams=false in both FAST steps.
• cdk-migrations.md: new "Upgrading to the next minor release (unreleased)" section.

Fleet impact

Measured locally against the published images of airbytehq/airbyte master (1e22e3c689c), on this branch. Each connector ran twice: with --read-from-streams=none (the read skipped, as today) and with the new default.

Python sources with an acceptance-test-config	Count
Green	18
Newly red	8 (2 certified)
Already red (check/discover fails today: expired or missing test credentials, test accounts gone)	8
Read skipped (no read config, or only status: failed ones)	6
Collection error, legacy flat tests: config (airbytehq/airbyte-internal-issues#16264)	9
Not run: no test secret (8), rotating refresh token (1)	9

Newly red Python connectors (stream names only)

Connector	Certified	Failing check
source-facebook-marketing	yes	no records: the 7 ads_insights* streams
source-google-ads	yes	no records: account_performance_report, geographic_view_with_metrics; its suggestedStreams lists 13 stream names that no longer exist
source-facebook-pages		no records: every stream of its catalog is in empty_streams
source-linnworks		no records: stock_locations, stock_location_details
source-marketo		no records: 27 activities_* streams (test activities aged out)
source-firebolt		configured catalog lists a table the test database no longer has
source-recharge		schema: plans.discount_amount is a decimal string, the schema says integer
source-zoho-crm		schema: picklist enum misses historical values (Industry, Lead_Source)

Cost of the default read per Python connector:

• median 24 s, p90 85 s;
• 229 s for shopify and facebook-marketing;
• 34 minutes for marketo, which declares no suggestedStreams and so reads all its activity exports.

On 5 certified manifest-only connectors the read took 11 to 159 s (klaviyo). --read-from-streams=all costs up to 4 minutes more per connector (stripe, freshdesk), so it is not the default.

Manifest-only connectors were sampled, not swept. Their in-process test_basic_read (#1218) already runs the same checks on every read config and stream, and its 23 red connectors were fixed on master. The Docker read covers a subset of that. It can only add failures from two places: the image itself (packaging, file system), or the connector's pinned SDM base version (baseImage in metadata.yaml), which can differ from the CDK of the unpinned CLI. The sample showed neither.

Rollout

• Minor release, no warn-only period. Do not release before P0.7 (airbytehq/airbyte-internal-issues#17133) lands its fixes for the newly red Python connectors, or accepts the remaining ones as known reds. P0.7's plan covers facebook-pages (page no longer listed in empty_streams), facebook-marketing and marketo (streams without test data declared in empty_streams), zoho-crm (picklist enum) and the google-ads suggestedStreams. It leaves recharge (a breaking type change), linnworks and firebolt (test data) and the google-ads test window to their owners.
• ci(connectors): skip the duplicate Docker read in manifest-only integration tests airbyte#88939 skips the duplicate Docker read in manifest-only poe test-integration-tests with --read-from-streams=false. Every CDK version honours that value: 7.36.0 skips the read with it, checked on source-pokeapi. That PR is a no-op until this release and safe to merge first.
• cdk-migrations.md conflicts with feat(standard-tests): remove the ignored scenario fields and reject acceptance-test-config keys that are not read #1226, feat(standard-tests): lint spec structure, form rendering and OAuth config paths #1227, feat(standard-tests): fail on an empty test-scenario set unless acceptance-test-config.yml declares a bypass_reason #1228 and feat(standard-tests): check primary keys, state messages and error traces in read tests #1230, which all add a section under the same "next minor release (unreleased)" heading. The resolution is mechanical: keep the sections side by side. The code files merge cleanly with all four.
• feat(standard-tests): check primary keys, state messages and error traces in read tests #1230 (P1.2 slice 2) adds primary key, state and trace checks to the in-process test_basic_read. Whichever lands second makes the Docker read pass the same arguments to assert_read_records.
• Migration note: cdk-migrations.md, "Upgrading to the next minor release (unreleased)".

Test plan

• unit_tests/test: 812 passed, 9 skipped (28 new in test_docker_read.py)
• ruff format, ruff check, poe type-check (mypy airbyte_cdk) clean
• Docker suite with none / default / all on 11 published connector images (5 certified manifest-only, 6 Python). The manifest-only rows ran before the writable /source working directory was added; source-klaviyo, source-pokeapi and source-intercom were re-run green on the final branch
• Docker suite with none / default on the 58 Python sources with an acceptance-test-config.yml: 49 run, 9 not run (8 no test secret, 1 rotating refresh token)
• Test Connectors matrix on this PR green (source-google-drive only runs when file_based changes)

🤖 Generated with Claude Code

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@tolik0/cdk/enable-docker-read-test#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch tolik0/cdk/enable-docker-read-test

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 776 tests  +28   5 746 ✅ +28   10m 35s ⏱️ - 1m 35s
    1 suites ± 0      30 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit e70cec4. ± Comparison against base commit 2082c2d.

This pull request skips 1 and un-skips 1 tests.

unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_spec_backward_compatibility

unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_build_and_read['valid_config' Test Scenario]

[github-actions]

PyTest Results (Full)

5 779 tests  +28   5 749 ✅ +28   16m 13s ⏱️ -47s
    1 suites ± 0      30 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit e70cec4. ± Comparison against base commit 2082c2d.

This pull request skips 1 and un-skips 1 tests.

unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_spec_backward_compatibility

unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_build_and_read['valid_config' Test Scenario]