feat(standard-tests): run each acceptance-test-config.yml config with its section's command, and assert the check status and basic_read records

[devin-ai-integration]

Changes the connector test surface; ships as a minor release (only the test harness changes). Both standard-test paths (airbyte-cdk connector test and airbyte-cdk image test) now run each config listed in acceptance-test-config.yml with the command of the section that lists it, apply the CAT succeed default to every command, and assert the CONNECTION_STATUS that check reports against the declared status. Connector CI installs the CDK CLI unpinned (uv tool install --upgrade 'airbyte-cdk[dev]'), so every Python and manifest-only connector picks this up with the next release, without a per-connector CDK bump. See the ## Upgrading to 7.36.0 section in cdk-migrations.md, shared by every standard-test PR of this release.

This PR also ships #1218. #1218 (per-stream records and record schemas in the basic read test) was stacked on this branch and merged into it on 2026-10-13, so merging this PR lands both. See Includes #1218 below.

Summary

Two gaps in the standard tests, one fix each.

check could not fail. A failing check reports status: FAILED in a CONNECTION_STATUS message and exits 0, so neither path could tell a failing check from a passing one: run_test_job (in-process) only looked for FAILED inside the expect_exception() branch, and the Docker path only asserted SUCCEEDED under expect_success(), so a config without a declared status accepted a failing check. This PR extracts one assertion, airbyte_cdk/test/standard_tests/_assertions.py::assert_check_outcome, calls it from both paths so they cannot drift again, and makes it follow the CAT status contract exactly:

status in acceptance-test-config.yml	check must
succeed	report SUCCEEDED
failed	report FAILED, gracefully (a raise is not a graceful failure)
exception	raise: a TRACE error and no SUCCEEDED; no CONNECTION_STATUS required
not declared	treated as succeed (the CAT default)

Half of acceptance-test-config.yml was never read, and the other half ran the wrong commands. Only spec, connection and basic_read were collected, every config found there ran every test, and configs listed only under discovery or full_refresh were never tested at all. Now every section with a tests list contributes its configs, except incremental, and each config runs the command CAT ran it with (SECTION_COMMANDS):

Section	The config must	Tests
spec	validate against the connector's connectionSpecification (CAT test_config_match_spec)	test_config_matches_spec (new), test_docker_image_build_and_config_matches_spec (new)
connection	make check report the declared status	test_check, test_docker_image_build_and_check
discovery	make discover succeed and return streams	test_discover, test_docker_image_build_and_discover (new)
basic_read, full_refresh	make a full-refresh read succeed with records. The read covers the streams of the config's configured_catalog_path, or every discovered stream when none is declared, minus empty_streams	test_basic_read, test_fail_read_with_bad_catalog, test_docker_image_build_and_read

A config listed under several sections runs the union of their commands and nothing else: a spec-only config is no longer checked, a connection-only config is no longer read. Tests that do not apply skip with a reason naming the config's sections. The succeed default now applies to every command, so for a status-less config discover and read errors, and a read without records, fail the test; ALLOW_ANY used to swallow them on the in-process path. The Docker read test keeps its --read-scenarios default (config, valid_config, default), so other read-section configs are read by airbyte-cdk connector test (manifest-only connectors in the monorepo) but not by airbyte-cdk image test (Python connectors).

Changes

• ExpectedOutcome gains EXPECT_UNCAUGHT_ERROR for status: exception, which is no longer an alias of failed. expect_exception() still covers both, so every existing caller keeps its behaviour; the new expect_uncaught_error() tells them apart. Folding exception into failed would have turned source-gcs and source-marketo red: both raise on integration_tests/invalid_config.json and emit a TRACE with no status, which is exactly what CAT's exception meant.
• assert_check_outcome implements the status table. The "no CONNECTION_STATUS" failure includes the formatted trace error and a hint to declare status: exception when raising is intended. In-process run_test_job("check") surfaces an uncaught error for every scenario that is not failed / exception, mirroring raise_if_errors on the Docker path. test_check no longer demands a CONNECTION_STATUS for status: exception scenarios (and the ConnectorTestSuiteBase message was not an f-string).
• ConnectorTestScenario: sections records where a config is listed (get_scenarios reads every section but incremental, _dedup_scenarios unions them), unknown sections are rejected, commands / runs() derive the commands from SECTION_COMMANDS, and with_default_success() applies the succeed default. skip_unless_runs(scenario, *commands) gates each test.
• assert_config_matches_spec (jsonschema, already a CDK dependency) validates a config against the reported spec; the offending value is replaced by <config value> in the failure so secrets do not land in CI logs.
• The incremental section is not collected, as on main. An earlier revision read its configs incrementally (in-process test_incremental_read, and incremental mode in the Docker read). Without input state, that read sends the same requests and returns the same records as a full-refresh one, so it only added read time (a second read for configs also under basic_read, 15+ minutes for twilio's lookback config) and reds from stale secrets. Real incremental coverage (reading with future_state) is left for a follow-up.
• Docker path: the three tests shared a copy-pasted image build, now _connector_image; new discover and config-vs-spec tests; get_discovered_streams is shared with the in-process tests. Fixed in passing: empty_streams crashed the Docker read with AttributeError: 'str' object has no attribute 'name' (stream names were filtered as if they were streams), on main too.
• configured_catalog_path is honoured, as in CAT (get_read_streams, shared by test_basic_read and test_docker_image_build_and_read). The read covers the streams named in the config's declared catalog, in its order and once each, with the definitions and JSON schemas discover reports, all in full-refresh mode whatever the catalog says. Without a declared catalog, every discovered stream is read, minus empty_streams as before. When several read entries list the same config, _dedup_scenarios keeps the catalog only if every read entry declares the same one; otherwise (one entry declares none, or they differ) every discovered stream is read. A catalog stream that discover does not report, or a missing catalog file, fails the read test with a message naming the catalog. The Docker read now configures every stream as full_refresh instead of the first mode it supports. The bundled pokeapi fixture gains the integration_tests/configured_catalog.json its config already pointed at.
• cdk-migrations.md: the ## Upgrading to 7.36.0 section with the two tables, the configured_catalog_path behaviour, migration steps and rationale. It also holds the sections of feat(standard-tests): assert discovered catalog properties in test_discover #1217, feat(standard-tests): assert per-stream records and record schemas in the basic read test #1218, feat(standard-tests): lint airbyte_secret marking and secret leaks #1219 and feat(standard-tests): check spec backward compatibility against the published version #1220, so all five ship in the 7.36.0 release.

Includes #1218

#1218 was merged into this branch (f0fad4e9), so this PR carries its change too. Full description, fleet measurement and rollout notes: #1218.

• SourceTestSuiteBase.test_basic_read checks, for configs listed under basic_read and expected to succeed, that every configured stream returns records (empty_streams are the exceptions) and that every record matches its stream's JSON schema (CAT's Draft 7 validator: strict integers, lenient date-time, extra columns allowed).
• validate_schema: false in a basic_read entry turns the schema check off for that config, except at test_strictness_level: high, where the test fails before reading, as in CAT.
• Failure messages name the stream, record number, record path and schema rule, and never print record values.
• New module airbyte_cdk/test/standard_tests/_read_assertions.py, tests in unit_tests/test/test_read_assertions.py.

Merged with main

main gained #1217 (catalog checks in test_discover), #1219 (spec lint and secret leak check) and #1220 (spec backward compatibility) after this branch was cut. Merging them in (b4e470b5) combined:

• test_discover keeps this PR's section gate and runs feat(standard-tests): assert discovered catalog properties in test_discover #1217's catalog checks. An empty catalog now fails feat(standard-tests): assert discovered catalog properties in test_discover #1217's enforced "catalog has streams" check (AssertionError) instead of this PR's ValueError; test_discover_fails_on_an_empty_catalog expects that.
• test_check and test_docker_image_build_and_check run feat(standard-tests): lint airbyte_secret marking and secret leaks #1219's secret leak check first, then this PR's assert_check_outcome. On the Docker path, raise_if_errors is applied after the leak check, still only for scenarios that do not expect a failure.
• test_docker_image_build_and_spec uses this PR's _connector_image and feat(standard-tests): lint airbyte_secret marking and secret leaks #1219's spec lint.

Test Coverage

• unit_tests/test/test_check_assertions.py (replacing unit_tests/test/test_docker_base_check_outcome.py): 19 direct cases over the four outcomes × (SUCCEEDED / FAILED / none) × (trace error / none), each asserting the failure reason via match=; 17 in-process cases through run_test_job with a fake source; the same 17 through SourceTestSuiteBase.test_check and ConnectorTestSuiteBase.test_check; 9 through test_docker_image_build_and_check with run_docker_airbyte_command stubbed (no Docker needed).
• unit_tests/test/test_scenario_sections.py: the section-to-command table; get_scenarios over a six-section config (incremental ignored, dedup, iam_role and config_path-less entries skipped, status carried, commands per config); commands / runs() across derived scenarios; unknown-section rejection; with_default_success; the exact skip_unless_runs message; a 5 tests × 5 sections matrix for the source suite and a 4 × 5 matrix for the Docker suite proving each test runs exactly for its sections (plus hand-built scenarios running everything); config-vs-spec validation on both paths with the secret value redacted; test_discover / test_basic_read failing on errors, an empty catalog and missing records for a status-less config; the Docker discover test (pass, empty catalog, trace error); the Docker read catalog per section (full refresh for basic_read / full_refresh / mixed), with empty_streams set; a declared configured_catalog_path on both read paths (the catalog's streams in its order, once each, forced to full refresh, with the discovered schema rather than the catalog's stale one, minus empty_streams, narrowed by --read-from-streams on Docker), a catalog stream that is not discovered, a missing catalog file, and the _dedup_scenarios catalog merge (same catalog kept; one entry without a catalog, or differing catalogs, read every stream; non-read entries ignored).
• The bundled pokeapi suite keeps its status: failed scenario (integration_tests/invalid_config.json) and now skips the tests that do not apply to each config, with the reason visible in -rs.

poetry run pytest unit_tests/test/ unit_tests/resources/source_pokeapi_w_components_py -m "not image_tests"   # 301 passed, 5 skipped (gated by section, by design)
poetry run ruff check . && poetry run ruff format --check . && poetry run mypy airbyte_cdk

After the merge with main (b4e470b5): unit_tests/test, test_entrypoint.py, destinations/test_destination.py, utils/test_traced_exception.py and legacy/sources/declarative/test_manifest_declarative_source.py: 925 passed, 9 skipped; ruff check, ruff format --check and mypy airbyte_cdk/test/standard_tests are clean.

Connectors affected on the next release

Scanned at airbytehq/airbyte@master (d71678a6fb2, 2026-10-12); counts leave out archived connectors. Local verification ran each config through its own command with the published connector image and, where needed, the GSM secret. "Gated" means the Docker read test's --read-scenarios default skips the config, so nothing runs for it in CI (Python connectors only run airbyte-cdk image test). Java connectors (source-mysql ×3, source-singlestore, source-scaffold-java-jdbc) are inert: the standard tests never run for them.

Companion connector fix PR: airbytehq/airbyte#88480 (source-zoho-crm, source-microsoft-lists, source-mixpanel; merged).

Known red on release, and their fixes:

Connector	Config	Why	Fix
destination-motherduck (certified)	secrets/config.json, connection, status: succeed	Already red without this PR: MotherDuck rejects the token ("Your request is not authenticated. Please check your MotherDuck token."). Check: destination-motherduck fails 2 tests at this head (test_docker_image_build_and_check, and the in-process test_check, which used to accept a reported FAILED) against 1 on a PR based on main (#1224, test_docker_image_build_and_check only). It is red on every PR.	Refresh the secret.
source-zoho-crm (Python)	integration_tests/invalid_config.json, status: failed	Verified with airbyte/source-zoho-crm:0.3.3: check exits 1 with one TRACE system_error ("Error while refreshing access token: 'access_token'") and no CONNECTION_STATUS. The old Docker assertion accepted that as the expected failure; failed now needs a reported FAILED.	airbytehq/airbyte#88480 declares status: exception for the invalid config (airbyte-cdk image test --no-creds: 1 failed → 2 passed).
source-microsoft-lists (manifest-only)	secrets/config.json, connection, no status	Must now succeed, but Entra rejects the GSM secret with HTTP 401 AADSTS7000215 (invalid client secret; the secret dates from 2025-04).	airbytehq/airbyte#88480 declares status: failed for it until the secret is rotated.
source-mixpanel (Python)	secrets/config.json, secrets/config_project_secret.json (connection, status: succeed); secrets/config_incremental.json (discovery)	Every test project gets HTTP 402 Your plan does not allow API calls: check fails on cohorts/list for both succeed configs (the old lenient Docker check hid it), and the first container discover fails on events/properties/top.	airbytehq/airbyte#88480 declares both connection configs status: failed and bypasses the discovery section, keeping the original entry in a comment (3 failed → 4 passed).
source-sentry (manifest-only)	secrets/config_limited_scopes.json, full_refresh	Now reads its catalog's 3 streams (project_detail, projects, releases); projects still returns 403 Forbidden with the limited-scope token.	airbytehq/airbyte#88412 (approved, not merged) lists the 403 streams under empty_streams.
source-hubspot (manifest-only)	secrets/config_oauth_no_start_date.json, full_refresh	Now reads its catalog's 24 streams instead of all 49, but campaigns, companies and contact_lists are in it and fail with 403 (missing crm.lists.read and other scopes); CAT's own read of this catalog fails the same way.	airbytehq/airbyte#88412 removes the entry.
source-vitally (manifest-only)	secrets/config.json, basic_read + full_refresh	Its integration_tests/configured_catalog.json lists nps_responses, which the manifest no longer defines, so the read test fails with "lists streams that connector 'source-vitally' does not discover" (found by a static scan, not run).	Not fixed: source-vitally has no test secret in CI (no testSecrets in metadata.yaml), so its standard tests already fail on any PR that touches it, before reading. Drop the stream from the catalog when a secret is added.

status: failed configs of Python connectors: stricter Docker check (38 configs in 35 connectors). The old Docker assertion accepted an uncaught error with no CONNECTION_STATUS as the expected failure; now failed needs a reported FAILED, and a raise needs status: exception. Verified with each connector's published image (all 38 invalid configs are committed files, so no secrets are needed): 37 report FAILED and pass; only source-zoho-crm (above) raises.

Configs that run less than before (listed only under spec; they used to run check, discover and read leniently):

Connector	Config	Now runs	Result
destination-motherduck (certified)	integration_tests/config.json	config vs spec	pass (verified). Its secrets/config.json is a separate connection config, see above.

Configs listed only under connection are no longer read, and discovery-only configs are no longer checked. Every other config listed under spec (for example in source-linkedin-ads, source-paypal-transaction, source-sendgrid, destination-duckdb and source-kyve) is also listed under connection, so it runs check as before.

status: exception configs: source-gcs, source-marketo and source-salesforce (config_error) raise on integration_tests/invalid_config.json and pass (they would be red if exception were folded into failed).

Status-less basic_read configs (they used to run everything leniently; now a full-refresh read must succeed with records, and check no longer runs for them):

Connector	Config	In CI	Result
source-gcs (config_jsonl.json), source-google-ads (config_click_view.json), source-s3 (parquet_dataset_config.json, v4_parquet_decimal_config.json, unstructured_config.json), source-shopify (config_transactions_with_user_id.json)	basic_read	gated (Python)	nothing runs for these configs any more
source-gitlab (manifest-only)	secrets/config_with_ids.json	in-process full read	pass (verified): 24 streams, 853 records, 0 errors, 29 s
source-recharge (Python)	secrets/config_order_modern_api.json	gated	nothing runs. For the record, a full read with this config exits 1 locally (Forbidden on bundle_selections, credit_adjustments, events), so it would be red if the gate were lifted.
source-tempo (manifest-only)	secrets/accounts_only_config.json	in-process full read	pass (verified): 4 streams, 16 records, 0 errors, 67 s

Configs collected for the first time (listed only under discovery or full_refresh):

Connector	Config	Section → command	Result
source-cart (config_central_api_router.json), source-google-ads (config_manager_account.json)	full_refresh	read, gated (Python)	nothing runs; the check reds of an earlier revision (HTTP 500, "no stream slices") do not apply
source-mixpanel (Python)	secrets/config_incremental.json	discovery → Docker discover	red, see above
source-monday, source-sendgrid (manifest-only)	secrets/config.json	discovery → discover	pass (8 and 15 streams)
source-zendesk-chat (manifest-only)	secrets/config_old.json	discovery → discover	pass (12 streams); the check failure of an earlier revision ('dict object' has no attribute 'credentials') does not apply because discover does not touch the credentials
source-tiktok-marketing (manifest-only)	secrets/config_oauth.json, secrets/prod_config.json	discovery → discover	unverifiable here: the reviewer's proxy answers business-api.tiktok.com with a 403 block page; CI has real egress
source-sentry, source-hubspot (manifest-only)	see above	full_refresh → in-process read of the declared catalog	red, see above

First container discover for 30 Python discovery configs (test_docker_image_build_and_discover is new). The Docker read, which runs only the config, valid_config and default configs, already ran discover in a container; these 30 never did: source-s3 ×15, source-azure-blob-storage ×12, source-gcs ×2 and source-mixpanel (red, see above). Verified with airbyte-cdk image test from this branch with secrets, on the published source-s3 4.15.26 and source-azure-blob-storage 0.8.37 images: all 15 s3 and 12 azure-blob-storage configs pass discover and check (s3: 36 passed, 53 skipped; azure-blob-storage: 28 passed, 29 skipped; reads skipped by the --read-scenarios default). The two gcs configs were not run. (Counting archived connectors, source-google-analytics-v4 and source-fauna add one each.)

configured_catalog_path now decides what is read. 174 read configs (153 manifest-only, 20 Python, 1 Java) declare the same catalog in every read entry, and now read only that catalog's streams instead of every discovered stream. For most the catalog lists every stream anyway; a narrower catalog reads less than at the previous head (which main read leniently). The 80 configs whose basic_read entry declares no catalog while their full_refresh entry does keep reading every discovered stream. A static scan of the manifest-only catalogs against their manifests found one stale catalog (source-vitally, above); the Python catalogs and the dynamic-stream manifests (source-hubspot, source-bing-ads) cannot be checked statically.

Merge order

Merging into main does not publish anything; the CDK release is the gate.

• feat(standard-tests): assert discovered catalog properties in test_discover #1217, feat(standard-tests): lint airbyte_secret marking and secret leaks #1219, feat(standard-tests): check spec backward compatibility against the published version #1220 merged into main, and feat(standard-tests): assert per-stream records and record schemas in the basic read test #1218 into this branch.
• Connector PRs for feat(standard-tests): assert per-stream records and record schemas in the basic read test #1218's checks, all merged: test(connectors): declare empty streams and schema opt-outs for basic read assertions airbyte#88406, test(source-amazon-seller-partner): declare an empty stream for basic read assertions airbyte#88477, fix(source-freshdesk): emit sla_policies ids as integers airbyte#88407, fix(source-sentry): type the sentry:replay_rage_click_issues project option as boolean airbyte#88408, fix(source-jira): type rendered comment bodies in issues as strings airbyte#88409, fix(source-shopify): keep the countries dedup list per stream instance airbyte#88405, fix(source-customer-io)!: 🚨🚨 declare integer from_id, reply_to_id and sent_at and typed array items 🚨🚨 airbyte#88159.
• Connector PRs for this PR's checks, all merged: test(source-sentry, source-hubspot): fix limited-scope full_refresh configs airbyte#88412, test(source-zoho-crm, source-microsoft-lists, source-mixpanel): fix configs for stricter standard-test checks airbyte#88480.
• Before merging: confirm the draft release is still v7.36.0 and unpublished (the migration guide heading says 7.36.0).
• Merge this PR.
• Re-run the manifest-only sweep on current master (test-account data drifts).
• Publish the v7.36.0 release.

Declarative-First Evaluation

N/A — this is a CDK test-harness change, not a connector change.

Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/17140

airbytehq/airbyte-internal-issues#16212 is not closed by this PR: its gap 2 (EXPECT_SUCCESS not asserting the reported status) is fixed here, but PR #991 and the rest of that issue remain open.

Link to Devin session: https://app.devin.ai/sessions/a3d02c9188e44c789e336784dced8888
Open in Devin Desktop: https://app.devin.ai/desktop/session/a3d02c9188e44c789e336784dced8888?variant=devin

🤖 Generated with Claude Code

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1788275534-shared-check-assertion#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1788275534-shared-check-assertion

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[devin-ai-integration]

CI note on the Check: destination-motherduck job (real evidence for the blast-radius discussion above):

That job is already red on main-based PRs for credential reasons — e.g. https://github.com/airbytehq/airbyte-python-cdk/actions/runs/33446043952 and https://github.com/airbytehq/airbyte-python-cdk/actions/runs/33192062338 both fail with Your request is not authenticated. Please check your MotherDuck token and PermissionError(13, 'Permission denied'). There it fails 2 tests (the Docker-path test_docker_image_build_and_check scenarios).

On this PR the same job fails 4 tests: the same 2 plus the two in-process test_check scenarios, which previously passed despite the connector reporting Status.FAILED. That is precisely the bug this PR fixes — the in-process path was silently green on a failing check — so I don't think the extra failures indicate a regression, though it does show the change makes an already-broken credential setup fail more loudly.

Devin session

[github-actions]

PyTest Results (Fast)

5 748 tests  +263   5 718 ✅ +245   10m 10s ⏱️ +31s
    1 suites ±  0      30 💤 + 18 
    1 files   ±  0       0 ❌ ±  0 

Results for commit b4e470b. ± Comparison against base commit 57e4c07.

This pull request removes 11 and adds 274 tests. Note that renamed tests count towards both.

unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[allow_any_failed]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[allow_any_no_status]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[allow_any_succeeded]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[failure_failed]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[failure_last_status_wins]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[failure_no_status]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[failure_succeeded]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[success_failed]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[success_last_status_wins]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[success_no_status]
…

unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_basic_read['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_check['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_config_matches_spec['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_config_matches_spec['valid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_discover['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_build_and_check['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_build_and_config_matches_spec['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_build_and_config_matches_spec['valid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_build_and_discover['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_build_and_discover['valid_config' Test Scenario]
…

♻️ This comment has been updated with latest results.

[Copilot]

Pull request overview

Unifies and tightens how Standard Tests assert the outcome of the check command so the in-process and Docker-based paths enforce the same expectations and don’t drift.

Changes:

• Extracts shared check assertion logic into airbyte_cdk/test/standard_tests/_assertions.py::assert_check_outcome.
• Updates both Docker (docker_base) and in-process (_job_runner.run_test_job) paths to use the shared assertion.
• Replaces/updates unit tests to cover the full outcome matrix (including end-to-end via run_test_job).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
unit_tests/test/test_docker_base_check_outcome.py	Removes old unit tests tied to the Docker-only assertion helper.
unit_tests/test/test_check_assertions.py	Adds comprehensive unit tests for shared check assertion and in-process coverage.
airbyte_cdk/test/standard_tests/docker_base.py	Switches Docker check path to call the shared assert_check_outcome.
airbyte_cdk/test/standard_tests/_job_runner.py	Switches in-process check path to call the shared assert_check_outcome.
airbyte_cdk/test/standard_tests/_assertions.py	Introduces shared assertion implementation for check outcome validation.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

PyTest Results (Full)

5 248 tests  +182   5 227 ✅ +174   13m 11s ⏱️ + 3m 24s
    1 suites ±  0      21 💤 +  9 
    1 files   ±  0       0 ❌  -   1 

Results for commit a07a090. ± Comparison against base commit 42f6023.

This pull request removes 11 and adds 193 tests. Note that renamed tests count towards both.

unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[allow_any_failed]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[allow_any_no_status]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[allow_any_succeeded]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[failure_failed]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[failure_last_status_wins]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[failure_no_status]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[failure_succeeded]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[success_failed]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[success_last_status_wins]
unit_tests.test.test_docker_base_check_outcome ‑ test_assert_check_outcome[success_no_status]
…

unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_basic_read['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_check['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_config_matches_spec['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_config_matches_spec['valid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_discover['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_build_and_check['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_build_and_config_matches_spec['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_build_and_config_matches_spec['valid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_build_and_discover['invalid_config' Test Scenario]
unit_tests.resources.source_pokeapi_w_components_py.integration_tests.test_airbyte_standards.TestSuiteSourcePokeAPI ‑ test_docker_image_build_and_discover['valid_config' Test Scenario]
…

♻️ This comment has been updated with latest results.

[Daryna Ishchenko (darynaishchenko)]

Requesting changes for the release title (inline) and the rollout. The harness work itself is solid. 182 tests pass, all 24 deliberate code breaks I tried were caught, and a probe of 8 jsonschema error kinds leaked no secret value into the config-vs-spec messages. A static sweep of get_scenarios() over 590 monorepo connectors (head vs merge-base) matches the intent: 10 configs collected for the first time, 9 read configs now strict, and the same collection errors before and after.

Also, please fix the fleet table in the PR body, since connector owners will read it as the release note:

• destination-motherduck is not resolved: secrets/config.json is under connection with status: succeed, and Check: destination-motherduck now fails 2 tests at this head vs 1 on baseline. Only integration_tests/config.json is spec-only.
• The "Configs that run less than before (listed only under spec or connection)" row is wrong for linkedin-ads, paypal-transaction, sendgrid, duckdb and kyve. After deduplication each of those is also under connection, so it runs as before.
• Two affected groups are missing: the stricter Docker check for 38 Python status: failed configs, and the first-time container discover for 31 Python discovery configs (see the comment on docker_base.py).

[devin-ai-integration]

❌ Cannot revive Devin session - the session is too old. Please start a new session instead.

View session