perf(declarative): skip runtime metaschema check of the component schema

[devin-ai-integration]

Summary

_validate_source() in ConcurrentDeclarativeSource and the legacy ManifestDeclarativeSource called jsonschema.validators.validate(manifest, schema). In jsonschema 4.x that first runs cls.check_schema(schema), which checks the CDK's own shipped declarative_component_schema.yaml against the Draft 7 metaschema again on every source build. It gets the same answer every time, and it runs a second time when _migrate_manifest validates again.

This PR builds the validator directly, keeps validate's error selection, and only checks the schema when validation fails:

validator_class = validator_for(schema)
try:
    error = best_match(validator_class(schema).iter_errors(manifest))
except Exception:
    validator_class.check_schema(schema)  # a broken schema is reported as SchemaError
    raise
if error is not None:
    validator_class.check_schema(schema)
    raise ValidationError("Validation against json schema defined in declarative_component_schema.yaml schema failed") from error

A valid manifest never reaches check_schema, so it gets the full saving. best_match is kept on purpose: Validator.validate() raises the first error, which is often a different one (see the issue).

The metaschema check also runs in CI: test_the_shipped_component_schema_is_a_valid_draft_7_schema.

Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/17933:

• https://github.com/airbytehq/airbyte-internal-issues/issues/17933

Close on merge: yes, internally filed, no customer to confirm with.

Part of https://github.com/airbytehq/airbyte-internal-issues/issues/17920. Requested by Patrick Nilan (Patrick Nilan (@pnilan)).

Timing

I timed validation directly. Each manifest is resolved with ManifestReferenceResolver and ManifestComponentTransformer, the best of 3 runs is kept, and the comparison is old validate() vs new best_match(iter_errors()). Setup: jsonschema 4.25.1, the 516 manifest.yaml files under airbytehq/airbyte@79bef47d, and a dev VM.

	p50	p90
Before: validate()	68.7 ms	94.6 ms
After: best_match(validator.iter_errors())	5.8 ms	32.0 ms
Draft7Validator.check_schema(component_schema) alone (min of 5)	64.1 ms

All 516 manifests are valid, so none of them reaches the failure-path check_schema. The outcome was identical for all 516. Absolute numbers on this VM are higher than the issue's 1-CPU container numbers (25 ms saving). The saved step is the same in both measurements. An invalid manifest still pays for check_schema once, as it did before.

Test Coverage

• test_the_shipped_component_schema_is_a_valid_draft_7_schema (test_declarative_component_schema_is_loadable.py): asserts that validator_for(schema) is Draft7Validator, then runs check_schema.
• test_invalid_manifest_raises_the_best_match_error_not_the_first (concurrent and legacy): with the shipped schema, the manifest has two errors. The first error from iter_errors is at $.streams[0] (a guard assertion checks this), and the best match is $.streams[1].retriever.requester.http_method (enum). The test checks that the wrapper message is unchanged. It also checks that __cause__ matches what jsonschema.validate raises on the same manifest in message, json_path, schema_path, validator and validator_value.
• test_building_a_source_does_not_check_the_component_schema (concurrent and legacy): patches Draft7Validator.check_schema to raise, then builds a valid source.
• test_a_broken_component_schema_that_fails_the_manifest_raises_schema_error (concurrent and legacy): HttpRequester.http_method gets enum: "GET", so a valid manifest fails validation. Building the source raises SchemaError.
• test_a_broken_component_schema_that_breaks_validation_raises_schema_error (concurrent and legacy): HttpRequester.path gets type: strng, so iter_errors raises UnknownType (a guard checks this). Building the source raises SchemaError.

Verified against unfixed code: both does_not_check tests fail on main, because validate() calls check_schema. The four broken-schema tests fail on this branch's first commit, which raised ValidationError and UnknownType instead. The best-match tests and the metaschema test pass on main too, which shows the error is the same as before. With the fix, all pass: 139 passed, 1 skipped across the three test files. On the first commit I also ran unit_tests/sources/declarative, unit_tests/legacy and unit_tests/connector_builder (2547 passed, 1 skipped). Ruff and mypy pass on the changed source files.

Risk

• Broken schema: a malformed component schema is still reported as SchemaError, because check_schema runs before any validation error or exception is raised. Only a CDK change can break the schema, since it ships inside the CDK wheel. The metaschema test catches that before release.
• Malformed schema that accepts the manifest: if a broken schema still lets the manifest validate, no error is raised at runtime. The metaschema test catches that case in CI.
• Out of scope: ValidateAdheresToSchema and schema_helpers are intentionally unchanged, because their schemas come from the manifest or the connector.

Link to Devin session: https://app.devin.ai/sessions/22162a3a9629407c95ac5ff107c3f0af
Open in Devin Desktop: https://app.devin.ai/desktop/session/22162a3a9629407c95ac5ff107c3f0af?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1791502007-skip-component-schema-check#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1791502007-skip-component-schema-check

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 072 tests  +38   5 060 ✅ +38   10m 2s ⏱️ -39s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 29b03be. ± Comparison against base commit 916d6f6.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

5 075 tests  +38   5 063 ✅ +38   15m 43s ⏱️ +11s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 29b03be. ± Comparison against base commit 916d6f6.

♻️ This comment has been updated with latest results.

[Patrick Nilan (pnilan)]

🤖 This comment was generated by an AI Agent.

The issue this PR resolves has one addition, under "Proposed change", "Risks to cover" and "Acceptance criteria" of airbytehq/airbyte-internal-issues#17933. Please update the PR for it.

Check the schema on the failure path

When validation finds an error, or raises, call check_schema before reporting, in both _validate_source implementations:

validator_class = validator_for(schema)
try:
    error = best_match(validator_class(schema).iter_errors(manifest))
except Exception:
    validator_class.check_schema(schema)  # a broken schema is reported as SchemaError
    raise
if error is not None:
    validator_class.check_schema(schema)
    raise ValidationError(...) from error

A valid manifest never reaches check_schema, so the saving is unchanged and test_building_a_source_does_not_check_the_component_schema still holds.

Why: a CDK developer who breaks declarative_component_schema.yaml locally gets an error that blames the manifest. On main each of these raises SchemaError; on this branch, with a valid manifest:

Mistake in the schema	Error on this branch
enum: GET where a list belongs, on HttpRequester.http_method	ValidationError: 'GET' is not one of 'GET'
type: strng	UnknownType
properties: []	AttributeError: 'list' object has no attribute 'items'

Please cover, for the concurrent and the legacy source:

• a schema that violates its metaschema and makes a manifest fail validation raises SchemaError;
• a schema that violates its metaschema and makes validation raise (type: strng) raises SchemaError;
• with the shipped schema, an invalid manifest raises the same error as it does now.

The "Risk" section of the description will need updating too.

[devin-ai-integration]

Patrick Nilan (@pnilan) Done in 29b03be, reply to this comment.

• Both _validate_source implementations now call validator_class.check_schema(schema) before raising. This happens when best_match finds an error, and when iter_errors raises (then the original exception is re-raised). A valid manifest never reaches check_schema, and test_building_a_source_does_not_check_the_component_schema still passes.
• New tests, for both the concurrent and the legacy source:
  • test_a_broken_component_schema_that_fails_the_manifest_raises_schema_error: HttpRequester.http_method gets enum: "GET". A guard asserts that the valid manifest now has errors. Building the source raises SchemaError.
  • test_a_broken_component_schema_that_breaks_validation_raises_schema_error: HttpRequester.path gets type: strng. A guard asserts that iter_errors raises UnknownType. Building the source raises SchemaError.
  • The shipped-schema case is the existing test_invalid_manifest_raises_the_best_match_error_not_the_first, unchanged. It compares the cause against what jsonschema.validate raises.
• All four new tests failed on the previous commit (ValidationError / UnknownType), and all pass now: 139 passed, 1 skipped across the three test files. Ruff and mypy are clean.

I updated the Summary, Test Coverage and Risk sections of the description to match.

Written by Devin