fix(low-code): keep $parameters out of JSON schemas with a list type

[Baz (bazarnov)]

Overview

👉 TL;DR: Parameters declared on a low-code stream are no longer copied into its inline record schema when the schema's type is a list other than exactly "null, object", such as "object, null" or just "object". They are also no longer copied into a schema's list of fields when one of the fields is named "type", where they could show up as extra columns. Specifically, ManifestComponentTransformer._is_json_schema_object now treats any list or dict type as JSON schema content, so propagate_types_and_parameters returns those mappings unchanged.

Changes

• _is_json_schema_object returns True for a list or dict type, not only for "object" and the exact list ["null", "object"]. Component types are always strings, so a list type is a JSON schema type union in any order, and a dict type is a properties map holding a property named type.
• test_do_not_propagate_parameters_on_json_schema_object runs over five schemas: root type "object", ["null", "object"], ["object", "null"] and ["object"], plus a root without type that has a type property. Each comes back unchanged while the stream and its InlineSchemaLoader still receive the parameters.

Review Spotlight

Reviewers with limited time, please review first:

• manifest_component_transformer.py#L193-L198: the predicate; the dict case stops the walk at a properties map reached from a schema root without type
• manifest_component_transformer.py#L131-L137: its only call site, reached for the whole manifest and again for each resolved dynamic stream

Risks

• Cleaner discovered schemas in 2 monorepo connectors, once they move to a source-declarative-manifest base image with this fix. Custom manifests with these spellings and parameters in scope get the same cleanup.

  • source-jira labels (root type: ["object", "null"]): path and $parameters leave the schema root and properties.label.
  • source-zendesk-support sla_policies (root type: ["object"]): name, path, primary_key, cursor_field and $parameters leave the root and 8 property schemas.

  No property is added or removed and no type changes, so destination columns stay the same. At most, a one-time non-breaking schema change is detected on these 2 streams.

• Other mappings with a list or dict type are now left as written instead of receiving the parameters, and components nested under them no longer receive them either. No monorepo manifest is affected: the comparison below shows no other change.

• How the blast radius was counted. All 537 manifest.yaml files under airbyte-integrations/connectors on airbyte master at fc800755 went through ManifestReferenceResolver and propagate_types_and_parameters once with main and once with this branch. Only the 2 manifests above differ, and only by removed keys inside schema_loader.schema (plus their unused definitions copies).

  • 91 other InlineSchemaLoader schemas in stream definitions use these spellings (24 ["object", "null"], 67 ["object"]). No parameters are in scope there, so their output does not change.
  • In the 9 manifests with dynamic_streams, the only list-typed schemas in templates and resolver definitions outside an object schema are 6 ["null", "object"] roots in source-linkedin-ads, which main already skips. Values a resolver fetches from an API at runtime are not covered by this count.

• Unchanged: a mapping with a string type other than "object", such as a schema root type: "array", is still treated as a component.

Test plan

• New test cases:

  python -m pytest "unit_tests/sources/declarative/parsers/test_manifest_component_transformer.py::test_do_not_propagate_parameters_on_json_schema_object"
  

  • With the source file from main (git checkout origin/main -- airbyte_cdk/sources/declarative/parsers/manifest_component_transformer.py): 3 failed (object_null, object_only, no_root_type_with_type_property), 2 passed.
  • This branch: 5 passed.
• python -m pytest unit_tests/sources/declarative/parsers unit_tests/sources/declarative/resolvers unit_tests/legacy: 583 passed.
• python -m pytest unit_tests/sources/declarative/test_concurrent_declarative_source.py: 86 passed, 3 failed. The 3 are test_read_with_concurrent_and_synchronous_streams*, which hit sqlite3.OperationalError: database table is locked in the shared HTTP cache locally and crash the interpreter with the main source file as well (Cached parent stream with more than one partition corrupts the shared sqlite request cache under concurrent reads (no such table: responses, file is not a database, no more rows available) #1150).
• python -m pytest unit_tests/sources/declarative unit_tests/connector_builder unit_tests/source_declarative_manifest -k "not test_read_with_concurrent_and_synchronous_streams": 2408 passed.
• ConcurrentDeclarativeSource.discover on a static stream and a dynamic stream template, with $parameters above an InlineSchemaLoader whose schema has id and type properties:
  • main: roots ["object", "null"] and ["object"] come back with name and $parameters on the root, on each property and as extra entries in properties; ["null", "object"] comes back as written.
  • This branch: all three come back as written.
• ruff format and ruff check on the 2 changed files and mypy --config-file mypy.ini on the source file: clean.
• Local runs added -p no:cacheprovider; the 2408-test run also added -p no:memray, since pytest-memray needs the cache plugin.

Fixes #1137

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@bazarnov/fix-parameters-json-schema-type#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch bazarnov/fix-parameters-json-schema-type

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

4 986 tests  +4   4 974 ✅ +4   7m 44s ⏱️ - 1m 49s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 27b25df. ± Comparison against base commit d214b0c.

This pull request removes 1 and adds 5 tests. Note that renamed tests count towards both.

unit_tests.sources.declarative.parsers.test_manifest_component_transformer ‑ test_do_not_propagate_parameters_on_json_schema_object

unit_tests.sources.declarative.parsers.test_manifest_component_transformer ‑ test_do_not_propagate_parameters_on_json_schema_object[no_root_type_with_type_property]
unit_tests.sources.declarative.parsers.test_manifest_component_transformer ‑ test_do_not_propagate_parameters_on_json_schema_object[null_object]
unit_tests.sources.declarative.parsers.test_manifest_component_transformer ‑ test_do_not_propagate_parameters_on_json_schema_object[object]
unit_tests.sources.declarative.parsers.test_manifest_component_transformer ‑ test_do_not_propagate_parameters_on_json_schema_object[object_null]
unit_tests.sources.declarative.parsers.test_manifest_component_transformer ‑ test_do_not_propagate_parameters_on_json_schema_object[object_only]

[github-actions]

PyTest Results (Full)

4 989 tests  +4   4 977 ✅ +4   16m 26s ⏱️ +22s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 27b25df. ± Comparison against base commit d214b0c.

This pull request removes 1 and adds 5 tests. Note that renamed tests count towards both.

unit_tests.sources.declarative.parsers.test_manifest_component_transformer ‑ test_do_not_propagate_parameters_on_json_schema_object

unit_tests.sources.declarative.parsers.test_manifest_component_transformer ‑ test_do_not_propagate_parameters_on_json_schema_object[no_root_type_with_type_property]
unit_tests.sources.declarative.parsers.test_manifest_component_transformer ‑ test_do_not_propagate_parameters_on_json_schema_object[null_object]
unit_tests.sources.declarative.parsers.test_manifest_component_transformer ‑ test_do_not_propagate_parameters_on_json_schema_object[object]
unit_tests.sources.declarative.parsers.test_manifest_component_transformer ‑ test_do_not_propagate_parameters_on_json_schema_object[object_null]
unit_tests.sources.declarative.parsers.test_manifest_component_transformer ‑ test_do_not_propagate_parameters_on_json_schema_object[object_only]