feat(low-code): add regex and copy options to KeysReplace

[devin-ai-integration]

Summary

Adds four optional fields to the declarative KeysReplace transformation so low-code connectors can drop custom key-mapping components (e.g. source-hubspot's NewtoLegacyFieldTransformation). All defaults reproduce today's behavior (literal match, recursive, rename), so existing manifests are unaffected.

Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/17742:

• https://github.com/airbytehq/airbyte-internal-issues/issues/17742

Close on merge: yes — internally filed, no customer to confirm with.

Part of epic https://github.com/airbytehq/airbyte-internal-issues/issues/17701. Requested via /ai-fix on the issue above (issue author: Anatolii Yatsuk (@tolik0)). The HubSpot migration is a separate sub-issue and is not part of this PR.

Field	Default	Behavior
regex	false	old is a regex, new a re.sub replacement (\1, \g<1>, \g<name>). Invalid pattern / group reference raises ValueError when the component is created (or at first transform if old/new are interpolated).
keep_original	false	Copy the value to the new key and keep the original (dict/list values are deep-copied).
only_if_missing	false	Only write the new key if absent or null. In rename mode the original key is left as-is when it is not written.
field_path	whole record	Scope to the object(s) at this path; same semantics as DpathFlattenFields.field_path (config-interpolated, * wildcard). Missing path / non-object → no-op.

Example (equivalent to the HubSpot contacts mapping, including "don't double-append _date"):

- type: KeysReplace
  old: "hs_v2_date_entered_(.*?)(?:_date)?$"
  new: "hs_lifecyclestage_\\1_date"
  regex: true
  keep_original: true
  only_if_missing: true
  field_path: ["properties"]   # omit inside DynamicSchemaLoader.schema_transformations (bare properties map)

Semantics / implementation notes

• Recursion is unchanged: the replacement still applies to nested objects, now relative to each field_path target. With keep_original, nested objects are transformed first, then copied.
• Keys are evaluated from a snapshot, so newly written keys are not re-processed (e.g. a -> aa with keep_original yields {a, aa}, not aaa).
• Non-interpolated old/new are used verbatim in regex mode: JinjaInterpolation runs ast.literal_eval on rendered output, which would turn a pattern like (1) into 1. Interpolated values are evaluated with valid_types=(str,).
• Default-mode collision behavior (last write wins) is intentionally unchanged.
• Docs: the component reference on docs.airbyte.com is rendered from declarative_component_schema.yaml, so the new descriptions and HubSpot-derived examples live there.
• Model regenerated with poe assemble; unrelated generator churn was reverted so only the KeysReplace fields are in the diff.

Reproduction

New tests run against unmodified main code (implementation stashed): 28 failed, 15 passed — the new options are silently ignored by the model and the transformation always does a literal rename (e.g. hs_v2_date_entered_poetry is lost on collision with hs_date_entered_poetry).

Declarative-First Evaluation

Not a connector change: this PR extends a built-in declarative component in the CDK itself so that connectors can replace a custom Python component with manifest-only config. No custom connector component is added. Existing alternatives (AddFields with dynamic keys, KeysToLower/KeysToSnakeCase, DpathFlattenFields) can't do regex-based key copy with "only if missing" semantics on a sub-object or a dynamic schema map.

Test Coverage

• unit_tests/sources/declarative/transformations/test_keys_replace_transformation.py: defaults (literal/recursive/rename/collision), regex \1 / \g<1> / \g<name>, backslashes in pattern, plain regex not literal-evaluated, interpolated regex, keep_original (incl. no duplication of unmatched keys, no aliasing), only_if_missing absent/null/present in both modes and key orders, field_path scoping/interpolation/wildcard/missing/non-dict, snapshot semantics, invalid regex at construction and at transform.
• unit_tests/sources/declarative/parsers/test_model_to_component_factory.py: defaults, all options parsed from YAML (backreference survives YAML parsing), invalid regex raises at creation.
• unit_tests/sources/declarative/schema/test_dynamic_schema_loader.py: end-to-end discover with KeysReplace in schema_transformations.
• Verified new tests fail without the change (28 failures) and pass with it.
• poetry run pytest unit_tests/sources/declarative: 2375 passed; 1 failure in test_jwt.py::test_get_request_options, a timing flake (iat off by 1s) that passes on rerun and is unrelated.
• Locally compared against HubSpot's NewtoLegacyFieldTransformation (deals and contacts mappings) on 12,000 randomized records/schemas: identical output.
• ruff check, ruff format --check, mypy on changed sources: clean.

Breaking change evaluation

Non-breaking: all new fields are optional with defaults that preserve current behavior; no existing fields changed or removed. CDK version is set at release time (feat: → minor).

Link to Devin session: https://app.devin.ai/sessions/bf3541d200a04648bbe2115190d3c03f
Open in Devin Desktop: https://app.devin.ai/desktop/session/bf3541d200a04648bbe2115190d3c03f?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1791298896-keys-replace-regex-copy-options#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1791298896-keys-replace-regex-copy-options

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 056 tests  +31   5 044 ✅ +31   11m 26s ⏱️ +22s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 589231a. ± Comparison against base commit 5381b8f.

[github-actions]

PyTest Results (Full)

5 059 tests  +31   5 047 ✅ +31   16m 17s ⏱️ + 3m 37s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 589231a. ± Comparison against base commit 5381b8f.