feat(RecordExpander): add `parent_fields` to copy named parent values onto expanded items by tolik0 · Pull Request #1165 · airbytehq/airbyte-python-cdk

Anatolii Yatsuk (tolik0) · 2026-09-16T14:34:35Z

What

Adds parent_fields to the existing RecordExpander: a list of named values to copy from the record being expanded onto each expanded item.

record_expander:
  type: RecordExpander
  expand_records_from_field: ["reviews", "nodes"]
  parent_fields:
    - type: ParentFieldPath
      parent_path: ["url"]
      record_path: ["pull_request_url"]

Also adds merge_parent, a boolean that shallow-merges the parent record, minus the expanded list, underneath each expanded item, the item's own keys winning on collision.

record_expander:
  type: RecordExpander
  expand_records_from_field: ["activity"]
  merge_parent: true

Why

RecordExpander can already carry parent context onto each expanded item, but only through remain_original_record: true, which deep-copies the whole parent once per item (record_expander.py, _apply_parent_context). For a GitHub pull request node with ten reviews that copies the entire pull request ten times in order to carry one URL. parent_fields copies only what is named.

This replaces #1164 (closed), which proposed a new NestedRecordExtractor component for the same job. An architecture review of #1149/#1162/#1163/#1164 found that DpathExtractor.record_expander — which has been on main since before #1164 was written, and which neither #1164 nor the source-github PR that motivated it mentioned — already reproduces both of source-github's custom extractors record-for-record across seven response shapes, including the four-level pull_request_comment_reactions traversal, when combined with AddFields and RemoveFields. What #1164 genuinely added over RecordExpander was the named copy. So the gap is real; a parallel component to close it is not.

Shipping #1164 would have left main with two ways to write "explode this list and stamp the parent onto each child", differing in which field carries the parent, whether the copy is whole or named, and whether truncation is handled — with no way for a Connector Builder user to choose between them, and every future feature needing to be added twice.

Adopters

source-github (branch tolik0/source-github/graphql-streams): NestedGraphQLRecordExtractor and DeepNestedGraphQLRecordExtractor are replaced by DpathExtractor with a RecordExpander using parent_fields, combined with the union mode of feat: add CombinedExtractor for combining multiple record extractors #1162 to cover the listing and drill-down documents, plus AddFields for repository and user.type. Verified by executing both on this branch against the connector's seven fixture shapes (reviews listing, drill-down, null repository; reactions from the repository root and from PullRequest, Review and Comment node roots): every shape matches record-for-record, and no RemoveFields step is needed because original_record is never embedded. About 120 lines of the connector's components.py; the two pagination strategies and ReleasesRecordTransformation stay.
source-zendesk-support events extractor (components.py:11-31): single-level flatten of ticket_events[].child_events[] copying three parent fields, with a RecordFilter on event_type == "Comment" after expansion.
source-mailchimp email_activity (components.py:16): {**record, **activity_item} for each item of record.pop("activity"). Replaced by expand_records_from_field: ["activity"] with merge_parent: true; nothing else is needed, since the child fields win and the activity key is removed exactly as the connector does. The factory test reproduces this shape end to end.

Checked and not covered by this PR alone, with the change that would cover them:

source-zendesk-talk IVR routes components.py:27 and source-hubspot associations components.py:894 (results[].to[].associationTypes[]) expand twice and stamp fields from both ancestors; that is chained expanders, see below.
source-hubspot property history components.py:144 expands the values of a dict and stamps the key onto each child; one adopter, and it interacts with truncated_list_retriever, which assumes a list.
source-orb subscription usage (components.py:16-55) is not a like-for-like replacement, although the fields line up with merge_parent. SubscriptionUsageTransformation.transform contains yield, so it is a generator function: the CDK calls it, discards the generator, and the body never runs. It would fail if it did (output = record.update(subrecord) is None, and del record["usage"] runs after the manifest's RemoveFields: [usage]). Production therefore emits one un-expanded record per response, without usage. Moving orb to expand_records_from_field: ["usage"] with merge_parent: true, a quantity RecordFilter, and AddFields/RemoveFields for the billable_metric unnest would emit one record per usage item, which changes the record count, primary key and cursor shape. That needs a breaking-change evaluation in the orb PR, and the mapping must also cover the metric_group un-nest the class does when subscription_usage_grouping_key is set.

Design decisions

parent_fields and remain_original_record are independent and may both be set. The review recommended that parent_fields replace the deep copy when present; I implemented them as orthogonal instead, because silently disabling a set option is surprising. The efficiency win comes from using parent_fields instead of remain_original_record, which the docstring and the schema description both state.
A parent_path the parent does not have copies None, rather than skipping the field. This matches parent.get(field) in the connector classes this replaces, and it means a present-but-null parent value and a missing one behave the same. Tests pin both.
An existing value at record_path is overwritten. Also matches the connector classes.
Globs are rejected in both paths, reusing RecordExpander._reject_globs and the rule already applied to truncation_indicator_path: each path must identify a single field. The shared message no longer claims that truncation handling is involved, since parent_fields reaches it too.
type: ParentFieldPath is optional in a manifest. RecordExpander.parent_fields is registered in manifest_component_transformer.DEFAULT_MODEL_TYPES, the same way AddFields.fields maps to AddedFieldDefinition, so a Builder user can write the entries without a type key. A factory test covers the untyped form end to end.
Copied containers and merged parent values are deep-copied per item. dpath.get returns the parent's object by reference and a shallow merge aliases every nested container, so without this a downstream AddFields with a multi-segment path would write into the parent record and into every item already yielded. For parent_fields both the lookup and the copy cost proportionally to the named value (the lookup reads only the containers on parent_path, see review round 3); for merge_parent it costs the parent minus the expanded list, which is still cheaper than remain_original_record. Scalars are passed through untouched.
The copy happens in _apply_parent_context, the one point both the embedded-items path and the truncated_list_retriever path pass through, so fetched items are stamped identically to embedded ones. Covered by a test.
Scalar list items. Previously a scalar was yielded bare unless remain_original_record was set, in which case it was wrapped as {"value": ..., "original_record": ...}. It is now wrapped as {"value": ...} plus the copied fields whenever any of the three parent-context options is set, and still yielded bare when none is. Both cases are tested, for merge_parent too.
Merge order is fixed: merge_parent, then parent_fields, then remain_original_record. The merge lays the parent underneath the item first, the named copies are written on top of that (so a parent_fields entry can overwrite a merged value, which is tested), and original_record is embedded last. All three options are independent and may be combined.
merge_parent removes only the expanded list from the parent's copy. For a single-segment path that is the top-level key; for a multi-segment path such as ["reviews", "nodes"] the reviews key stays with its other fields (totalCount, has_more) and only nodes is dropped from a copy of it. Every list matched by a glob path is removed; a non-list match is kept, since nothing is expanded from it. The stripped parent is built once per parent record, not once per item, and then deep-copied per item so that the parent and the sibling items stay isolated.

Breaking change: NON_BREAKING

parent_fields is a new optional field defaulting to None, and merge_parent a new boolean defaulting to false. The only behavioural change to existing code paths is the scalar-item wrapping described above, and it is unreachable without setting one of the parent-context options — _carries_parent_context() returns False otherwise, which is exactly the pre-existing condition. A manifest that sets neither parent_fields nor merge_parent produces byte-identical output.

Generated model

Hand-applied, then verified. Docker was unavailable, so rather than skip the check I ran the pinned generator directly — datamodel_code_generator==0.26.3 with the flag set from bin/generate_component_manifest_files.py — and confirmed the ParentFieldPath class and the parent_fields field are byte-identical to its output after ruff format. The merge_parent field was hand-applied the same way, following the remain_original_record pattern on the same class (Optional[bool] = Field(False, ...) with the description copied from the YAML). The rest of the generated file was discarded: the pinned generator emits conint(...)/confloat(...) annotations that mypy rejects, because the invocation does not pass --field-constraints.

Testing

19 new tests in unit_tests/sources/declarative/expanders/test_record_expander.py — copy onto every item, no original_record embedded, independence from remain_original_record, nested read and nested write, overwrite, missing parent field, present-null parent field, several entries applied in order, parent not mutated, scalar wrapping with and without parent context, items from the truncated-list retriever, glob rejection in each path, empty-path rejection, config interpolation of both paths.
2 factory tests, one reproducing the source-github reviews listing shape end to end through RecordSelector and asserting the three expected records.
12 new tests for merge_parent in the same file — the source-mailchimp shape end to end asserted on the full list, item wins on collision, multi-segment path keeps siblings and removes only the list, glob path removes every matched list (sections as a dict and as a list), parent_fields applied after the merge overwrites a merged value, combined with remain_original_record, scalar items wrapped as {"value": ...} and merged, parent not mutated, each item getting its own copy of a nested value (writing into one reaches neither the parent nor its siblings), items from the truncated-list retriever merged (both a mapping and a scalar), and unset merge_parent leaves items unchanged.
2 more factory tests: a manifest with merge_parent: true builds the expander with the flag set and reproduces the mailchimp shape through RecordSelector; one without it defaults to False.
Review round 2 adds 3 tests: each item gets its own copy of a container copied by parent_fields, the same for a value merged by merge_parent (both assert that writing into item 0 leaves item 1 and the parent untouched), and the glob-rejection message no longer mentions truncation handling. Plus a factory test building parent_fields from a manifest that omits type: ParentFieldPath. The test asserting that untouched nested values were shared is replaced by its opposite.
At 120c1efd, run per directory with the worktree on the import path: expanders 77 passed, parsers 212 passed, retrievers 52 passed, extractors 82 passed with test_response_to_file_extractor_memory_usage deselected (it dies with SIGILL on this machine on main as well, before any of this code runs). mypy clean on record_expander.py and model_to_component_factory.py; ruff format and ruff check clean on every touched file.

Not in scope

Chained expanders, so a grandchild can be stamped from both parent and grandparent: two fleet instances (source-zendesk-talk components.py:27, source-hubspot components.py:894), own PR when one of them migrates. merge_parent landed in this PR (second commit).

Review round 2

Addresses the review of 120c1efd (third commit):

Sev	Finding	Fix
P1	`merge_parent` aliased every nested container of the parent into every sibling item; a nested `AddFields` after the expander overwrote already-yielded records and the parent	`merge_base` is deep-copied per item in `_apply_parent_context`. Reviewer's repro now yields `item_id` 1, 2, 3 and leaves `parent["meta"]` at `{"list": "L"}`
P1	Same aliasing for a container-valued `parent_fields` copy	`ParentFieldPath.copy_onto` deep-copies a container value; scalars pass through. Reviewer's repro now yields `review_id` 1 and 2 with `parent["repository"]` unchanged
P2	The docstring example omits `type: ParentFieldPath` and did not validate, since `RecordExpander.parent_fields` was missing from `DEFAULT_MODEL_TYPES`	Registered it there, so the untyped form a Builder user writes resolves; new factory test
P3	Glob-rejection message said "when truncation handling is configured" for a `parent_fields` path	Clause dropped from the shared message; test pins it
P3	`dpath.new` behaviour on `record_path` (scalar intermediate raises, numeric segment creates a list) was undocumented	Stated in the `record_path` schema description, noting it matches `AddFields`. Round 3 turns the raise into a config error

The two P1 fixes make the "no deep copy" claim in the previous description wrong, and the design-decision bullets and docstrings are updated accordingly: the efficiency argument for parent_fields over remain_original_record is that it copies the named value rather than the whole parent, not that it copies nothing.

Review round 3

Addresses the review of 20436ea3 (fourth commit, 92daa4eb):

Sev	Finding	Fix
Medium	`parent_fields` looked each value up with `dpath.get`, which folds over the whole parent, expanded list included, for every item and every field. Expansion was quadratic in the list length: 17.9 s at 2,000 items with three fields, against 2.8 s for `remain_original_record`	`ParentFieldPath` walks only the containers on `parent_path` (mapping key compared as a string, list segment taken as an index, negative from the end, as `dpath.get` resolves them), and evaluates both paths once at construction. Now 0.013 s at 2,000 items and 0.024 s at 4,000. A parametrized test pins the lookup against `dpath.get` on 13 paths, and a test fails if `dpath.get` is called during expansion
Low	A `record_path` running through a value the item holds as a scalar or a list crashed with a raw `PathNotFound`/`TypeError`	Raised as an `AirbyteTracedException` with `failure_type=config_error` naming the `record_path`; tested for both
Low	`merge_parent` with a glob path also stripped non-list matches, which were never expanded	Only list matches are stripped, the same predicate the expansion uses; tested
Low	Three behaviours not pinned: the reversed strip order, `$parameters` propagation into `parent_fields` through the factory, and merge-before-copy order	Tests for several lists stripped from one list and for a match nested in another (both fail with forward order), a factory test interpolating `parameters` in both paths, and a nested `record_path` writing into a merged object
Low	The source-orb adopter entry described a behaviour change as a replacement	Moved to the not-covered list with the reason, above

One side effect of the new lookup: a parent_path segment that interpolates to an integer ("0" evaluates to 0) now finds a mapping key "0". dpath.get could not match an integer segment against a string key, so it copied None. No fleet manifest uses parent_fields yet.

CI at this head

92daa4eb (review round 3): CI pending. Locally: expanders, parsers/test_model_to_component_factory.py and extractors 420 passed; mypy clean on record_expander.py; ruff check and ruff format clean.

Earlier heads: at 120c1efd, Pytest (Fast) and Pytest (All) on Python 3.10–3.13, MyPy, Ruff Lint and Ruff Format all passed; Check: destination-motherduck failed and is unrelated, it is red on #1149 and #1162 at the same time. Locally at 606c32f3 (review round 2): unit_tests/sources/declarative full suite passed, mypy clean on record_expander.py and manifest_component_transformer.py, ruff check and ruff format clean.

🤖 Generated with Claude Code

… onto expanded items `RecordExpander` can already carry parent context onto each expanded item, but only by way of `remain_original_record: true`, which deep-copies the entire parent record once per item. When the parent is large and the nested list is long - a GitHub pull request node with ten reviews, say - that copies the whole pull request ten times to carry one URL. `parent_fields` names the values to copy instead: ```yaml record_expander: type: RecordExpander expand_records_from_field: ["reviews", "nodes"] parent_fields: - type: ParentFieldPath parent_path: ["url"] record_path: ["pull_request_url"] ``` Both paths are lists, so a value nested in the parent can be written to a nested position on the item; intermediate objects are created as needed. An existing value at `record_path` is overwritten, and a `parent_path` the parent does not have copies `None` - which is what `parent.get(field)` does in the connector classes this replaces. Glob metacharacters are rejected in both paths, matching the existing rule on `truncation_indicator_path`. The copy happens in `_apply_parent_context`, the single point both the embedded-items path and the `truncated_list_retriever` path go through, so fetched items are stamped the same way as embedded ones. `parent_fields` and `remain_original_record` are independent and may both be set. Making one suppress the other would be surprising; the reason to prefer `parent_fields` is that it does not need the deep copy, which the docstring and the schema description both say. Scalar list items previously fell through unwrapped unless `remain_original_record` was set, in which case they were wrapped as `{"value": ..., "original_record": ...}`. They are now wrapped as `{"value": ...}` plus the copied fields whenever either option asks for parent context, and still yielded bare when neither does. The generated model was hand-applied and verified against the pinned generator (`datamodel_code_generator==0.26.3` with the flag set from `bin/generate_component_manifest_files.py`): the `ParentFieldPath` class and the `parent_fields` field are byte-identical to its output after `ruff format`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-16T14:34:54Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@tolik0/cdk/record-expander-parent-fields#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch tolik0/cdk/record-expander-parent-fields

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

/autofix - Fixes most formatting and linting issues
/poetry-lock - Updates poetry.lock file
/test - Runs connector tests with the updated CDK
/prerelease - Triggers a prerelease publish with default arguments
/poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
/poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

CDK API Reference

📝 Edit this welcome message.

github-actions · 2026-09-16T14:45:37Z

PyTest Results (Fast)

4 828 tests +154 4 816 ✅ +154 8m 13s ⏱️ - 1m 39s
1 suites ± 0 12 💤 ± 0
1 files ± 0 0 ❌ ± 0

Results for commit eba17fb. ± Comparison against base commit d8d8e6f.

♻️ This comment has been updated with latest results.

github-actions · 2026-09-16T14:49:56Z

PyTest Results (Full)

4 831 tests +154 4 819 ✅ +154 10m 13s ⏱️ - 3m 41s
1 suites ± 0 12 💤 ± 0
1 files ± 0 0 ❌ ± 0

Results for commit eba17fb. ± Comparison against base commit d8d8e6f.

♻️ This comment has been updated with latest results.

…onto each expanded item With `merge_parent: true` each expanded item becomes the parent record shallow-merged with the item, the item's own keys winning on collision, and the value at `expand_records_from_field` removed from the parent's copy. Only that value is removed: on a multi-segment path the top-level key stays with its other fields, and the parent record itself is never mutated. This is the `{**record, **item}` flatten that source-mailchimp (`components.py:16`) and source-orb (`components.py:16-45`) implement in custom Python today. The order inside `_apply_parent_context` is fixed: the merge first, then `parent_fields` copies, then `original_record` when `remain_original_record` is set, so a named copy can overwrite a merged value. The three options are independent and may be combined. Scalar list items are wrapped as `{"value": item}` before the merge, and items fetched through `truncated_list_retriever` are merged the same way as embedded ones. The stripped parent is built once per parent record rather than once per item. The generated model change was hand-applied in the same style as `parent_fields`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

devin-ai-integration

Review of `120c1efd` (`tolik0/cdk/record-expander-parent-fields`)

Checked out the branch and ran everything below; no finding here is by inspection alone.

Breaking-change call: NON_BREAKING

Evidence from airbytehq/airbyte master (checkout dated 2026-09-16):

rg -l "record_expander:" airbyte-integrations/connectors --glob '*.yaml' → 1 of 511 manifests: source-stripe/manifest.yaml, 2 occurrences, both expand_records_from_field: [data, object, {lines|items}, data] with remain_original_record: true.
rg -l "RecordExpander|record_expander" --glob '*.py' → 1 file, source-stripe/unit_tests/integration/test_original_record_guard.py (tests only, no runtime use).
rg -l "merge_parent|parent_fields|ParentFieldPath" → 0 connectors.
Ran RecordExpander on a stripe-shaped record (dict items, a string item and an int item under data.object.lines.data) with remain_original_record=True and with no options, on main and on 120c1efd: diff → byte-identical (744 bytes each). So the scalar-wrapping refactor does not change the one production consumer, and the generated model's Config.extra = Extra.allow means older CDKs ignore the new keys as the schema description claims.

Findings

Sev	Where	What
P1	`record_expander.py:439`	`merge_parent` aliases every nested container of the parent into every sibling item. A downstream nested `AddFields` overwrote the value on all previously yielded records and on the parent (repro through `RecordSelector.select_records` inline). Contradicts the "parent is never mutated" docstring.
P1	`record_expander.py:101`	Same aliasing for `parent_fields` when `parent_path` points at a dict/list; `copy.deepcopy(value)` fixes it at cost proportional to the named value.
P2	`record_expander.py:142`	Docstring example omits `type: ParentFieldPath` and fails validation, because `RecordExpander.parent_fields` is not in `manifest_component_transformer.DEFAULT_MODEL_TYPES`.
P3	`record_expander.py:79`	Glob rejection error text says "when truncation handling is configured" for `parent_fields` paths.
P3	`record_expander.py:101`	A `record_path` whose intermediate already holds a scalar on the item (`child={"parent": 3}`, `record_path=["parent","id"]`) surfaces as `dpath.exceptions.PathNotFound: Path: ['parent', 'id'][0]`, not a CDK error. A numeric-string segment (`["items","0"]`) creates a list (`{"items": [5]}`). Both are `dpath.new` behaviour and match `AddFields`; informational, but worth a sentence in the schema description.

Verified as claimed (no action): on_no_records: emit_parent yields the parent untouched with merge_parent; glob and ** expand paths strip every matched list from the merge base and leave the parent intact (sections as a dict and as a list); a parent_path through a scalar ({"a": 5}, ["a","b"]) copies None rather than raising; scalar items are bare with no options and {"value": ..} + context otherwise.

What I ran

poetry run pytest unit_tests/sources/declarative/expanders unit_tests/sources/declarative/parsers/test_model_to_component_factory.py → 248 passed.
poetry run ruff check . / ruff format --check . → clean. mypy --config-file mypy.ini on record_expander.py and model_to_component_factory.py → clean.
Codegen drift: per the note that the ~350-line poe assemble delta is pre-existing on main, not re-reported here; the hand-applied ParentFieldPath / parent_fields / merge_parent model entries load and validate (factory tests above).
CI at review time: 21 passed, 1 failed (destination-motherduck, stated baseline), 3 pending. No connector in the matrix uses the new options, so green CI does not exercise them.

Verdict: request changes for the two P1s. The design is sound and the one production consumer is provably unaffected, but with merge_parent or a container-valued parent_fields the first nested AddFields a connector adds will silently emit wrong data.

Devin session

…d sibling items `merge_parent` shallow-merged the parent, so every nested container of the parent was aliased into every expanded item and into the parent itself. The same held for a `parent_fields` entry whose `parent_path` pointed at a dict or a list, because `dpath.get` returns the parent's object by reference. A downstream `AddFields`/`RemoveFields` with a multi-segment path then wrote into records already yielded, and into the record being expanded. `merge_base` is now deep-copied per item, and a container copied by `ParentFieldPath.copy_onto` is deep-copied as well; scalars pass through untouched. `merge_base` still excludes the expanded list and is still built once per parent record, so the copy stays cheaper than `remain_original_record`. Also: - register `RecordExpander.parent_fields` in `DEFAULT_MODEL_TYPES`, so a manifest may omit `type: ParentFieldPath` as it may for `AddFields.fields` - drop the "when truncation handling is configured" clause from the shared glob-rejection message, which `parent_fields` paths also reach - document `dpath.new` behaviour on `record_path` and the per-item copying in the schema descriptions and docstrings Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

devin-ai-integration

Re-review of `606c32f3` — all four findings from the `120c1efd` review are resolved

Everything below was executed on a checkout of 606c32f3f8aa0b80f30d82cea8017b8151219e95 (the only new commit since the previous review).

Findings status

Prev	Finding	Status	Evidence
P1	`merge_parent` aliased nested containers across items and the parent	Fixed	`{copy.deepcopy(merge_base), child_record}`. End-to-end repro (selector + nested `AddFields` on `meta.item_id`) now yields `item_ids: [1, 2, 3]` (was `[3, 3, 3]`); `out[0]['meta'] is out[1]['meta']` -> `False`; parent record unchanged after mutating an item.
P1	`parent_fields` aliased container-valued parent fields	Fixed	`_detached()` deep-copies Mapping/list/set/tuple values before `dpath.new`. Same repro on `repository.review_id` yields `[1, 2]` per item; parent and siblings untouched. Nested lists inside a copied list are also detached.
P2	Docstring example without `type: ParentFieldPath` failed validation	Fixed	`"RecordExpander.parent_fields": "ParentFieldPath"` added to `DEFAULT_MODEL_TYPES`. The previously failing YAML (no `type`) now builds and extracts records through the factory; covered by `test_create_record_expander_with_parent_fields_without_explicit_type`.
P3	Glob error text mentioned truncation handling	Fixed	Message is now: `Glob characters ('*', '?', '[') are not supported in` parent_path`: the path must identify a single field.`
P3 (info)	`dpath.new` scalar-intermediate / numeric-segment behavior	Documented	`record_path` description now states intermediate objects are created, scalar pass-through fails, numeric segment creates an array — matches `AddFields`.

Checks run on `606c32f3`

poetry run pytest unit_tests/sources/declarative/expanders unit_tests/sources/declarative/parsers/test_model_to_component_factory.py unit_tests/sources/declarative/parsers/test_manifest_component_transformer.py -q
  -> 269 passed
poetry run ruff check .            -> clean (PR files)
poetry run ruff format --check .   -> 792 files already formatted
poetry run mypy record_expander.py model_to_component_factory.py manifest_component_transformer.py
  -> Success: no issues found in 3 source files

New tests test_merge_parent_gives_each_item_its_own_copy_of_nested_values and test_parent_fields_gives_each_item_its_own_copy_of_a_copied_container assert identity (is not) and post-mutation isolation, so a regression to shallow sharing would fail.

Breaking-change call: NON_BREAKING (unchanged)

The fix commit only touches the new merge_parent / parent_fields code paths, the transformer default type for RecordExpander.parent_fields, and descriptions. The remain_original_record path and existing RecordExpander behavior are untouched; the airbytehq/airbyte count from the previous review still holds (1/511 manifests use record_expander, source-stripe, remain_original_record only; 0 use the new options).

No new findings. Approving.

Devin session

Anatolii Yatsuk (tolik0) · 2026-09-23T15:24:54Z

/prerelease

Prerelease Job Info

This job triggers the publish workflow with default arguments to create a prerelease.

Prerelease job started... Check job output.

✅ Prerelease workflow triggered successfully.

View the publish workflow run: https://github.com/airbytehq/airbyte-python-cdk/actions/runs/35881424906

Copilot

Copilot review overview

🟡 Changes recommended

New fields alter existing positional RecordExpander constructor argument binding, creating a critical API compatibility break.

Review effort: Lite
Findings: 1

Open (1)

Preserve positional RecordExpander constructor compatibility · New

What changed in this PR

Adds parent_fields and merge_parent to RecordExpander for propagating parent context to expanded records.

Changes:

Adds schema, model, factory, and transformer support.
Implements named field copying and parent merging with deep-copy isolation.
Adds comprehensive expansion and factory tests.
Requires preserving existing positional constructor argument compatibility.

File	Summary
`unit_tests/sources/declarative/parsers/test_model_to_component_factory.py`	Tests manifest construction.
`unit_tests/sources/declarative/expanders/test_record_expander.py`	Tests expansion and parent-context behavior.
`airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py`	Constructs new components.
`airbyte_cdk/sources/declarative/parsers/manifest_component_transformer.py`	Supports implicit `ParentFieldPath` types.
`airbyte_cdk/sources/declarative/models/declarative_component_schema.py`	Adds generated models.
`airbyte_cdk/sources/declarative/expanders/record_expander.py`	Implements parent-field copying and parent merging.
`airbyte_cdk/sources/declarative/expanders/__init__.py`	Exports `ParentFieldPath`.
`airbyte_cdk/sources/declarative/declarative_component_schema.yaml`	Defines the new manifest fields.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…e_parent stripping - Look up `parent_path` by walking only the containers on the path instead of calling `dpath.get`, which folds over the whole parent (the expanded list included) for every item and every field. The expansion was quadratic in the list length and slower than `remain_original_record`. Paths are evaluated once at construction. - Raise a config-error `AirbyteTracedException` naming the `record_path` when it runs through a value the item holds as a scalar or a list, instead of a raw `PathNotFound`/`TypeError`. - `merge_parent` strips only the list matches of a glob path; non-list matches were never expanded and are kept. - Pin the strip order, `$parameters` propagation into `parent_fields`, and the merge-then-copy order with tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…existing fields Inserting them after `remain_original_record` shifted every later positional argument of the `RecordExpander` constructor. No caller constructs it positionally, but appending them keeps the change strictly additive; a test pins the order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

devin-ai-integration Bot requested changes Sep 16, 2026

View reviewed changes

devin-ai-integration Bot approved these changes Sep 16, 2026

View reviewed changes

Anatolii Yatsuk (tolik0) self-assigned this Sep 17, 2026

Merge branch 'main' into tolik0/cdk/record-expander-parent-fields

20436ea

octavia-bot Bot deployed to PyPi September 23, 2026 15:25

View deployment

octavia-bot Bot deployed to DockerHub September 23, 2026 15:25

View deployment

Anatolii Yatsuk (tolik0) marked this pull request as ready for review September 28, 2026 14:57

Anatolii Yatsuk (tolik0) requested a review from a team as a code owner September 28, 2026 14:57

Copilot AI lite review requested due to automatic review settings September 28, 2026 14:57

Copilot started reviewing on behalf of Anatolii Yatsuk (tolik0) September 28, 2026 14:57

View session

Copilot AI reviewed Sep 28, 2026

View reviewed changes

Comment thread airbyte_cdk/sources/declarative/expanders/record_expander.py Outdated

Daryna Ishchenko (darynaishchenko) approved these changes Sep 28, 2026

View reviewed changes

Anatolii Yatsuk (tolik0) merged commit 7d8f4da into main Sep 28, 2026
26 of 27 checks passed

Anatolii Yatsuk (tolik0) deleted the tolik0/cdk/record-expander-parent-fields branch September 28, 2026 16:52

Navigation Menu

Repository navigation

feat(RecordExpander): add `parent_fields` to copy named parent values onto expanded items - #1165

feat(RecordExpander): add `parent_fields` to copy named parent values onto expanded items#1165
Anatolii Yatsuk (tolik0) merged 6 commits into
mainfrom
tolik0/cdk/record-expander-parent-fields

Anatolii Yatsuk (tolik0) commented Sep 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 16, 2026

Testing This CDK Version

PR Slash Commands

Helpful Resources

Uh oh!

github-actions Bot commented Sep 16, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 16, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

Anatolii Yatsuk (tolik0) commented Sep 23, 2026 •

edited by github-actions Bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Anatolii Yatsuk (tolik0) commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Adopters

Design decisions

Breaking change: NON_BREAKING

Generated model

Testing

Not in scope

Review round 2

Review round 3

CI at this head

Uh oh!

github-actions Bot commented Sep 16, 2026

👋 Greetings, Airbyte Team Member!

Testing This CDK Version

PR Slash Commands

Helpful Resources

Uh oh!

github-actions Bot commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PyTest Results (Fast)

Uh oh!

github-actions Bot commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PyTest Results (Full)

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Review of 120c1efd (tolik0/cdk/record-expander-parent-fields)

Breaking-change call: NON_BREAKING

Findings

What I ran

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Re-review of 606c32f3 — all four findings from the 120c1efd review are resolved

Findings status

Checks run on 606c32f3

Breaking-change call: NON_BREAKING (unchanged)

Uh oh!

Anatolii Yatsuk (tolik0) commented Sep 23, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Anatolii Yatsuk (tolik0) commented Sep 16, 2026 •

edited

Loading

github-actions Bot commented Sep 16, 2026 •

edited

Loading

github-actions Bot commented Sep 16, 2026 •

edited

Loading

Review of `120c1efd` (`tolik0/cdk/record-expander-parent-fields`)

Re-review of `606c32f3` — all four findings from the `120c1efd` review are resolved

Checks run on `606c32f3`

Anatolii Yatsuk (tolik0) commented Sep 23, 2026 •

edited by github-actions Bot

Loading