Skip to content

Make parameterize_extract_columns respect reassign_columns=False - #1732

Merged
skrawcz merged 1 commit into
apache:mainfrom
breken-ai:fix/parameterize-extract-columns-reassign-false
Sep 26, 2026
Merged

skrawcz merged 1 commit into
apache:mainfrom
breken-ai:fix/parameterize-extract-columns-reassign-false

Conversation

@breken-ai

Copy link
Copy Markdown
Contributor

@parameterize_extract_columns(..., reassign_columns=False) ignores the flag. expand_node builds the node it passes to @parameterize with callabl=wrapper_fn every time, and parameterize.expand_node calls node_.callable. So the fn_to_call choice never takes effect: the output columns are still renamed by position. Each extracted column can then silently carry another column's data.

@parameterize_extract_columns(
    ParameterizedExtract(("y", "x"), {"k": value(1)}),
    reassign_columns=False,
)
def df(k: int) -> pd.DataFrame:
    return pd.DataFrame({"x": [k], "y": [k * 100]})

dr.execute(["x", "y"])
# on main: x == 100, y == 1  (columns renamed positionally despite reassign_columns=False)
# expected: x == 1, y == 100

If the function returns a different number of columns than outputs, the rename also raises a length-mismatch error, which is the other case reassign_columns=False is meant to allow.

Changes

  • hamilton/function_modifiers/expanders.py: build the node from fn_to_call (the renaming wrapper only when reassign_columns=True), so the flag takes effect. The default True path is unchanged.
  • tests/function_modifiers/test_expanders.py: test_parameterized_extract_columns_no_reassign_columns.

How I tested this

  • New test on unmodified main (661aa888): fails with DataFrame.columns are different ... [left]: ['y', 'x'] [right]: ['x', 'y'].
  • With the fix, it passes. The existing test_parameterized_extract_columns (the default path) still passes.
  • pytest tests/function_modifiers tests/test_function_modifiers.py tests/test_end_to_end.py tests/test_hamilton_driver.py tests/test_async_driver.py: 528 passed.
  • ruff check and ruff format --check (0.15.14) are clean. git diff --check is clean.

Notes

The flag has had no effect since parameterize_extract_columns was added in 72b0a09 (2022-11).

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output) — no new functions
  • Placeholder code is flagged / future TODOs are captured in comments — none added
  • Project documentation has been updated if adding/changing functionality — no documented behavior changes; this makes the code match the existing docs

AI disclosure (per the ASF Generative Tooling guidance): this change was written with an AI coding tool (Claude Code, Claude Opus 5.5) working through the breken-ai account. The tool found the bug, wrote the fix and the test, and wrote this description. The commit carries a Generated-by: trailer. The diff is a small original change to existing Hamilton code and includes no third-party material. The red/green runs above are real and can be re-run from the diff. If you would rather not take AI-assisted contributions here, say so and I will close this.

The node handed to @parameterize always wrapped the function in the
column-renaming wrapper, so reassign_columns=False had no effect: the
output columns were still renamed by position, and an extracted column
could silently carry another column's data.

Build the node from the function chosen by reassign_columns.

Generated-by: Claude Code (Claude Opus 5.5)
@skrawcz
skrawcz force-pushed the fix/parameterize-extract-columns-reassign-false branch from ce2bbb1 to d0442a0 Compare September 26, 2026 18:52
@skrawcz
skrawcz merged commit ed4a58e into apache:main Sep 26, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants