[claudevdm]

AddFiles: dry_run reports what the schema pre-pass would do

SchemaEvolutionConfig.setDryRun(true) (dry_run: true in YAML) turns AddFiles into a report-only transform. The read side runs as usual (footers, distinct schemas); instead of committing and registering, a new dry_run_report output gets one row per distinct file schema plus a create row, marker rows for unreadable and ORC/Avro files, and a summary row, told apart by a row_type column. snapshots and errors are empty. The output only exists when dry_run is set, so existing YAML pipelines are unaffected.

The diff has two halves. Reading them in this order is easiest.

1. CommitSchemaUnion: the commit now plans before it writes.

Everything a commit decides is pulled into a Plan, built by plan(): which distinct file schemas get merged (SchemaToMerge), which are refused and why (IncompatibleSchema), the schema the table ends with, and any configuration problems. It is an EvolutionPlan against an existing table (classify and fold on scratch transactions, all against one snapshot of the table schema, plus whether the name mapping gets repaired) or a CreationPlan when the table is missing (fold for creation, then the partition spec and sort order resolved against the union, or the problem that blocks creation). commitOnce only dispatches to evolve or create; each fails or warns under the handling mode first, then writes. DryRunReport turns the same plan into rows. A check added to the plan reaches both, so the report cannot drift from the commit. In particular a schema that is fine against the table but conflicts with another schema of the input is reported with the blame a real run assigns.

Because these types are now shared with the report, some merged code was renamed and tidied:

• Accepted is now SchemaToMerge and Incompatible is now IncompatibleSchema (each is a whole distinct file schema, not a column change); TableCreation is now NewTableSettings; reportIncompatible is now failOrWarnOnIncompatibleSchemas, since it can end the run.
• Settings (the config, the handling resolved for the pipeline mode, the new-table settings) replaces the three loose arguments that commit, CommitSchemaOnce and now DryRunReport carried.
• Classify and fold no longer append to lists passed in: a private Verdicts holder tracks where each schema stands, and staging returns the Conflict it found.

2. The dry run itself.

DryRunReport (new), the expand wiring in AddFiles, the dry_run knob in SchemaEvolutionConfig and the YAML provider, the marker keys in ReadFooterSchema, the transform-level tests in AddFilesTest, and a YAML fixture that runs a dry run against a missing table, writes the report to JSON and asserts the full rows.

Report rows

column	meaning
row_type	schema, create, unreadable, unchecked or summary
schema_key	short murmur3 key of the schema JSON on schema rows, to group by; empty on the others
schema	canonical schema JSON; on the create row, the union the table would be created with; empty on the others
num_files	files the row covers
changes	the SchemaDelta descriptions on schema rows; one create optional <name> <type> per column on the create row; the totals line plus any table-level change (name mapping) on the summary
allowed	whether a real run would accept it; on the summary, no schema is incompatible and the configuration raises no problem
reason	why not, else empty; the consequence on the summary
would_create_table	false whenever a real run would not create the table, including when it would fail first

Part of #40144

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[claudevdm]

R: @ahmedabu98

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.46%. Comparing base (d375603) to head (2d2d5de).
⚠️ Report is 7 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #40167      +/-   ##
============================================
- Coverage     58.47%   58.46%   -0.01%     
+ Complexity    13593    13591       -2     
============================================
  Files          2576     2576              
  Lines        269266   269266              
  Branches      11079    11079              
============================================
- Hits         157441   157427      -14     
- Misses       105863   105871       +8     
- Partials       5962     5968       +6     

Flag	Coverage Δ
java	64.68% <ø> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ahmedabu98]

Left some comments.
Really cool to see how it looks like in the YAML test

[ahmedabu98]

Just some doc nits. Everything else LGTM, thanks!