[claudevdm]

…nvertToDataFile (#40143)

• AddFiles: per-file coverage check and pinned-column enforcement in ConvertToDataFile

ConvertToDataFile gains a SchemaEvolutionConfig.
With options set, every Parquet file is checked against the table before its DataFile is built, and every failure is one row on the error output:

1. FileSchemas.effective(footer) (convert plus tighten, shared with the read side so both sides see the same schema) is computed. A footer whose schema cannot be converted (unannotated repeated leaf, legacy MAP_KEY_VALUE, uint64, ...) yields "Could not read the file's schema: ".
2. Coverage: SchemaDelta.classify against the cached table. The pre-pass commits the schema before paths reach this DoFn, so the delta is expected to be empty. If it is not, the table is refreshed once and classified again. A remaining delta is reported as "Table schema does not cover the file after refresh: ", where the reason is the delta's disallowed reason when the options forbid the change (the ROUTE_TO_ERRORS path: this is where an incompatible file lands) or "changes not applied: ..." when the change was allowed but somehow not committed (defensive catch all).
3. Pins: for each pinned column the table has, the file must contain it and be provably null-free: the footer's own null count for the column (FileSchemas.nullCount, summed over row groups) must be present and zero. A missing count is a violation too: "cannot prove" is not "proven". Messages: "Pinned required column X is absent from the file" / "has no null count statistics in the file" / "has N null(s) in the file".

The pin evidence deliberately does NOT come from the Metrics object built for the DataFile: the table's write.metadata.metrics configuration shapes those (mode none, or the inferred-column cap on wide schemas, drops null counts entirely), and pin enforcement must not be configurable away. Two tests pin this down under write.metadata.metrics.default=none: a null-free pinned file still registers and a file with nulls is still caught.

With evolution enabled, a non-Parquet file (ORC, Avro) is routed to the error output instead of registering unchecked: none of the checks above can read those formats, and silently registering a file the options promised to verify would be a hole in the guarantee. With evolution disabled they register exactly as before.

SchemaEvolutionConfig.UnverifiableFileHandling relaxes exactly the "cannot verify" cases, never a failed check: REJECT (default) routes them to the error output as above; ACCEPT registers a non-Parquet file unchecked, and a Parquet file whose footer has no null count for a pinned column on trust (a writer with statistics disabled, or a pin under a list or map, whose physical chunk path is never mapped). An absent pinned column and a counted null stay violations under ACCEPT, and the pin walk continues past an unproven pin so a counted one still fails the file.

• comments

• spotbugs

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[claudevdm]

R: @ahmedabu98
R: @chamikaramj

[claudevdm]

Cherry pick of #40143

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[claudevdm]

@Amar3tto can you PTAL?