[khaledh]

Problem

A Dataflow streaming pipeline writing deeply nested rows to BigQuery through the Storage Write API with withAutoSchemaUpdate(true) could not keep up with its input. ~20% of conversion CPU went to TableRow map operations on the unknown-fields collector, not to converting values.

It happens when rows are wide and nested (~50 STRUCT fields each), the schema is current, and auto schema update is on. The converter then creates a nested unknown-fields TableRow for every struct, checks whether it stayed empty, and removes it: ~150 TableRow calls per row that usually find nothing. Each call is expensive because TableRow extends GenericData: get/set/remove/computeIfAbsent intern the key and do a reflective declared-field lookup before falling through to the backing HashMap, and isEmpty() allocates an entry-set view just to test for zero.

Fix

Use the backing map directly via GenericData.getUnknownKeys(). For any key that isn't a declared field, getUnknownKeys().put(k, v) stores exactly what set(k, v) would. Three helpers (unknownFieldMap, setUnknownField, isUnknownMapEmpty) make that substitution for ordinary keys and keep the original TableRow path for declared fields. Same collectors, same allocation, same output, same public API.

Safety

• For non-declared keys, GenericData.set() ends in put on the same map getUnknownKeys() returns; only the preceding lookup is skipped. TableRow is final and doesn't enable ignoreCase, so there's no key normalization to preserve.
• f, the one declared field, keeps its original path: a column named f still goes through setF, and a non-null f (even an empty list) still counts as an entry in the emptiness check, as AbstractMap.isEmpty() did.
• Declared fields are read from TableRow.getClassInfo() at class load. If a future client version adds fields, they take the original path and the emptiness shortcut disables itself.

Other paths

No path gets slower: the helpers add one contains on a one-element list in place of an intern and a reflective lookup. Rows that do carry unknown columns save the same cost. Users without auto schema update or unknown columns never ran this code and see no change. Legacy cell-format rows still convert through getF().

Testing

New TableRowToStorageApiProtoUnknownFieldsTest (17 tests): unknown columns as bytes and JSON, nested and repeated collectors, empty-struct pruning, a column named f, f as an empty list, legacy cell format. Existing tests pass unchanged. A backport in the pipeline above raised committed throughput ~27% and cut CPU per record ~20% in a matched staging comparison.

Closes #40492. Independent of #40203 and #40204.

[github-actions]

Assigning reviewers:

R: @chamikaramj for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[github-actions]

Reminder, please take a look at this pr: @chamikaramj

[github-actions]

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @kennknowles for label java.
Backup reviewer: ahmedabu98 (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

[kennknowles]

I honestly don't understand the PR description. Can you try to revise it to make it simpler and explain what is done and why?

[khaledh]

I honestly don't understand the PR description. Can you try to revise it to make it simpler and explain what is done and why?

I updated the PR description. Basically the PR improves throughput by short-circuiting collection of unknown fields by interacting with the underlying hashmap directly, instead of using reflection in a per-row hot path.