Repository navigation
[Cherry-pick][#39723] Iceberg Side Input Cache Public API and Wrap-Up - #40248
Merged
Merged
Conversation
* Expose Public API for Iceberg Side Input Cache * Update sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java Co-authored-by: Ahmed Abualsaud <65791736+ahmedabu98@users.noreply.github.com> * Streamline API, remove redundant checks * Check polled table metric * Evolve Iceberg schema mid-execution in streaming side input cache test * Throw if sub-options are enabled without side input cache enabled * test with partition spec evolution --------- Co-authored-by: Ahmed Abualsaud <65791736+ahmedabu98@users.noreply.github.com>
* Address wrap-up items for Iceberg side input table cache - Bind historical sort orders unchecked via SortOrderParser.fromJson(schema, node, getOrderId()) in SerializableTableSpec. - Expose Hadoop Configuration via IcebergCatalogConfig.getHadoopConfiguration() and propagate to SerializableTableSpec.getFileIO(conf) through SideInputTable in write transforms. - Add worker-local transient Guava cache to ExtractTableIdsDoFn to eliminate hot-key shuffle bottlenecks on streaming writes. - Add comprehensive unit and integration tests. * Propagate Clock to ExtractTableIdsDoFn for deterministic cache expiration in tests - Ensure TableMetadataDriver passes its Clock to ExtractTableIdsDoFn so virtual-time advancement in tests (e.g. testUnusedTablesEvictedFromStreamingCache) correctly expires worker-local table ID caches instead of using wall-clock time. - Fixes testUnusedTablesEvictedFromStreamingCache flakiness on CI environments with limited concurrency. * Make streaming refresh and spec evolution tests robust against worker cache TTL - Advance ControllableTestClock on trigger rows in testMetadataRefreshedAcrossIntervals, testMetadataRefreshedAcrossIntervalsAsSideInput, and testMetadataRefreshedAcrossIntervalsAsSideInputWithMultipleTables so worker-local table ID caches expire as intended across virtual intervals. - In testStreamingSpecEvolutionWithoutPipelineRestart, wait for the worker-local cache TTL (500ms) to elapse after spec evolution before emitting the second element. * Declare jackson-databind dependency in Iceberg IO module - Add library.java.jackson_databind to dependencies in sdks/java/io/iceberg/build.gradle to resolve usedUndeclaredArtifacts dependency analysis warning caused by JsonNode usage in SerializableTableSpec. * Avoid potential flakes in EvolveSpec DoFn, address nits * Simplify worker-local cache in TableMetadataDriver using LinkedHashMap LRU - Replace Guava Cache, CacheBuilder, and Ticker in ExtractTableIdsDoFn with a standard LinkedHashMap-based LRU cache capped at 10,000 entries. - Track lastEmitted timestamp per table identifier and emit when (now - lastEmitted) >= refreshInterval / 2. - Remove getClock() and setClock() from TableMetadataDriver and its Builder to keep the AutoValue API clean and free of testing methods. - Provide testing hooks via static globalTestClock and instance setClock on ExtractTableIdsDoFn. * simplify dofn + fix unit test
Contributor
Author
Contributor
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
Amar3tto
approved these changes
Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #40140 and #40222 to complete feature for 2.77.0 release.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.