Repository navigation
[#39723] Implement model for Iceberg side input cache - #39724
jrmccluskey merged 2 commits into
Conversation
|
Assigning reviewers: R: @Abacn for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| public abstract String getPartitionSpecJson(); | ||
|
|
||
| @SchemaFieldNumber("6") | ||
| public abstract String getSortOrderJson(); |
There was a problem hiding this comment.
I think we should keep track of the full map of schemas / partition specs / sort orders. There are instances where we're not always working with the most recent table metadata, so we'll need to have access to historical schema/spec/sort orders.
Might make sense to add 3 int fields to indicate the "current" schemaId, specId, orderId, and use them to return the respective metadata property.
ahmedabu98
left a comment
ahmedabu98
left a comment
There was a problem hiding this comment.
Looks good, just left one nit
| public Map<Integer, Schema> getSchemas() { | ||
| Map<Integer, Schema> local = cachedSchemas; | ||
| if (local == null) { | ||
| synchronized (this) { | ||
| local = cachedSchema; | ||
| local = cachedSchemas; | ||
| if (local == null) { | ||
| cachedSchema = local = SchemaParser.fromJson(getSchemaJson()); | ||
| ImmutableMap.Builder<Integer, Schema> builder = ImmutableMap.builder(); | ||
| for (Map.Entry<Integer, String> entry : getSchemasJson().entrySet()) { | ||
| builder.put(entry.getKey(), SchemaParser.fromJson(entry.getValue())); | ||
| } | ||
| cachedSchemas = local = builder.build(); | ||
| } | ||
| } | ||
| } | ||
| return local; | ||
| } |
There was a problem hiding this comment.
Can we do this in a more lazy way? If the table historically has 10 schemas and I want to get one, I should only have to serialize that one schema, not all 10.
There was a problem hiding this comment.
Same comment for the partition spec and sort order maps
There was a problem hiding this comment.
How heavy-weight are these schemas/specs/sort orders? If they're not particularly intensive to serialize and deserialize I don't know if there's much benefit to adding an extra level of tuning here, the downstream usages would be able to make determinations as to what they need without having the extra overhead of a separately configurable thing here.
Probably something we could explore after the e2e implementation is done, benchmarking to see if tuning the number of serialized objects matters or not.
There was a problem hiding this comment.
Sounds good, we can leave it as a later improvement
Implements the SerializableTableSpec and SideInputTable classes to provide a serializable representation of Iceberg table metadata + corresponding tests.
Part of #39723
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.