Skip to content

Delta Sharing protocol: Query Table predicate and limit pushdown - #113

Draft
littlegrasscao wants to merge 1 commit into
server/split2/20-protocol-query-refreshfrom
server/split2/21-protocol-query-pushdown
Draft

littlegrasscao wants to merge 1 commit into
server/split2/20-protocol-query-refreshfrom
server/split2/21-protocol-query-pushdown

Conversation

@littlegrasscao

@littlegrasscao littlegrasscao commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • convert jsonPredicateHints to a Kernel predicate for ScanBuilder.withFilter, so Kernel prunes partitions and skips files by min/max stats
  • hints are best effort, as in the Delta Sharing server: oversized, too deep, malformed, or unsupported hints return every file instead of an error
  • the deprecated SQL predicateHints are logged at debug and never filter
  • apply limitHint only when no predicate hint is present, stopping once the returned files cover the limit (stats.numRecords minus deletion vector cardinality)
  • read scan files with stats, so file actions include stats
  • clients still filter rows and apply the limit, so returning extra files is always correct

Testing

  • mvn clean install: 220 tests passed
  • JsonPredicateHintsTest: every op and value type, and/or/not/isNull, bool leaves, the depth limit, and hints that are skipped
  • DeltaTableQueryReaderTest: partition pruning, stats-based skipping, an unsupported hint, and the limitHint cutoff on a local Delta log, with stub signing instead of Mockito
  • ProtocolTablePushdownIntegrationTest: limits and filters on the S3, Azure, and GCS test tables; each case is skipped without credentials

Stacked on #110.

@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 16700d1 to 6bac9b1 Compare October 7, 2026 00:44
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from dde54a7 to 8d9acdb Compare October 7, 2026 01:12
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 1172cf8 to f79c4a4 Compare October 7, 2026 01:13
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from 8d9acdb to f9fd14c Compare October 7, 2026 01:56
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from f79c4a4 to 0b6b3ea Compare October 7, 2026 01:56
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from f9fd14c to 27f5e9b Compare October 7, 2026 17:27
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 0b6b3ea to fd4dfa1 Compare October 7, 2026 17:28
@littlegrasscao littlegrasscao changed the title Query Table predicate and limit pushdown protocol: Query Table predicate and limit pushdown Oct 9, 2026
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from 27f5e9b to 87d4ec5 Compare October 14, 2026 17:26
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from fd4dfa1 to 81ab161 Compare October 14, 2026 17:31
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from 87d4ec5 to 4df052f Compare October 14, 2026 19:25
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 81ab161 to 9ccea3e Compare October 14, 2026 19:25
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from 4df052f to 1f0f218 Compare October 15, 2026 16:47
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 9ccea3e to 9e15672 Compare October 15, 2026 16:47
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from 1f0f218 to b35ac17 Compare October 15, 2026 18:14
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 9e15672 to 77177a1 Compare October 15, 2026 18:14
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from b35ac17 to 0b6b7d4 Compare October 15, 2026 18:29
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 77177a1 to 5ec2957 Compare October 15, 2026 18:30
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from 0b6b7d4 to 2471df2 Compare October 15, 2026 23:35
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 5ec2957 to af34658 Compare October 15, 2026 23:35
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from 2471df2 to a6e5aa5 Compare October 15, 2026 23:38
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from af34658 to 7d57a1e Compare October 15, 2026 23:38
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from a6e5aa5 to f592d4b Compare October 15, 2026 23:54
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 7d57a1e to 2e26c60 Compare October 15, 2026 23:54
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from f592d4b to 40ea5dd Compare October 16, 2026 00:18
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 2e26c60 to 819be21 Compare October 16, 2026 00:18
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from 40ea5dd to 59f23fe Compare October 16, 2026 00:41
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 819be21 to 162317e Compare October 16, 2026 00:41
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from 59f23fe to 3185340 Compare October 19, 2026 17:13
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 162317e to d82395a Compare October 19, 2026 17:13
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from 3185340 to 1719cf1 Compare October 19, 2026 21:44
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from d82395a to f7be628 Compare October 19, 2026 21:44
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from 1719cf1 to 535b1a8 Compare October 19, 2026 22:35
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from f7be628 to 76ed426 Compare October 19, 2026 22:35
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from 535b1a8 to 8f1537f Compare October 19, 2026 22:37
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 76ed426 to d30057f Compare October 19, 2026 22:37
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from 8f1537f to afb8502 Compare October 20, 2026 00:31
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from d30057f to 515fd94 Compare October 20, 2026 00:31
@littlegrasscao
littlegrasscao force-pushed the server/split2/20-protocol-query-refresh branch from afb8502 to a874165 Compare October 21, 2026 19:21
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 515fd94 to 88e900b Compare October 21, 2026 19:25
@littlegrasscao
littlegrasscao added this pull request to stack #116 October 21, 2026 20:09
@littlegrasscao
littlegrasscao removed this pull request from stack #116 October 21, 2026 20:27
@littlegrasscao littlegrasscao changed the title protocol: Query Table predicate and limit pushdown Delta Sharing protocol: Query Table predicate and limit pushdown Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant