Skip to content

protocol: Query Table startingVersion, endingVersion, and historical protocol - #114

Draft
littlegrasscao wants to merge 1 commit into
server/split2/21-protocol-query-pushdownfrom
server/split2/22-protocol-query-versions
Draft

littlegrasscao wants to merge 1 commit into
server/split2/21-protocol-query-pushdownfrom
server/split2/22-protocol-query-versions

Conversation

@littlegrasscao

@littlegrasscao littlegrasscao commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • startingVersion (with optional endingVersion) on Query Table now returns data change files instead of 501, following the OSS server's queryDataChangeSinceStartVersion
    • reads commits with Kernel getChanges and returns add and remove files that have dataChange=true, each with its version and timestamp; compaction rewrites are skipped
    • protocol and metaData lines come from startingVersion, and Delta-Table-Version is startingVersion
    • later metaData changes are returned with their version
    • parquet responses use add/remove lines; delta responses wrap them in file.deltaSingleAction, with on-disk deletion vectors signed as in snapshot queries
  • includeHistoricalProtocol=true adds later protocol changes, with their version, to delta responses. Parquet responses ignore it, matching the client, which sends it only when it accepts delta
  • every protocol in the range is checked against delta-sharing-capabilities reader features, so a parquet query across a deletion vector upgrade returns 400
  • endingVersion without startingVersion, a negative startingVersion, endingVersion < startingVersion, versions after the latest, and combining startingVersion with version, timestamp, includeRefreshToken, or refreshToken return 400 INVALID_PARAMETER_VALUE
  • predicate and limit hints apply to snapshot files only, so they are ignored for startingVersion queries, as in the OSS server
  • includeendstreamaction=true appends an endStreamAction without a refresh token

Testing

  • mvn clean install: 227 tests passed
  • DeltaTableQueryReaderTest: a local Delta log with delete-and-insert, metaData change, deletion vector protocol upgrade, and compaction commits shows add/remove files with versions, historical metaData, endingVersion bounds, skipped compaction files, historical protocol only when requested, parquet rejection after the upgrade, and versions after the latest rejected
  • ProtocolTableStartingVersionTest: a startingVersion query on a local Delta log returns Delta-Table-Version set to startingVersion and an endStreamAction; every invalid version combination returns 400
  • ProtocolTableStartingVersionIntegrationTest: change files on the S3, Azure, and GCS test tables in both formats; each case is skipped without credentials

Stacked on #113.

@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 1172cf8 to f79c4a4 Compare October 7, 2026 01:13
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 4907551 to 88c5f4d Compare October 7, 2026 01:21
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from f79c4a4 to 0b6b3ea Compare October 7, 2026 01:56
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch 2 times, most recently from de7eacb to 5dd60cf Compare October 7, 2026 02:05
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 0b6b3ea to fd4dfa1 Compare October 7, 2026 17:28
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 5dd60cf to 0134c26 Compare October 7, 2026 17:28
@littlegrasscao littlegrasscao changed the title Query Table startingVersion, endingVersion, and historical protocol protocol: Query Table startingVersion, endingVersion, and historical protocol Oct 9, 2026
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from fd4dfa1 to 81ab161 Compare October 14, 2026 17:31
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch 2 times, most recently from 70c4977 to fbdbf5f Compare October 14, 2026 17:38
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 81ab161 to 9ccea3e Compare October 14, 2026 19:25
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from fbdbf5f to 51f652b Compare October 14, 2026 19:25
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 9ccea3e to 9e15672 Compare October 15, 2026 16:47
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 51f652b to 40ceb7e Compare October 15, 2026 16:47
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 9e15672 to 77177a1 Compare October 15, 2026 18:14
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 40ceb7e to 095395e Compare October 15, 2026 18:14
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 77177a1 to 5ec2957 Compare October 15, 2026 18:30
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 095395e to 53e5e9c Compare October 15, 2026 18:30
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 5ec2957 to af34658 Compare October 15, 2026 23:35
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 53e5e9c to cf2759e Compare October 15, 2026 23:35
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from af34658 to 7d57a1e Compare October 15, 2026 23:38
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from cf2759e to 052fed4 Compare October 15, 2026 23:38
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 7d57a1e to 2e26c60 Compare October 15, 2026 23:54
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 052fed4 to 29222a5 Compare October 15, 2026 23:54
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 2e26c60 to 819be21 Compare October 16, 2026 00:18
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 29222a5 to 0d8789f Compare October 16, 2026 00:18
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 819be21 to 162317e Compare October 16, 2026 00:41
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 0d8789f to e1d55ac Compare October 16, 2026 00:41
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 162317e to d82395a Compare October 19, 2026 17:13
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from e1d55ac to f10c878 Compare October 19, 2026 17:13
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from d82395a to f7be628 Compare October 19, 2026 21:44
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from f10c878 to db0a270 Compare October 19, 2026 21:44
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from f7be628 to 76ed426 Compare October 19, 2026 22:35
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from db0a270 to 3bcd860 Compare October 19, 2026 22:35
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from 76ed426 to d30057f Compare October 19, 2026 22:37
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 3bcd860 to f2152d8 Compare October 19, 2026 22:37
@littlegrasscao
littlegrasscao force-pushed the server/split2/21-protocol-query-pushdown branch from d30057f to 515fd94 Compare October 20, 2026 00:31
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from f2152d8 to fbbbb98 Compare October 20, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant