Skip to content

protocol: Query Table Changes (Change Data Feed) - #115

Draft
littlegrasscao wants to merge 1 commit into
server/split2/22-protocol-query-versionsfrom
server/split2/23-protocol-query-changes
Draft

littlegrasscao wants to merge 1 commit into
server/split2/22-protocol-query-versionsfrom
server/split2/23-protocol-query-changes

Conversation

@littlegrasscao

@littlegrasscao littlegrasscao commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • add GET {prefix}/shares/{share}/schemas/{schema}/tables/{table}/changes (Read Change Data Feed) to OpenSharingProtocolService, following the OSS server's queryCDF and DeltaSharingCDCReader
    • the start comes from startingVersion or startingTimestamp (first version at or after it); the end from endingVersion or endingTimestamp (last version at or before it), or the latest version. Versions take precedence over timestamps, and an end after the latest version is capped
    • a commit with cdc files returns only those; otherwise it returns add and remove files with dataChange=true. MERGE commits whose metrics report no inserted, updated, or deleted rows return no files
    • the Change Data Feed must be enabled at the start, and metadata that disables it within the range returns 400
    • protocol and metaData come from the start version with includeHistoricalMetadata=true, which also returns later metaData with their version; otherwise they come from the latest version. Delta-Table-Version is the start version
    • includeHistoricalProtocol=true adds later protocols to delta responses, as for startingVersion queries in protocol: Query Table startingVersion, endingVersion, and historical protocol #114
    • parquet responses use add, cdf, and remove lines; delta responses wrap them in file.deltaSingleAction as add, cdc, and remove
    • fileidhash, includeendstreamaction, url access checks, and reader feature checks behave as in Query Table
  • request validation lives in ProtocolTables.changes, with the parameters in TableChangesRequest
  • startingVersion Query Table and the Change Data Feed now share one per-commit reader

Testing

  • mvn clean install: 235 tests passed
  • DeltaTableQueryReaderTest: a local CDF-enabled Delta log (insert, update with cdc files, delete without cdc, no-op MERGE, CDF disabled) checks that cdc files win, no-op MERGEs are skipped, parquet and delta lines are correct, timestamps resolve to versions, header metaData follows includeHistoricalMetadata, and a missing start, an end before the start, a start after the latest, and ranges without the Change Data Feed are rejected; the existing startingVersion tests still pass
  • ProtocolTableChangesTest: on a local Delta log without a Change Data Feed, /changes returns 400 INVALID_PARAMETER_VALUE; a missing start, a malformed timestamp, or a malformed version returns 400, and an ungranted share returns 404
  • ProtocolTableChangesIntegrationTest: the S3 CDF test table returns change files in both formats, rejects a range through the version that disables the Change Data Feed, caps an end after the latest version, and rejects an end before the start; table1 on S3, Azure, and GCS rejects the Change Data Feed. Each case is skipped without credentials

Stacked on #114.

@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 4907551 to 88c5f4d Compare October 7, 2026 01:21
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from a787a38 to 821b13a Compare October 7, 2026 01:21
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 88c5f4d to de7eacb Compare October 7, 2026 01:56
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from 821b13a to 73fb9d3 Compare October 7, 2026 01:56
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from de7eacb to 5dd60cf Compare October 7, 2026 02:05
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from 73fb9d3 to 0cf65cd Compare October 7, 2026 02:05
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 5dd60cf to 0134c26 Compare October 7, 2026 17:28
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch 3 times, most recently from 9fa2cbe to 062bf43 Compare October 8, 2026 18:26
@littlegrasscao littlegrasscao changed the title Query Table Changes (Change Data Feed) protocol: Query Table Changes (Change Data Feed) Oct 9, 2026
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch 2 times, most recently from 70c4977 to fbdbf5f Compare October 14, 2026 17:38
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from a4eae87 to f2d88f1 Compare October 14, 2026 17:42
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from fbdbf5f to 51f652b Compare October 14, 2026 19:25
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from f2d88f1 to bbda0ee Compare October 14, 2026 19:25
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 51f652b to 40ceb7e Compare October 15, 2026 16:47
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from bbda0ee to 125450b Compare October 15, 2026 16:47
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 40ceb7e to 095395e Compare October 15, 2026 18:14
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from 125450b to da6e0e2 Compare October 15, 2026 18:14
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 095395e to 53e5e9c Compare October 15, 2026 18:30
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from da6e0e2 to b41b977 Compare October 15, 2026 18:30
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 53e5e9c to cf2759e Compare October 15, 2026 23:35
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from b41b977 to 71c7ebc Compare October 15, 2026 23:35
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from cf2759e to 052fed4 Compare October 15, 2026 23:38
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from 71c7ebc to 861c956 Compare October 15, 2026 23:39
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 052fed4 to 29222a5 Compare October 15, 2026 23:54
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from 861c956 to e4ae196 Compare October 15, 2026 23:54
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 29222a5 to 0d8789f Compare October 16, 2026 00:18
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from e4ae196 to efacab7 Compare October 16, 2026 00:18
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 0d8789f to e1d55ac Compare October 16, 2026 00:41
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from efacab7 to 9cba40d Compare October 16, 2026 00:41
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from e1d55ac to f10c878 Compare October 19, 2026 17:13
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from 9cba40d to bd93636 Compare October 19, 2026 17:13
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from f10c878 to db0a270 Compare October 19, 2026 21:44
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from bd93636 to c441a2f Compare October 19, 2026 21:44
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from db0a270 to 3bcd860 Compare October 19, 2026 22:35
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from c441a2f to 74d6edd Compare October 19, 2026 22:35
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from 3bcd860 to f2152d8 Compare October 19, 2026 22:37
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from 74d6edd to 502f0fc Compare October 19, 2026 22:37
@littlegrasscao
littlegrasscao force-pushed the server/split2/22-protocol-query-versions branch from f2152d8 to fbbbb98 Compare October 20, 2026 00:32
@littlegrasscao
littlegrasscao force-pushed the server/split2/23-protocol-query-changes branch from 502f0fc to 7a4ee80 Compare October 20, 2026 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant