feat: add CompositeOperation for multi-statement transactions#7737
Draft
jackye1995 wants to merge 3 commits into
Draft
feat: add CompositeOperation for multi-statement transactions#7737jackye1995 wants to merge 3 commits into
jackye1995 wants to merge 3 commits into
Conversation
Adds a Composite operation that applies an ordered list of existing operations as one atomic commit. Sub-operations reuse the existing build_manifest apply logic (a sequential fold with the version pinned to the single committed version) and the existing TransactionRebase conflict rules by delegation on both sides of a conflict check. Composite transactions are not inlined into the manifest so released readers can still open post-composite versions; the eager inline transaction decode in load_manifest is also made tolerant so future operation types no longer prevent opening a dataset. Restore, Clone, nested composites, and empty composites are rejected.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
github-actions
Bot
added
A-python
- Reject sub-operations that reference fragment ids assigned to fragments added earlier in the same composite: those ids shift when the commit is rebuilt on conflict retry and would silently retarget another writer's fragment. Explicitly reserved ids stay allowed. - Apply the outer transaction's tag to the final manifest. - Keep the inline transaction section when disable_transaction_file is set so a composite commit always leaves a transaction record. - Stamp row-version metadata with the true previous version for non-first Update sub-operations on stable-row-id datasets. - Advance the outer read_version after sub-operation rebases. - Recognize overwrite-leading composites in Dataset::commit and the CommitBuilder storage-format check. - Surface unsupported operations as errors instead of panics across the JNI boundary; warn when affected_rows is ignored for composites.
Composites now write the inline manifest transaction section normally. The inability of released readers to decode an unknown inline operation is an independent robustness problem: the opportunistic decode on dataset open now tolerates undecodable transactions (log and skip) instead of failing the open, which protects against any future operation type.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
CompositeOperation: an ordered list of existing operations applied sequentially as one atomic commit, enabling multi-statement transactions such as an atomic append + index update (with fragment IDs reserved viaReserveFragmentsso index metadata can reference them).Design
CompositeOperation { repeated Transaction transactions }as oneof field 116 intransaction.proto(field 115 is intentionally left for the in-flight feat(transaction): compound append + add-index via action transactions #7674). Only each sub-transaction'soperationis meaningful; the outer transaction governsread_version/uuid/tag/transaction_properties.build_manifestfolds sub-operations sequentially through intermediate manifests, with the version pinned to the single committed version — row-level version metadata on stable-row-id datasets is stamped with the final version. Each sub-operation is validated against the intermediate manifest it actually applies to.TransactionRebaseper sub-operation; a committed composite is decomposed at the top ofcheck_txn. Two composites conflict iff any pair of their sub-operations conflicts.Restore, noClone(both are applied outsidebuild_manifestin the commit driver), no nesting, non-empty. Anything else composes, including Overwrite-first to atomically create a dataset with config.LanceOperation.Composite(operations=[...])through the existingLanceDataset.commit. Java binding will follow in a separate PR.