-
Notifications
You must be signed in to change notification settings - Fork 768
fix: don't fail dataset open on an undecodable inline transaction #7740
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -208,6 +208,35 @@ async fn test_session_store_registry() { | |
| assert_eq!(registry.active_stores().len(), 0); | ||
| } | ||
|
|
||
| #[test] | ||
| fn test_decode_inline_transaction_tolerates_unknown_operations() { | ||
| use crate::dataset::decode_inline_transaction; | ||
| use lance_table::format::pb; | ||
| use prost::Message; | ||
|
Comment on lines
+213
to
+215
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win Move these imports to module scope. The test-local As per coding guidelines: Rust 🤖 Prompt for AI AgentsSource: Coding guidelines |
||
|
|
||
| // A transaction written by a newer version of Lance may carry an operation | ||
| // this version cannot decode; prost surfaces it as a missing oneof. This | ||
| // must not fail (it would prevent opening the dataset), only skip caching. | ||
| let unknown_operation = pb::Transaction { | ||
| read_version: 1, | ||
| uuid: "test".to_string(), | ||
| ..Default::default() | ||
| }; | ||
| assert!(decode_inline_transaction(&unknown_operation.encode_to_vec(), 42).is_none()); | ||
|
Comment on lines
+220
to
+225
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
# Inspect the relevant test and decoder code
git ls-files | rg 'rust/lance/src/dataset/tests/dataset_transactions.rs|decode_inline_transaction|transaction'
rg -n "decode_inline_transaction|unknown_operation|operation: None|Transaction" rust/lance/src/dataset/tests/dataset_transactions.rs rust/lance/src -g '!**/target/**'
# Show the surrounding test lines
sed -n '180,260p' rust/lance/src/dataset/tests/dataset_transactions.rs
# Find the protobuf definition for Transaction
rg -n "message Transaction|oneof operation|operation =" rust -g '*.proto' -g '*.rs'Repository: lance-format/lance Length of output: 48965 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Inspect decoder implementation and protobuf-generated accessors around the transaction type
rg -n "fn decode_inline_transaction|decode_inline_transaction" rust/lance/src -g '*.rs'
rg -n "pub struct Transaction|impl Transaction|enum Operation|oneof" rust -g '*.rs' -g '*.proto'
# If the generated Rust file is available, inspect the relevant section
python3 - <<'PY'
from pathlib import Path
paths = list(Path('rust').rglob('*.rs'))
for p in paths:
txt = p.read_text(errors='ignore')
if 'decode_inline_transaction' in txt or 'struct Transaction' in txt and 'uuid' in txt and 'read_version' in txt:
print(p)
PYRepository: lance-format/lance Length of output: 2822 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Show the decoder and protobuf conversion paths around the relevant match arms
sed -n '3060,3370p' rust/lance/src/dataset/transaction.rs
printf '\n---\n'
sed -n '3438,3478p' rust/lance/src/dataset/transaction.rs
printf '\n---\n'
sed -n '3668,3692p' rust/lance/src/dataset.rs
# Inspect the protobuf schema for Transaction
sed -n '1,120p' protos/transaction.protoRepository: lance-format/lance Length of output: 20014 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Look for any existing test that intentionally appends unknown protobuf fields
rg -n "unknown field|unrecognized|append.*field|varint|tag" rust/lance/src rust/lance-table/src protos -g '*.rs' -g '*.proto'Repository: lance-format/lance Length of output: 50375 Split the missing-operation and unknown-operation cases. This fixture only leaves 🤖 Prompt for AI Agents |
||
|
|
||
| // Corrupt bytes are likewise tolerated. | ||
| assert!(decode_inline_transaction(&[0xff, 0xff, 0xff], 42).is_none()); | ||
|
|
||
| // A decodable transaction is returned. | ||
| let known = pb::Transaction::from(&Transaction::new( | ||
| 1, | ||
| Operation::Append { fragments: vec![] }, | ||
| None, | ||
| )); | ||
| let decoded = decode_inline_transaction(&known.encode_to_vec(), 42).unwrap(); | ||
| assert!(matches!(decoded.operation, Operation::Append { .. })); | ||
| } | ||
|
|
||
| #[tokio::test] | ||
| async fn test_migrate_v2_manifest_paths() { | ||
| let test_uri = TempStrDir::default(); | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
Repository: lance-format/lance
Length of output: 23925
🏁 Script executed:
Repository: lance-format/lance
Length of output: 2198
🏁 Script executed:
Repository: lance-format/lance
Length of output: 12618
Propagate malformed inline Update conversions instead of panicking.
Transaction::try_fromstill unwrapsMergedGeneration::try_from(m)in theUpdateandUpdateMemWalStatepaths, so a corrupt inline update can abort dataset opening instead of falling back toNone. Return that conversion error and add a regression test.🤖 Prompt for AI Agents
Source: Coding guidelines