Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 36 additions & 10 deletions rust/lance/src/dataset.rs
Original file line number Diff line number Diff line change
Expand Up @@ -738,16 +738,17 @@ impl Dataset {
let message_len =
LittleEndian::read_u32(&last_block[offset_in_block..offset_in_block + 4]) as usize;
let message_data = &last_block[offset_in_block + 4..offset_in_block + 4 + message_len];
let transaction: Transaction =
lance_table::format::pb::Transaction::decode(message_data)?.try_into()?;

let metadata_cache = session.metadata_cache.for_dataset(uri);
let metadata_key = TransactionKey {
version: manifest_location.version,
};
metadata_cache
.insert_with_key(&metadata_key, Arc::new(transaction))
.await;
if let Some(transaction) =
decode_inline_transaction(message_data, manifest_location.version)
{
let metadata_cache = session.metadata_cache.for_dataset(uri);
let metadata_key = TransactionKey {
version: manifest_location.version,
};
metadata_cache
.insert_with_key(&metadata_key, Arc::new(transaction))
.await;
}
}

if manifest.should_use_legacy_format() {
Expand Down Expand Up @@ -3662,6 +3663,31 @@ impl ManifestWriteConfig {
}
}

/// Decode an inline transaction section for opportunistic caching.
///
/// Returns `None` instead of failing when the transaction cannot be decoded:
/// the section may have been written by a newer version of Lance with an
/// operation type this version does not know, and that must not prevent
/// opening the dataset. Paths that need the transaction contents surface the
/// error at their call sites instead.
fn decode_inline_transaction(message_data: &[u8], version: u64) -> Option<Transaction> {
match lance_table::format::pb::Transaction::decode(message_data)
.map_err(Error::from)
.and_then(Transaction::try_from)
{
Comment on lines +3673 to +3677

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the relevant areas around decode_inline_transaction and Transaction::try_from
sed -n '3650,3705p' rust/lance/src/dataset.rs
printf '\n====\n'
sed -n '1,260p' rust/lance/src/dataset/transaction.rs
printf '\n====\n'
rg -n "MergedGeneration::try_from\\(|unwrap\\(|Transaction::try_from|decode_inline_transaction" rust/lance/src/dataset/transaction.rs rust/lance/src/dataset.rs

Repository: lance-format/lance

Length of output: 23925


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Find any tests or call sites that rely on decode_inline_transaction returning None on malformed data.
rg -n "decode_inline_transaction|inline transaction|Transaction::decode|MergedGeneration::try_from" rust/lance -g '!target'

Repository: lance-format/lance

Length of output: 2198


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the Transaction::try_from branches that unwrap MergedGeneration::try_from
sed -n '3160,3365p' rust/lance/src/dataset/transaction.rs
printf '\n====\n'
sed -n '6180,6245p' rust/lance/src/dataset/transaction.rs
printf '\n====\n'
sed -n '200,250p' rust/lance/src/dataset/tests/dataset_transactions.rs

Repository: lance-format/lance

Length of output: 12618


Propagate malformed inline Update conversions instead of panicking.

Transaction::try_from still unwraps MergedGeneration::try_from(m) in the Update and UpdateMemWalState paths, so a corrupt inline update can abort dataset opening instead of falling back to None. Return that conversion error and add a regression test.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance/src/dataset.rs` around lines 3673 - 3677, Update
Transaction::try_from in the Update and UpdateMemWalState conversion paths to
propagate MergedGeneration::try_from failures instead of unwrapping, allowing
decode_inline_transaction to return None for malformed inline updates. Add a
regression test covering corrupt inline update data during dataset opening.

Source: Coding guidelines

Ok(transaction) => Some(transaction),
Err(err) => {
log::warn!(
"Failed to decode the inline transaction of version {}; \
it may have been written by a newer version of Lance: {}",
version,
err
);
None
}
}
}

/// Commit a manifest file and create a copy at the latest manifest path.
#[allow(clippy::too_many_arguments)]
pub(crate) async fn write_manifest_file(
Expand Down
29 changes: 29 additions & 0 deletions rust/lance/src/dataset/tests/dataset_transactions.rs
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,35 @@ async fn test_session_store_registry() {
assert_eq!(registry.active_stores().len(), 0);
}

#[test]
fn test_decode_inline_transaction_tolerates_unknown_operations() {
use crate::dataset::decode_inline_transaction;
use lance_table::format::pb;
use prost::Message;
Comment on lines +213 to +215

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Move these imports to module scope.

The test-local use declarations should be placed at the top of rust/lance/src/dataset/tests/dataset_transactions.rs.

As per coding guidelines: Rust use imports must be placed at the top of the file, not inside function bodies.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance/src/dataset/tests/dataset_transactions.rs` around lines 213 - 215,
Move the `decode_inline_transaction`, `lance_table::format::pb`, and
`prost::Message` imports from the test function to module scope at the top of
`dataset_transactions.rs`, leaving their usage unchanged.

Source: Coding guidelines


// A transaction written by a newer version of Lance may carry an operation
// this version cannot decode; prost surfaces it as a missing oneof. This
// must not fail (it would prevent opening the dataset), only skip caching.
let unknown_operation = pb::Transaction {
read_version: 1,
uuid: "test".to_string(),
..Default::default()
};
assert!(decode_inline_transaction(&unknown_operation.encode_to_vec(), 42).is_none());
Comment on lines +220 to +225

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the relevant test and decoder code
git ls-files | rg 'rust/lance/src/dataset/tests/dataset_transactions.rs|decode_inline_transaction|transaction'
rg -n "decode_inline_transaction|unknown_operation|operation: None|Transaction" rust/lance/src/dataset/tests/dataset_transactions.rs rust/lance/src -g '!**/target/**'

# Show the surrounding test lines
sed -n '180,260p' rust/lance/src/dataset/tests/dataset_transactions.rs

# Find the protobuf definition for Transaction
rg -n "message Transaction|oneof operation|operation =" rust -g '*.proto' -g '*.rs'

Repository: lance-format/lance

Length of output: 48965


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect decoder implementation and protobuf-generated accessors around the transaction type
rg -n "fn decode_inline_transaction|decode_inline_transaction" rust/lance/src -g '*.rs'
rg -n "pub struct Transaction|impl Transaction|enum Operation|oneof" rust -g '*.rs' -g '*.proto'

# If the generated Rust file is available, inspect the relevant section
python3 - <<'PY'
from pathlib import Path
paths = list(Path('rust').rglob('*.rs'))
for p in paths:
    txt = p.read_text(errors='ignore')
    if 'decode_inline_transaction' in txt or 'struct Transaction' in txt and 'uuid' in txt and 'read_version' in txt:
        print(p)
PY

Repository: lance-format/lance

Length of output: 2822


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Show the decoder and protobuf conversion paths around the relevant match arms
sed -n '3060,3370p' rust/lance/src/dataset/transaction.rs
printf '\n---\n'
sed -n '3438,3478p' rust/lance/src/dataset/transaction.rs
printf '\n---\n'
sed -n '3668,3692p' rust/lance/src/dataset.rs

# Inspect the protobuf schema for Transaction
sed -n '1,120p' protos/transaction.proto

Repository: lance-format/lance

Length of output: 20014


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Look for any existing test that intentionally appends unknown protobuf fields
rg -n "unknown field|unrecognized|append.*field|varint|tag" rust/lance/src rust/lance-table/src protos -g '*.rs' -g '*.proto'

Repository: lance-format/lance

Length of output: 50375


Split the missing-operation and unknown-operation cases. This fixture only leaves operation unset, so it exercises the missing-operation path. Keep that case separate and use bytes with an unrecognized operation field here.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance/src/dataset/tests/dataset_transactions.rs` around lines 220 - 225,
Update the transaction test around unknown_operation so it encodes a non-empty,
unrecognized operation field, ensuring this assertion exercises the
unknown-operation path. Preserve the existing fixture separately for the
missing-operation case, where operation remains unset, and keep the expected
decode_inline_transaction result as None.


// Corrupt bytes are likewise tolerated.
assert!(decode_inline_transaction(&[0xff, 0xff, 0xff], 42).is_none());

// A decodable transaction is returned.
let known = pb::Transaction::from(&Transaction::new(
1,
Operation::Append { fragments: vec![] },
None,
));
let decoded = decode_inline_transaction(&known.encode_to_vec(), 42).unwrap();
assert!(matches!(decoded.operation, Operation::Append { .. }));
}

#[tokio::test]
async fn test_migrate_v2_manifest_paths() {
let test_uri = TempStrDir::default();
Expand Down
Loading