Skip to content

fix(dataset): reject all system column names on write #7797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wjones127 merged 1 commit into from
Jul 15, 2026
+38 −3
Merged

fix(dataset): reject all system column names on write #7797

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 38 additions & 3 deletions rust/lance/src/dataset/write/insert.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ use arrow_array::{RecordBatch, RecordBatchIterator};
use datafusion::execution::SendableRecordBatchStream;
use humantime::format_duration;
use lance_core::datatypes::{NullabilityComparison, Schema, SchemaCompareOptions};
use lance_core::is_system_column;
use lance_core::utils::tracing::{DATASET_WRITING_EVENT, TRACE_DATASET_EVENTS};
use lance_core::{ROW_ADDR, ROW_ID, ROW_OFFSET};
use lance_datafusion::utils::StreamingWriteSource;
use lance_file::version::LanceFileVersion;
use lance_io::object_store::ObjectStore;
Expand Down Expand Up @@ -328,9 +328,12 @@ impl<'a> InsertBuilder<'a> {
normalized_data_schema.check_compatible(dataset.schema(), &schema_cmp_opts)?;
}

// Make sure we aren't using any reserved column names
// The system columns (`_rowid`, `_rowaddr`, `_rowoffset`, and the row-version
// columns) are virtual: they're injected into scan results at read time and
// never stored. A stored column sharing one of these names would collide with
// the system column on read, so reject it at write time.
for field in data_schema.fields.iter() {
if field.name == ROW_ID || field.name == ROW_ADDR || field.name == ROW_OFFSET {
if is_system_column(&field.name) {
return Err(Error::invalid_input_source(
format!(
"The column {} is a reserved name and cannot be used in a Lance dataset",
Expand Down Expand Up @@ -506,6 +509,38 @@ mod test {
);
}

#[rstest::rstest]
#[case::row_id("_rowid")]
#[case::row_addr("_rowaddr")]
#[case::row_offset("_rowoffset")]
#[case::row_created_at_version("_row_created_at_version")]
#[case::row_last_updated_at_version("_row_last_updated_at_version")]
#[tokio::test]
async fn rejects_reserved_system_column_names(#[case] reserved_name: &str) {
// Every system column name must be rejected on write. The row-version
// columns (`_row_created_at_version`, `_row_last_updated_at_version`) are
// computed at read time and appended by `Projection::to_schema`; a user
// data column sharing one of those names would otherwise pass ingest and
// later collide with the appended field.
let schema = Arc::new(Schema::new(vec![Field::new(
reserved_name,
DataType::Int32,
false,
)]));
let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(Int32Array::from(vec![1]))])
.unwrap();

let result = InsertBuilder::new("memory://")
.execute_stream(RecordBatchIterator::new(vec![Ok(batch)], schema.clone()))
.await;

let err = result.expect_err("writing a reserved system column name should fail");
assert!(
err.to_string().contains("reserved name"),
"unexpected error for {reserved_name}: {err}"
);
Comment on lines +537 to +541

@coderabbitai coderabbitai Bot Jul 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Assert the error variant and the specific column name.

This test currently validates only a generic message fragment. Match Error::InvalidInput and assert that the message contains reserved_name, so unrelated failures cannot pass the test.

As per coding guidelines: “Assert on both the error variant and the message content in tests.”

Proposed fix 
-        let err = result.expect_err("writing a reserved system column name should fail");
-        assert!(
-            err.to_string().contains("reserved name"),
-            "unexpected error for {reserved_name}: {err}"
-        );
+        let err = result.expect_err("writing a reserved system column name should fail");
+        match err {
+            Error::InvalidInput { source, .. } => {
+                let message = source.to_string();
+                assert!(message.contains("reserved name"));
+                assert!(message.contains(reserved_name));
+            }
+            other => panic!("expected InvalidInput, got: {other:?}"),
+        }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
let err = result.expect_err("writing a reserved system column name should fail");
assert!(
err.to_string().contains("reserved name"),
"unexpected error for {reserved_name}: {err}"
);
let err = result.expect_err("writing a reserved system column name should fail");
match err {
Error::InvalidInput { source, .. } => {
let message = source.to_string();
assert!(message.contains("reserved name"));
assert!(message.contains(reserved_name));
}
other => panic!("expected InvalidInput, got: {other:?}"),
}
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance/src/dataset/write/insert.rs` around lines 537 - 541, Update the
error assertion in the reserved system-column test to match the returned error
as Error::InvalidInput and verify its message contains the specific
reserved_name value. Replace the generic “reserved name” substring check while
preserving the existing unexpected-error context.

Source: Coding guidelines

}

#[tokio::test]
async fn allow_overwrite_to_v2_2_without_blob_upgrade() {
let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int32, false)]));
Expand Down
Loading