Skip to content

feat(lance): support TIME type #478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lxy-9602 merged 1 commit into from
Oct 28, 2026
Merged

feat(lance): support TIME type #478

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion docs/source/user_guide/data_types.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,13 +146,29 @@ and `Arrow DataTypes <https://arrow.apache.org/docs/format/Columnar.html#data-ty
* - ``TIME``

``TIME(p)``
- Not Supported
- Time32
- Data type of a time without time zone consisting of hour:minute:second[.fractional] with up to nanosecond precision and values ranging from 00:00:00.000000000 to 23:59:59.999999999.

The type can be declared using ``TIME(p)`` where p is the number of digits of
fractional seconds (precision). p must have a value between 0 and 9
(both inclusive). If no precision is specified, p is equal to 0.

Paimon C++ maps every ``TIME(p)`` to an Arrow ``Time32`` in milliseconds
(``time32[ms]``) and accepts no other Arrow time type. The declared
precision p is kept in the ``paimon.time.precision`` field metadata; a
``Time32`` field without it is ``TIME(0)``. As in Paimon Java, values are
stored as milliseconds of the day whatever p is, so ``TIME(6)`` and
``TIME(9)`` values keep only millisecond precision.

Paimon C++ reads TIME columns from Parquet, Lance and Vortex data files
and writes them to Lance and Vortex data files. ORC and Avro data files
do not support TIME.

**Note:** Only append-only tables support TIME columns. Primary-key
tables do not support TIME, whether as a primary key or as a value
column. A TIME column also cannot be a partition or bucket key, and
predicates cannot reference TIME columns.

Comment thread
SteNicholas marked this conversation as resolved.
Resolved
* - ``TIMESTAMP``

``TIMESTAMP(p)``
Expand Down
1 change: 1 addition & 0 deletions src/paimon/core/schema/schema_validation.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1001,6 +1001,7 @@ Status SchemaValidation::ValidateLanceDataField(const std::shared_ptr<arrow::Fie
case arrow::Type::FLOAT:
case arrow::Type::DOUBLE:
case arrow::Type::DATE32:
case arrow::Type::TIME32:
case arrow::Type::STRING:
case arrow::Type::BINARY:
case arrow::Type::DECIMAL128:
Expand Down
12 changes: 6 additions & 6 deletions src/paimon/core/schema/schema_validation_test.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,9 +437,13 @@ TEST(SchemaValidationTest, TestLanceDataTypes) {
arrow::field("string", arrow::utf8()),
arrow::field("binary", arrow::binary()),
arrow::field("date", arrow::date32()),
arrow::field("time", arrow::time32(arrow::TimeUnit::MILLI)),
arrow::field("timestamp", arrow::timestamp(arrow::TimeUnit::MICRO)),
arrow::field("decimal", arrow::decimal128(38, 2)),
arrow::field("array", arrow::list(arrow::float32())),
arrow::field("nested_time",
arrow::struct_({arrow::field(
"values", arrow::list(arrow::time32(arrow::TimeUnit::MILLI)))})),
arrow::field("row", arrow::struct_({arrow::field("value", arrow::int32())}),
/*nullable=*/false),
arrow::field("nullable_row", arrow::struct_({arrow::field("value", arrow::int32())})),
Expand Down Expand Up @@ -472,13 +476,9 @@ TEST(SchemaValidationTest, TestLanceDataTypes) {
arrow::field("map", arrow::map(arrow::int32(), arrow::utf8())),
arrow::field("ltz", arrow::timestamp(arrow::TimeUnit::MICRO, "UTC")),
VariantTypeUtils::ToArrowField("variant"),
arrow::field("time_millis", arrow::time32(arrow::TimeUnit::MILLI)),
arrow::field("nested_time",
arrow::struct_({arrow::field(
"values", arrow::list(arrow::time32(arrow::TimeUnit::MILLI)))})),
};
std::vector<std::string> expected_errors = {"type MAP", "LOCAL_ZONED_TIMESTAMP", "type VARIANT",
"type time32", "type time32"};
std::vector<std::string> expected_errors = {"type MAP", "LOCAL_ZONED_TIMESTAMP",
"type VARIANT"};
for (size_t i = 0; i < unsupported_fields.size(); ++i) {
ASSERT_OK_AND_ASSIGN(
table_schema,
Expand Down
46 changes: 38 additions & 8 deletions src/paimon/format/lance/lance_file_format_test.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
#include "arrow/ipc/json_simple.h"
#include "gtest/gtest.h"
#include "paimon/common/types/data_field.h"
#include "paimon/common/types/data_type.h"
#include "paimon/common/utils/arrow/mem_utils.h"
#include "paimon/common/utils/arrow/status_utils.h"
#include "paimon/common/utils/checked_cast.h"
Expand Down Expand Up @@ -176,6 +177,11 @@ TEST(LanceUtilsTest, ConvertsObjectStoreOptions) {
}

TEST_F(LanceFileFormatTest, WriteThenReadSupportedTypes) {
auto time_field = [](const std::string& name, int32_t precision) {
return arrow::field(
name, arrow::time32(arrow::TimeUnit::MILLI), /*nullable=*/true,
arrow::key_value_metadata({DataType::kTimePrecision}, {std::to_string(precision)}));
};
arrow::FieldVector fields = {
arrow::field("bool_col", arrow::boolean()),
arrow::field("i8_col", arrow::int8()),
Expand All @@ -187,22 +193,30 @@ TEST_F(LanceFileFormatTest, WriteThenReadSupportedTypes) {
arrow::field("string_col", arrow::utf8()),
arrow::field("binary_col", arrow::binary()),
arrow::field("date_col", arrow::date32()),
time_field("time_col", 3),
arrow::field("timestamp_col", arrow::timestamp(arrow::TimeUnit::MICRO)),
arrow::field("decimal_col", arrow::decimal128(12, 2)),
arrow::field("array_col", arrow::list(arrow::float32())),
arrow::field("row_col", arrow::struct_({arrow::field("value", arrow::int32())})),
arrow::field(
"vector_col",
arrow::fixed_size_list(arrow::field("item", arrow::float32(), /*nullable=*/false), 2)),
arrow::field("time_array_col", arrow::list(time_field("item", 0))),
arrow::field("time_row_col", arrow::struct_({time_field("value", 6)})),
arrow::field("time_row_array_col",
arrow::struct_({arrow::field("values", arrow::list(time_field("item", 9)))})),
};
std::shared_ptr<arrow::Schema> schema = arrow::schema(fields);
std::shared_ptr<arrow::Array> expected =
arrow::ipc::internal::json::ArrayFromJSON(arrow::struct_(fields), R"([
[true,-1,-2,-3,-4,1.5,2.5,"one","abc",1,
"1970-01-01 00:00:00.000001","12.34",[1.0,2.0],[7],[3.0,4.0]],
[false,1,2,3,4,3.5,4.5,"two","xyz",2,
"2030-12-31 23:59:59.999999","-12.34",[],[8],[5.0,6.0]],
[null,null,null,null,null,null,null,null,null,null,null,null,null,[null],null]
[true,-1,-2,-3,-4,1.5,2.5,"one","abc",1,0,
"1970-01-01 00:00:00.000001","12.34",[1.0,2.0],[7],[3.0,4.0],
[0,null,45296000],[45296123],[[3723123,null]]],
[false,1,2,3,4,3.5,4.5,"two","xyz",2,86399999,
"2030-12-31 23:59:59.999999","-12.34",[],[8],[5.0,6.0],
[],[86399999],[[]]],
[null,null,null,null,null,null,null,null,null,null,null,null,null,null,[null],null,
null,[null],[null]]
])")
.ValueOrDie();
std::string path = PathUtil::JoinPath(directory_->Str(), "supported-types.lance");
Expand All @@ -219,6 +233,19 @@ TEST_F(LanceFileFormatTest, WriteThenReadSupportedTypes) {
ASSERT_TRUE(actual->Equals(arrow::ChunkedArray(expected))) << actual->ToString() << "\nvs\n"
<< expected->ToString();
}

ASSERT_OK_AND_ASSIGN(std::unique_ptr<FileBatchReader> reader, OpenReader(path, 2));
::ArrowSchema ffi_schema = {};
ASSERT_TRUE(arrow::ExportSchema(*schema, &ffi_schema).ok());
ASSERT_OK(reader->SetReadSchema(&ffi_schema, /*predicate=*/nullptr,
/*selection_bitmap=*/std::nullopt));
ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatch batch, reader->NextBatch());
arrow::Result<std::shared_ptr<arrow::Array>> batch_result =
arrow::ImportArray(batch.first.get(), batch.second.get());
ASSERT_TRUE(batch_result.ok()) << batch_result.status().ToString();
std::shared_ptr<arrow::Array> batch_array = std::move(batch_result).MoveValueUnsafe();
ASSERT_TRUE(batch_array->type()->Equals(arrow::struct_(fields), /*check_metadata=*/true))
<< batch_array->type()->ToString(/*show_metadata=*/true);
}

TEST_F(LanceFileFormatTest, RejectsNullTopLevelRows) {
Expand Down Expand Up @@ -351,10 +378,12 @@ TEST_F(LanceFileFormatTest, EmptyProjectionAndEmptySelection) {

TEST_F(LanceFileFormatTest, ExtractStatistics) {
arrow::FieldVector fields = {arrow::field("id", arrow::int32()),
arrow::field("name", arrow::utf8())};
arrow::field("name", arrow::utf8()),
arrow::field("time", arrow::time32(arrow::TimeUnit::MILLI))};
std::shared_ptr<arrow::Schema> schema = arrow::schema(fields);
std::shared_ptr<arrow::Array> data =
arrow::ipc::internal::json::ArrayFromJSON(arrow::struct_(fields), R"([[1,"a"],[2,null]])")
arrow::ipc::internal::json::ArrayFromJSON(arrow::struct_(fields),
R"([[1,"a",3723123],[2,null,null]])")
.ValueOrDie();
std::string path = PathUtil::JoinPath(directory_->Str(), "stats.lance");
ASSERT_OK(WriteFile(path, schema, data, /*batch_size=*/2));
Expand All @@ -365,9 +394,10 @@ TEST_F(LanceFileFormatTest, ExtractStatistics) {
format_->CreateStatsExtractor(&ffi_schema));
ASSERT_OK_AND_ASSIGN(auto result, extractor->ExtractWithFileInfo(file_system_, path, pool_));
ASSERT_EQ(result.second.GetRowCount(), 2);
ASSERT_EQ(result.first.size(), 2);
ASSERT_EQ(result.first.size(), 3);
ASSERT_EQ(result.first[0]->ToString(), "min null, max null, null count null");
ASSERT_EQ(result.first[1]->ToString(), "min null, max null, null count null");
ASSERT_EQ(result.first[2]->ToString(), "min null, max null, null count null");
}

TEST_F(LanceFileFormatTest, RejectsIncompatibleReadTypes) {
Expand Down
2 changes: 2 additions & 0 deletions src/paimon/format/lance/lance_stats_extractor.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,8 @@ Result<std::unique_ptr<ColumnStats>> LanceStatsExtractor::CreateEmptyStats(
return ColumnStats::CreateStringColumnStats(std::nullopt, std::nullopt, std::nullopt);
case arrow::Type::DATE32:
return ColumnStats::CreateDateColumnStats(std::nullopt, std::nullopt, std::nullopt);
case arrow::Type::TIME32:
return ColumnStats::CreateIntColumnStats(std::nullopt, std::nullopt, std::nullopt);
case arrow::Type::TIMESTAMP: {
std::shared_ptr<arrow::TimestampType> timestamp_type =
checked_pointer_cast<arrow::TimestampType>(type);
Expand Down
24 changes: 14 additions & 10 deletions test/inte/scan_and_read_inte_test.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3479,6 +3479,8 @@ TEST_F(ScanAndReadInteTest, TestLanceJavaCompatibility) {
arrow::field("f_binary", arrow::binary()),
arrow::field("f_varbinary", arrow::binary()),
arrow::field("f_date", arrow::date32()),
arrow::field("f_time_0", arrow::time32(arrow::TimeUnit::MILLI)),
arrow::field("f_time_3", arrow::time32(arrow::TimeUnit::MILLI)),
arrow::field("f_ts_0", arrow::timestamp(arrow::TimeUnit::SECOND)),
arrow::field("f_ts_3", arrow::timestamp(arrow::TimeUnit::MILLI)),
arrow::field("f_ts_6", arrow::timestamp(arrow::TimeUnit::MICRO)),
Expand All @@ -3497,10 +3499,10 @@ TEST_F(ScanAndReadInteTest, TestLanceJavaCompatibility) {
// from null parents; see the README alongside the Java compatibility table.
auto expected = std::make_shared<arrow::ChunkedArray>(
arrow::ipc::internal::json::ArrayFromJSON(arrow::struct_(fields), R"([
[0, 1, true, -5, -1000, 10000000001, 1.25, -2.5, "char0001", "value-1", "bin1", "\u0000\u0001\u0002\u007f", -1, "1970-01-01 00:00:01", "1970-01-01 00:00:01.123", "1970-01-01 00:00:01.123456", "1970-01-01 00:00:01.123456789", "-3", "1.25", "12345678901234567.89", "12345678901234567890.123456789012345678", [1, null, -1], [[1, null], null, []], [1, "required"], [1, "nullable"], [1.0, -1.5, 0.25]],
[0, 2, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, [null, null], [null, null], null],
[0, 10, false, -5, -1000, 10000000010, 1.25, -2.5, "char0001", "", "bin1", "\u0000\u0001\u0002\u007f", 20000, "1970-01-01 00:00:01", "1970-01-01 00:00:01.123", "1970-01-01 00:00:01.123456", "1970-01-01 00:00:01.123456789", "-3", "1.25", "12345678901234567.89", "12345678901234567890.123456789012345678", [], [[10, null], null, []], [10, "required"], [10, "nullable"], [10.0, -1.5, 0.25]],
[0, 11, true, -5, -1000, 10000000011, 1.25, -2.5, "char0001", "value-11", "bin1", "\u0000\u0001\u0002\u007f", 20000, "1970-01-01 00:00:01", "1970-01-01 00:00:01.123", "1970-01-01 00:00:01.123456", "1970-01-01 00:00:01.123456789", "-3", "1.25", "12345678901234567.89", "12345678901234567890.123456789012345678", [11, null, -11], [[11, null], null, []], [11, "required"], [11, "nullable"], [11.0, -1.5, 0.25]]
[0, 1, true, -5, -1000, 10000000001, 1.25, -2.5, "char0001", "value-1", "bin1", "\u0000\u0001\u0002\u007f", -1, 0, 86399999, "1970-01-01 00:00:01", "1970-01-01 00:00:01.123", "1970-01-01 00:00:01.123456", "1970-01-01 00:00:01.123456789", "-3", "1.25", "12345678901234567.89", "12345678901234567890.123456789012345678", [1, null, -1], [[1, null], null, []], [1, "required"], [1, "nullable"], [1.0, -1.5, 0.25]],
[0, 2, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, [null, null], [null, null], null],
[0, 10, false, -5, -1000, 10000000010, 1.25, -2.5, "char0001", "", "bin1", "\u0000\u0001\u0002\u007f", 20000, 45296000, 1123, "1970-01-01 00:00:01", "1970-01-01 00:00:01.123", "1970-01-01 00:00:01.123456", "1970-01-01 00:00:01.123456789", "-3", "1.25", "12345678901234567.89", "12345678901234567890.123456789012345678", [], [[10, null], null, []], [10, "required"], [10, "nullable"], [10.0, -1.5, 0.25]],
[0, 11, true, -5, -1000, 10000000011, 1.25, -2.5, "char0001", "value-11", "bin1", "\u0000\u0001\u0002\u007f", 20000, 86399000, 45296789, "1970-01-01 00:00:01", "1970-01-01 00:00:01.123", "1970-01-01 00:00:01.123456", "1970-01-01 00:00:01.123456789", "-3", "1.25", "12345678901234567.89", "12345678901234567890.123456789012345678", [11, null, -11], [[11, null], null, []], [11, "required"], [11, "nullable"], [11.0, -1.5, 0.25]]
])")
.ValueOrDie());
const std::string table_path = GetDataDir() + "/lance/append_java_compat.db/append_java_compat";
Expand Down Expand Up @@ -3546,7 +3548,7 @@ TEST_F(ScanAndReadInteTest, TestLanceJavaCompatibility) {

auto projected = std::make_shared<arrow::ChunkedArray>(
arrow::ipc::internal::json::ArrayFromJSON(
arrow::struct_({fields[0], fields[24], fields[1], fields[25]}), R"([
arrow::struct_({fields[0], fields[26], fields[1], fields[27]}), R"([
[0, [1, "nullable"], 1, [1.0, -1.5, 0.25]],
[0, [null, null], 2, null],
[0, [10, "nullable"], 10, [10.0, -1.5, 0.25]],
Expand All @@ -3570,6 +3572,8 @@ TEST_F(ScanAndReadInteTest, TestLancePythonCompatibility) {
arrow::field("f_string", arrow::utf8()),
arrow::field("f_binary", arrow::binary()),
arrow::field("f_date", arrow::date32()),
arrow::field("f_time_0", arrow::time32(arrow::TimeUnit::MILLI)),
arrow::field("f_time_3", arrow::time32(arrow::TimeUnit::MILLI)),
arrow::field("f_ts_0", arrow::timestamp(arrow::TimeUnit::SECOND)),
arrow::field("f_ts_3", arrow::timestamp(arrow::TimeUnit::MILLI)),
arrow::field("f_ts_6", arrow::timestamp(arrow::TimeUnit::MICRO)),
Expand All @@ -3586,10 +3590,10 @@ TEST_F(ScanAndReadInteTest, TestLancePythonCompatibility) {
};
auto expected = std::make_shared<arrow::ChunkedArray>(
arrow::ipc::internal::json::ArrayFromJSON(arrow::struct_(fields), R"([
[0, 1, true, -5, -1000, 10000000001, 1.25, -2.5, "value-1", "\u0000\u0001\u0002\u007f", -1, "1970-01-01 00:00:01", "1970-01-01 00:00:01.123", "1970-01-01 00:00:01.123456", "1970-01-01 00:00:01.123456789", "-3", "1.25", "12345678901234567.89", "12345678901234567890.123456789012345678", [1, null, -1], [[1, null], null, []], [1, "row-1"], [1.0, -1.5, 0.25]],
[0, 2, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, [null, null], null],
[0, 10, false, -5, -1000, 10000000010, 1.25, -2.5, "", "\u0000\u0001\u0002\u007f", 20000, "1970-01-01 00:00:01", "1970-01-01 00:00:01.123", "1970-01-01 00:00:01.123456", "1970-01-01 00:00:01.123456789", "-3", "1.25", "12345678901234567.89", "12345678901234567890.123456789012345678", [], [[10, null], null, []], [10, "row-10"], [10.0, -1.5, 0.25]],
[0, 11, true, -5, -1000, 10000000011, 1.25, -2.5, "value-11", "\u0000\u0001\u0002\u007f", 20000, "1970-01-01 00:00:01", "1970-01-01 00:00:01.123", "1970-01-01 00:00:01.123456", "1970-01-01 00:00:01.123456789", "-3", "1.25", "12345678901234567.89", "12345678901234567890.123456789012345678", [11, null, -11], [[11, null], null, []], [11, "row-11"], [11.0, -1.5, 0.25]]
[0, 1, true, -5, -1000, 10000000001, 1.25, -2.5, "value-1", "\u0000\u0001\u0002\u007f", -1, 0, 86399999, "1970-01-01 00:00:01", "1970-01-01 00:00:01.123", "1970-01-01 00:00:01.123456", "1970-01-01 00:00:01.123456789", "-3", "1.25", "12345678901234567.89", "12345678901234567890.123456789012345678", [1, null, -1], [[1, null], null, []], [1, "row-1"], [1.0, -1.5, 0.25]],
[0, 2, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, [null, null], null],
[0, 10, false, -5, -1000, 10000000010, 1.25, -2.5, "", "\u0000\u0001\u0002\u007f", 20000, 45296000, 1123, "1970-01-01 00:00:01", "1970-01-01 00:00:01.123", "1970-01-01 00:00:01.123456", "1970-01-01 00:00:01.123456789", "-3", "1.25", "12345678901234567.89", "12345678901234567890.123456789012345678", [], [[10, null], null, []], [10, "row-10"], [10.0, -1.5, 0.25]],
[0, 11, true, -5, -1000, 10000000011, 1.25, -2.5, "value-11", "\u0000\u0001\u0002\u007f", 20000, 86399000, 45296789, "1970-01-01 00:00:01", "1970-01-01 00:00:01.123", "1970-01-01 00:00:01.123456", "1970-01-01 00:00:01.123456789", "-3", "1.25", "12345678901234567.89", "12345678901234567890.123456789012345678", [11, null, -11], [[11, null], null, []], [11, "row-11"], [11.0, -1.5, 0.25]]
])")
.ValueOrDie());
const std::string table_path =
Expand Down Expand Up @@ -3623,7 +3627,7 @@ TEST_F(ScanAndReadInteTest, TestLancePythonCompatibility) {
check(2, /*projection=*/{}, expected);
auto projected = std::make_shared<arrow::ChunkedArray>(
arrow::ipc::internal::json::ArrayFromJSON(
arrow::struct_({fields[0], fields[21], fields[1], fields[22], fields[19]}), R"([
arrow::struct_({fields[0], fields[23], fields[1], fields[24], fields[21]}), R"([
[0, [1, "row-1"], 1, [1.0, -1.5, 0.25], [1, null, -1]],
[0, [null, null], 2, null, null],
[0, [10, "row-10"], 10, [10.0, -1.5, 0.25], []],
Expand Down
Loading
Loading