From 3cd26f24cf7142ebd8e3fde60aad2e1f45e267af Mon Sep 17 00:00:00 2001 From: seokjin0414 Date: Thu, 24 Sep 2026 23:53:19 +0900 Subject: [PATCH 1/6] feat(connectors): add Apache Fluss source connector Apache Fluss keeps streams as schema-aware columnar log tables, so feeding one into Apache Iggy so far meant writing a bespoke client. This connector reads a Fluss log table through fluss-rs 1.0 and publishes each row as JSON, tracking offsets per bucket through the runtime state API so a restart resumes where the previous run stopped. Buckets already present in the restored state keep their offset, which keeps a widened bucket count from rewinding buckets that were already consumed. Offsets are staged with each batch and committed only when the runtime acknowledges it. The scanner moves past records as soon as it returns them, so a rejected batch, or one that cannot be built, rewinds it to the committed offsets and those rows are read again rather than skipped. The start offsets ride the first poll even when it has no rows. Otherwise a restart before the first row would resolve `latest` again and skip whatever was written in between. Primary-key tables stream a changelog whose change types need their own mapping onto messages, and arrow_ipc payloads need the batch scanner with its own offset-tracking path, so both are rejected at startup instead of being silently downgraded. Column projection is pushed down to the server when `columns` is set. Temporal values are formatted with every fractional digit the column holds, the way the PostgreSQL source formats them. A number of milliseconds would drop the microseconds of the default TIMESTAMP(6). Settings that would otherwise fail silently are rejected at startup: a zero batch_size, which the client accepts and which makes every poll come back empty, a negative starting offset, and only one of the two SASL credentials. Signed-off-by: seokjin0414 --- core/connectors/README.md | 1 + .../connectors/fluss_source.toml | 45 + core/connectors/sources/README.md | 1 + .../sources/fluss_source/Cargo.toml | 56 + .../connectors/sources/fluss_source/README.md | 87 ++ .../sources/fluss_source/config.toml | 48 + .../sources/fluss_source/src/lib.rs | 1037 +++++++++++++++++ .../sources/fluss_source/src/mapping.rs | 355 ++++++ 8 files changed, 1630 insertions(+) create mode 100644 core/connectors/runtime/example_config/connectors/fluss_source.toml create mode 100644 core/connectors/sources/fluss_source/Cargo.toml create mode 100644 core/connectors/sources/fluss_source/README.md create mode 100644 core/connectors/sources/fluss_source/config.toml create mode 100644 core/connectors/sources/fluss_source/src/lib.rs create mode 100644 core/connectors/sources/fluss_source/src/mapping.rs diff --git a/core/connectors/README.md b/core/connectors/README.md index b334009278..600e76a721 100644 --- a/core/connectors/README.md +++ b/core/connectors/README.md @@ -122,6 +122,7 @@ Please refer to the **[Source documentation](https://github.com/apache/iggy/tree ### Available Sources - **Elasticsearch Source** - polls documents from Elasticsearch indices +- **Fluss Source** - reads rows from Apache Fluss log tables with per-bucket offset tracking - **PostgreSQL Source** - reads rows from PostgreSQL tables with multiple consumption strategies (delete after read, mark as processed, timestamp tracking) - **Random Source** - generates random test messages (useful for testing/development) diff --git a/core/connectors/runtime/example_config/connectors/fluss_source.toml b/core/connectors/runtime/example_config/connectors/fluss_source.toml new file mode 100644 index 0000000000..9af4a7c387 --- /dev/null +++ b/core/connectors/runtime/example_config/connectors/fluss_source.toml @@ -0,0 +1,45 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +type = "source" +key = "fluss" +enabled = true +version = 0 +name = "Apache Fluss source" +path = "/target/release/libiggy_connector_fluss_source" +plugin_config_format = "toml" +verbose = false + +[[streams]] +stream = "events" +topic = "fluss_events" +schema = "json" +batch_length = 100 +linger_time = "5ms" + +[plugin_config] +bootstrap_servers = "localhost:9123" +database = "mydb" +table = "events" +table_type = "log" +starting_offset = "earliest" +poll_interval = "1s" +poll_timeout = "5s" +batch_size = 500 +payload_format = "json" +include_metadata = false +verbose_logging = false diff --git a/core/connectors/sources/README.md b/core/connectors/sources/README.md index e63f8cb17f..540e07a4a8 100644 --- a/core/connectors/sources/README.md +++ b/core/connectors/sources/README.md @@ -9,6 +9,7 @@ Source connectors are responsible for ingesting data from external sources into | Source | Description | | ------ | ----------- | | **elasticsearch_source** | Polls documents from Elasticsearch indices with timestamp-based tracking | +| **fluss_source** | Reads rows from Apache Fluss log tables with per-bucket offset tracking and column projection | | **http_source** | Webhook gateway: an embedded HTTP server shared by every instance, with per-endpoint bearer/HMAC auth and a management API for endpoints registered at runtime | | **influxdb_source** | Polls InfluxDB with cursor-based timestamp tracking; supports V2 (Flux, annotated CSV) and V3 (SQL, JSONL) | | **postgres_source** | Reads rows from PostgreSQL tables with multiple strategies: delete after read, mark as processed, or timestamp tracking | diff --git a/core/connectors/sources/fluss_source/Cargo.toml b/core/connectors/sources/fluss_source/Cargo.toml new file mode 100644 index 0000000000..eb08c0189a --- /dev/null +++ b/core/connectors/sources/fluss_source/Cargo.toml @@ -0,0 +1,56 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[package] +name = "iggy_connector_fluss_source" +version = "0.5.0" +description = "Iggy Apache Fluss source connector for producing messages from Fluss log tables" +edition = "2024" +license = "Apache-2.0" +keywords = ["iggy", "messaging", "streaming", "fluss", "source"] +categories = ["command-line-utilities", "database", "network-programming"] +homepage = "https://iggy.apache.org" +documentation = "https://iggy.apache.org/docs" +repository = "https://github.com/apache/iggy" +readme = "../../README.md" +publish = false + +[package.metadata.cargo-machete] +ignored = ["dashmap"] + +[lib] +crate-type = ["cdylib", "lib"] + +[dependencies] +async-trait = { workspace = true } +base64 = { workspace = true } +chrono = { workspace = true } +dashmap = { workspace = true } +fluss-rs = { workspace = true } +humantime = { workspace = true } +iggy_common = { workspace = true } +iggy_connector_sdk = { workspace = true } +rmp-serde = { workspace = true } +secrecy = { workspace = true } +serde = { workspace = true } +serde_json = { workspace = true } +simd-json = { workspace = true } +tokio = { workspace = true } +tracing = { workspace = true } + +[lints] +workspace = true diff --git a/core/connectors/sources/fluss_source/README.md b/core/connectors/sources/fluss_source/README.md new file mode 100644 index 0000000000..8ff5fa601f --- /dev/null +++ b/core/connectors/sources/fluss_source/README.md @@ -0,0 +1,87 @@ +# Apache Fluss source connector + +Reads rows from an [Apache Fluss](https://fluss.apache.org/) log table and publishes them to an Apache Iggy stream as JSON. + +Each Fluss row becomes one Apache Iggy message. Offsets are tracked per bucket and persisted through the runtime state API, so a restart resumes where the previous run stopped. Offsets only advance once the runtime acknowledges a delivered batch; a rejected batch rewinds the scanner to the last acknowledged offsets and is read again. The start position is persisted with the first poll even when it returns no rows, so a restart does not resolve `latest` again past rows written in between. + +## Configuration + +| Field | Required | Default | Description | +| ----- | -------- | ------- | ----------- | +| `bootstrap_servers` | yes | | Coordinator server address, for example `localhost:9123`. | +| `database` | yes | | Apache Fluss database name. | +| `table` | yes | | Apache Fluss table name. | +| `table_type` | no | `log` | Only `log` is accepted. See [Limitations](#limitations). | +| `starting_offset` | no | `earliest` | `earliest`, `latest` (each bucket's tail, resolved at startup), or an explicit non-negative offset. Applies only to buckets absent from the persisted state. | +| `columns` | no | all columns | Column projection pushed down to the server. | +| `poll_interval` | no | `1s` | Delay before each poll. | +| `poll_timeout` | no | `5s` | How long a single server poll waits for records. | +| `batch_size` | no | client default | Maximum records returned per poll (`scanner.log.max-poll-records`). Must be greater than 0. | +| `payload_format` | no | `json` | Only `json` is accepted. See [Limitations](#limitations). | +| `include_metadata` | no | `false` | Adds `_fluss_bucket`, `_fluss_offset` and `_fluss_timestamp` to each JSON object. | +| `sasl_username` | no | | Enables SASL/PLAIN together with `sasl_password`. Set both or neither. | +| `sasl_password` | no | | Stored as a secret and redacted from logs and the `/stats` endpoint. | +| `verbose_logging` | no | `false` | Logs per-batch counts at info instead of debug. | + +## Example + +```toml +type = "source" +key = "fluss" +enabled = true +version = 0 +name = "Apache Fluss source" +path = "libiggy_connector_fluss_source" + +[[streams]] +stream = "fluss_events" +topic = "events" +schema = "json" +batch_length = 100 + +[plugin_config] +bootstrap_servers = "localhost:9123" +database = "mydb" +table = "events" +poll_interval = "1s" +batch_size = 500 +``` + +Bring up a local cluster with the [official Docker compose recipe](https://fluss.apache.org/docs/install-deploy/deploying-with-docker/) (ZooKeeper, one coordinator server, one tablet server), create the stream and topic with the Apache Iggy CLI, then start the connectors runtime. + +## Type mapping + +| Apache Fluss type | JSON | +| ----------------- | ---- | +| `BOOLEAN` | boolean | +| `TINYINT`, `SMALLINT`, `INT`, `BIGINT` | number | +| `FLOAT`, `DOUBLE` | number, `null` when not finite | +| `CHAR`, `STRING` | string | +| `DECIMAL` | string, to keep the full precision | +| `DATE` | string, `2024-02-29` | +| `TIME` | string, `12:34:56.789` | +| `TIMESTAMP` | string without a timezone, `2023-11-14 22:13:20.123456` | +| `TIMESTAMP_LTZ` | string in UTC, RFC 3339, `2023-11-14T22:13:20.123456+00:00` | +| `BINARY`, `BYTES` | base64 string | +| `ARRAY`, `MAP`, `ROW` | not supported, rejected at startup | + +Temporal values keep every fractional digit the column holds, so the default `TIMESTAMP(6)` keeps its microseconds. They are formatted the same way the PostgreSQL source formats them: `TIMESTAMP` carries no timezone and is written without one, while `TIMESTAMP_LTZ` is an instant and is written in UTC. + +Rows are decoded with the table schema read when the connector starts. A row written before a column was added carries `null` for it, and a column added after startup stays out of the output until the connector restarts. + +Every message carries an `id` derived from its bucket and offset, so a consumer can spot a record replayed after an at-least-once redelivery (the Apache Iggy server does not deduplicate on it), and an `origin_timestamp` taken from the Fluss record timestamp. + +## Limitations + +These are limits of this connector, not of the `fluss-rs` client. + +- **Primary-key tables are not supported yet.** `fluss-rs` 1.0 reads their changelog, but mapping its insert, update and delete records onto messages is left for a follow-up, so `table_type = "primary_key"` is rejected at startup. +- **`payload_format = "arrow_ipc"` is not implemented yet.** The client does expose an Arrow `RecordBatch` scanner, but it uses a different offset-tracking path, so it is left for a follow-up. +- **Partitioned tables are not supported.** They are detected and rejected at startup. + +## Build and test + +```bash +cargo build --release -p iggy_connector_fluss_source +cargo test -p iggy_connector_fluss_source +``` diff --git a/core/connectors/sources/fluss_source/config.toml b/core/connectors/sources/fluss_source/config.toml new file mode 100644 index 0000000000..499b476834 --- /dev/null +++ b/core/connectors/sources/fluss_source/config.toml @@ -0,0 +1,48 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +type = "source" +key = "fluss" +enabled = true +version = 0 +name = "Apache Fluss source" +path = "../../target/release/libiggy_connector_fluss_source" +verbose = false + +[[streams]] +stream = "fluss_events" +topic = "events" +schema = "json" +batch_length = 100 + +[plugin_config] +bootstrap_servers = "localhost:9123" +database = "mydb" +table = "events" +table_type = "log" +starting_offset = "earliest" +poll_interval = "1s" +poll_timeout = "5s" +batch_size = 500 +payload_format = "json" +include_metadata = false +verbose_logging = false +# Read a subset of columns. The projection is pushed down to Apache Fluss. +# columns = ["id", "payload"] +# SASL/PLAIN credentials, when the cluster requires them. +# sasl_username = "user" +# sasl_password = "secret" diff --git a/core/connectors/sources/fluss_source/src/lib.rs b/core/connectors/sources/fluss_source/src/lib.rs new file mode 100644 index 0000000000..db47946dc8 --- /dev/null +++ b/core/connectors/sources/fluss_source/src/lib.rs @@ -0,0 +1,1037 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +mod mapping; + +use async_trait::async_trait; +use fluss::client::{EARLIEST_OFFSET, FlussConnection, LogScanner}; +use fluss::config::Config; +use fluss::metadata::{DataField, TablePath}; +use fluss::record::ScanRecords; +use fluss::rpc::message::OffsetSpec; +use iggy_connector_sdk::retry::{RetryPolicy, retry_async}; +use iggy_connector_sdk::{ + ConnectorState, Error, ProducedMessage, ProducedMessages, Schema, Source, + source::SourceBatchResult, source_connector, +}; +use secrecy::{ExposeSecret, SecretString}; +use serde::{Deserialize, Serialize}; +use serde_json::Value; +use std::collections::HashMap; +use std::str::FromStr; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::Duration; +use tokio::sync::Mutex; +use tokio::time::sleep; +use tracing::{debug, info, warn}; + +source_connector!(FlussSource); + +const CONNECTOR_NAME: &str = "Apache Fluss source"; +const DEFAULT_POLL_INTERVAL: Duration = Duration::from_secs(1); +const DEFAULT_POLL_TIMEOUT: Duration = Duration::from_secs(5); +const LOG_TABLE_TYPE: &str = "log"; +const JSON_PAYLOAD_FORMAT: &str = "json"; +const METADATA_BUCKET: &str = "_fluss_bucket"; +const METADATA_OFFSET: &str = "_fluss_offset"; +const METADATA_TIMESTAMP: &str = "_fluss_timestamp"; +const NANOS_PER_MILLI: u64 = 1_000_000; +/// A rewind is local scanner bookkeeping unless the client has to refresh the table's +/// metadata first. An error from `on_batch_result` stops the source, so that refresh gets a +/// few attempts before the failure is reported. +const REWIND_RETRY: RetryPolicy = RetryPolicy { + max_attempts: 3, + base_delay: Duration::from_millis(200), + max_delay: Duration::from_secs(1), +}; + +#[derive(Debug, Serialize, Deserialize)] +pub struct FlussSourceConfig { + pub bootstrap_servers: String, + pub database: String, + pub table: String, + /// Only `log` is accepted today. A primary-key table streams a changelog whose change + /// types need their own mapping onto messages, so the value is validated rather than + /// silently ignored. + pub table_type: Option, + /// `earliest` (default), `latest`, or an explicit numeric offset applied to every bucket. + pub starting_offset: Option, + /// Column projection pushed down to the server. Omit to read every column. + pub columns: Option>, + pub poll_interval: Option, + pub poll_timeout: Option, + pub batch_size: Option, + /// Only `json` is accepted today. `arrow_ipc` needs the batch scanner and a different + /// offset-tracking path, so it is rejected rather than quietly downgraded. + pub payload_format: Option, + pub include_metadata: Option, + pub sasl_username: Option, + #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")] + pub sasl_password: Option, + pub verbose_logging: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize, Default)] +struct State { + /// Next offset to read per bucket. Absent buckets fall back to the configured start. + bucket_offsets: HashMap, + messages_produced: u64, +} + +#[derive(Debug, Clone, Copy)] +enum StartingOffset { + Earliest, + Latest, + Explicit(i64), +} + +impl FromStr for StartingOffset { + type Err = Error; + + fn from_str(value: &str) -> Result { + match value { + "earliest" => Ok(StartingOffset::Earliest), + "latest" => Ok(StartingOffset::Latest), + other => match other.parse::() { + Ok(offset) if offset >= 0 => Ok(StartingOffset::Explicit(offset)), + _ => Err(Error::InitError(format!( + "invalid starting_offset '{other}' for {CONNECTOR_NAME}, expected 'earliest', \ + 'latest' or a non-negative offset" + ))), + }, + } + } +} + +pub struct FlussSource { + id: u32, + config: FlussSourceConfig, + table_path: TablePath, + poll_interval: Duration, + poll_timeout: Duration, + include_metadata: bool, + verbose_logging: bool, + connection: Option, + scanner: Option, + fields: Vec, + state: Mutex, + pending_state: Mutex>, + /// Set when `open()` resolved start offsets that are not on disk yet. The next poll carries + /// them through the ACK handshake even without rows, so a restart before the first row does + /// not resolve `latest` again and skip what was written in between. + start_offsets_unsaved: AtomicBool, +} + +/// `FlussConnection` and `LogScanner` do not implement `Debug`, so the derive is replaced by +/// a hand-written one that reports connection presence instead of client internals. +impl std::fmt::Debug for FlussSource { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter + .debug_struct("FlussSource") + .field("id", &self.id) + .field("table_path", &self.table_path) + .field("poll_interval", &self.poll_interval) + .field("poll_timeout", &self.poll_timeout) + .field("include_metadata", &self.include_metadata) + .field("columns", &self.fields.len()) + .field("opened", &self.scanner.is_some()) + .finish_non_exhaustive() + } +} + +impl FlussSource { + pub fn new(id: u32, config: FlussSourceConfig, state: Option) -> Self { + let poll_interval = parse_duration( + config.poll_interval.as_deref(), + DEFAULT_POLL_INTERVAL, + "poll_interval", + id, + ); + let poll_timeout = parse_duration( + config.poll_timeout.as_deref(), + DEFAULT_POLL_TIMEOUT, + "poll_timeout", + id, + ); + let include_metadata = config.include_metadata.unwrap_or(false); + let verbose_logging = config.verbose_logging.unwrap_or(false); + let table_path = TablePath::new(config.database.clone(), config.table.clone()); + + let restored_state = state + .and_then(|state| state.deserialize::(CONNECTOR_NAME, id)) + .inspect(|state| { + info!( + "Restored state for {CONNECTOR_NAME} connector with ID: {id}. \ + Buckets tracked: {}, messages produced: {}", + state.bucket_offsets.len(), + state.messages_produced + ); + }); + + FlussSource { + id, + config, + table_path, + poll_interval, + poll_timeout, + include_metadata, + verbose_logging, + connection: None, + scanner: None, + fields: Vec::new(), + state: Mutex::new(restored_state.unwrap_or_default()), + pending_state: Mutex::new(None), + start_offsets_unsaved: AtomicBool::new(false), + } + } + + fn serialize_state(&self, state: &State) -> Option { + ConnectorState::serialize(state, CONNECTOR_NAME, self.id) + } + + fn client_config(&self) -> Config { + let mut config = Config { + bootstrap_servers: self.config.bootstrap_servers.clone(), + ..Config::default() + }; + if let Some(batch_size) = self.config.batch_size { + config.scanner_log_max_poll_records = batch_size as usize; + } + if let (Some(username), Some(password)) = + (&self.config.sasl_username, &self.config.sasl_password) + { + config.security_protocol = "sasl".to_owned(); + config.security_sasl_mechanism = "PLAIN".to_owned(); + config.security_sasl_username = username.clone(); + config.security_sasl_password = password.expose_secret().to_owned(); + } + config + } + + fn validate_config(&self) -> Result { + let table_type = self.config.table_type.as_deref().unwrap_or(LOG_TABLE_TYPE); + if table_type != LOG_TABLE_TYPE { + return Err(Error::InitError(format!( + "{CONNECTOR_NAME} supports only table_type '{LOG_TABLE_TYPE}', got '{table_type}'. \ + Primary-key changelog tables are not supported yet" + ))); + } + + let payload_format = self + .config + .payload_format + .as_deref() + .unwrap_or(JSON_PAYLOAD_FORMAT); + if payload_format != JSON_PAYLOAD_FORMAT { + return Err(Error::InitError(format!( + "{CONNECTOR_NAME} supports only payload_format '{JSON_PAYLOAD_FORMAT}', got '{payload_format}'" + ))); + } + + // The client accepts a zero limit, which makes every poll come back empty. + if self.config.batch_size == Some(0) { + return Err(Error::InitError(format!( + "batch_size for {CONNECTOR_NAME} must be greater than 0" + ))); + } + // One without the other would connect without SASL and only fail at the server. + if self.config.sasl_username.is_some() != self.config.sasl_password.is_some() { + return Err(Error::InitError(format!( + "sasl_username and sasl_password for {CONNECTOR_NAME} must be set together" + ))); + } + + self.config + .starting_offset + .as_deref() + .unwrap_or("earliest") + .parse() + } + + /// Buckets already present in the restored state keep their offset. Everything else + /// starts at the given default, so a widened bucket count does not rewind buckets + /// that were already consumed. + fn resolve_start_offsets( + bucket_count: i32, + start_offset: i64, + tracked: &HashMap, + ) -> HashMap { + let mut offsets = tracked.clone(); + for bucket in 0..bucket_count { + offsets.entry(bucket).or_insert(start_offset); + } + offsets + } + + async fn build_batch( + &self, + records: &ScanRecords, + ) -> Result<(Vec, Option), Error> { + let mut messages = Vec::with_capacity(records.count()); + let mut polled_offsets: HashMap = HashMap::new(); + for (bucket, bucket_records) in records.records_by_buckets() { + let bucket_id = bucket.bucket_id(); + for record in bucket_records { + let row = mapping::row_to_json(record.row(), &self.fields)?; + messages.push(self.build_message( + bucket_id, + record.offset(), + record.timestamp(), + row, + )?); + polled_offsets.insert(bucket_id, record.offset() + 1); + } + } + + let state = self + .stage_batch_state(polled_offsets, messages.len()) + .await?; + Ok((messages, state)) + } + + fn build_message( + &self, + bucket: i32, + offset: i64, + timestamp_millis: i64, + mut record: serde_json::Map, + ) -> Result { + if self.include_metadata { + record.insert(METADATA_BUCKET.to_owned(), Value::from(bucket)); + record.insert(METADATA_OFFSET.to_owned(), Value::from(offset)); + record.insert(METADATA_TIMESTAMP.to_owned(), Value::from(timestamp_millis)); + } + + let payload = simd_json::to_vec(&Value::Object(record)).map_err(|error| { + Error::Serialization(format!( + "failed to serialize Apache Fluss row at bucket {bucket}, offset {offset}: {error}" + )) + })?; + + Ok(ProducedMessage { + id: Some(message_id(bucket, offset)), + headers: None, + checksum: None, + timestamp: None, + origin_timestamp: origin_timestamp_nanos(timestamp_millis), + payload, + }) + } + + /// Stages the state a polled batch would leave behind until the runtime reports its + /// result. A batch without rows changes no offset, so it carries no state unless the start + /// offsets from `open()` still have to reach disk. + async fn stage_batch_state( + &self, + polled_offsets: HashMap, + produced: usize, + ) -> Result, Error> { + if produced == 0 && !self.start_offsets_unsaved.load(Ordering::Acquire) { + return Ok(None); + } + + let mut candidate = self.state.lock().await.clone(); + candidate.bucket_offsets.extend(polled_offsets); + candidate.messages_produced += produced as u64; + let persisted = self.serialize_state(&candidate).ok_or_else(|| { + Error::Serialization(format!( + "failed to serialize state for {CONNECTOR_NAME} connector with ID: {}", + self.id + )) + })?; + *self.pending_state.lock().await = Some(candidate); + Ok(Some(persisted)) + } + + /// The scanner moves past records as soon as it returns them, so a batch that is not + /// acknowledged is only read again once the scanner points back at the committed offsets. + /// A fetch still buffered from the old position is dropped by the scanner's own + /// expected-offset check. + async fn rewind(&self, scanner: &LogScanner) -> Result<(), Error> { + let committed = { self.state.lock().await.bucket_offsets.clone() }; + let context = format!("{CONNECTOR_NAME} connector with ID: {} rewind", self.id); + retry_async( + REWIND_RETRY, + &context, + fluss::error::Error::is_retriable, + || scanner.subscribe_buckets(&committed), + ) + .await + .map_err(|failure| { + Error::Connection(format!( + "failed to rewind the Apache Fluss scanner to the last acknowledged offsets: \ + {failure}" + )) + }) + } +} + +#[async_trait] +impl Source for FlussSource { + async fn open(&mut self) -> Result<(), Error> { + let start = self.validate_config()?; + + let connection = FlussConnection::new(self.client_config()) + .await + .map_err(connection_error)?; + + // The scanner aligns every row to the schema it was created with, so the columns used to + // decode rows come from the same table snapshot rather than from a second lookup that a + // schema change could land between. + let (scanner, fields, bucket_count) = { + let table = connection + .get_table(&self.table_path) + .await + .map_err(connection_error)?; + let table_info = table.get_table_info(); + + if table_info.has_primary_key() { + return Err(Error::InitError(format!( + "table '{}' is a primary-key table. {CONNECTOR_NAME} supports log tables only", + self.table_path + ))); + } + if table_info.is_partitioned() { + return Err(Error::InitError(format!( + "table '{}' is partitioned, which {CONNECTOR_NAME} does not support yet", + self.table_path + ))); + } + + let row_type = match self.config.columns.as_ref() { + Some(columns) => table_info + .get_row_type() + .project_with_field_names(columns) + .map_err(|error| { + Error::InitError(format!( + "invalid columns projection {columns:?} for table '{}': {error}", + self.table_path + )) + })?, + None => table_info.get_row_type().clone(), + }; + mapping::ensure_supported_types(row_type.fields())?; + + let scan = match self.config.columns.as_ref() { + Some(columns) => { + let names: Vec<&str> = columns.iter().map(String::as_str).collect(); + table.new_scan().project_by_name(&names).map_err(|error| { + Error::InitError(format!("failed to project columns: {error}")) + })? + } + None => table.new_scan(), + }; + let scanner = scan.create_log_scanner().map_err(|error| { + Error::InitError(format!("failed to create log scanner: {error}")) + })?; + ( + scanner, + row_type.fields().clone(), + table_info.get_num_buckets(), + ) + }; + + let restored = { self.state.lock().await.bucket_offsets.clone() }; + let offsets = match start { + StartingOffset::Earliest => { + Self::resolve_start_offsets(bucket_count, EARLIEST_OFFSET, &restored) + } + StartingOffset::Explicit(offset) => { + Self::resolve_start_offsets(bucket_count, offset, &restored) + } + StartingOffset::Latest => { + let missing: Vec = (0..bucket_count) + .filter(|bucket| !restored.contains_key(bucket)) + .collect(); + let mut offsets = restored.clone(); + if !missing.is_empty() { + let admin = connection.get_admin().map_err(connection_error)?; + let tails = admin + .list_offsets(&self.table_path, &missing, OffsetSpec::Latest) + .await + .map_err(connection_error)?; + for bucket in missing { + let tail = tails.get(&bucket).copied().ok_or_else(|| { + Error::InitError(format!( + "Apache Fluss returned no latest offset for bucket {bucket} of table '{}'", + self.table_path + )) + })?; + offsets.insert(bucket, tail); + } + } + offsets + } + }; + scanner + .subscribe_buckets(&offsets) + .await + .map_err(connection_error)?; + + self.start_offsets_unsaved + .store(offsets != restored, Ordering::Release); + { + let mut state = self.state.lock().await; + state.bucket_offsets = offsets; + } + + self.fields = fields; + self.connection = Some(connection); + self.scanner = Some(scanner); + + info!( + "Opened {CONNECTOR_NAME} connector with ID: {}, table: {}, buckets: {bucket_count}, \ + columns: {}, poll interval: {:?}", + self.id, + self.table_path, + self.fields.len(), + self.poll_interval + ); + Ok(()) + } + + async fn poll(&self) -> Result { + sleep(self.poll_interval).await; + + let Some(scanner) = self.scanner.as_ref() else { + return Err(Error::InitError(format!( + "{CONNECTOR_NAME} connector with ID: {} polled before it was opened", + self.id + ))); + }; + + let records = scanner + .poll(self.poll_timeout) + .await + .map_err(|error| Error::Connection(format!("failed to poll Apache Fluss: {error}")))?; + + let (messages, state) = match self.build_batch(&records).await { + Ok(batch) => batch, + Err(error) => { + // The scanner is already past these records, so they are read again on the next + // poll rather than skipped along with the error. + warn!( + "Rewinding {CONNECTOR_NAME} connector with ID: {} after a batch it could not \ + build: {error}", + self.id + ); + self.rewind(scanner).await?; + return Err(error); + } + }; + + let produced = messages.len(); + if produced > 0 { + if self.verbose_logging { + info!( + "{CONNECTOR_NAME} connector with ID: {} produced {produced} messages from table: {}", + self.id, self.table_path + ); + } else { + debug!( + "{CONNECTOR_NAME} connector with ID: {} produced {produced} messages from table: {}", + self.id, self.table_path + ); + } + } + + Ok(ProducedMessages { + schema: Schema::Json, + messages, + state, + }) + } + + async fn on_batch_result(&self, result: SourceBatchResult) -> Result<(), Error> { + let Some(candidate_state) = self.pending_state.lock().await.take() else { + return Ok(()); + }; + match result { + SourceBatchResult::Ack => { + *self.state.lock().await = candidate_state; + self.start_offsets_unsaved.store(false, Ordering::Release); + Ok(()) + } + SourceBatchResult::Nack => { + let Some(scanner) = self.scanner.as_ref() else { + return Ok(()); + }; + warn!( + "Rewinding {CONNECTOR_NAME} connector with ID: {} to the last acknowledged \ + offsets after a rejected batch", + self.id + ); + self.rewind(scanner).await + } + } + } + + async fn close(&mut self) -> Result<(), Error> { + // `FlussConnection::close` only drains the writer client, which a source never creates, + // so dropping the scanner and the connection is what releases them. + self.scanner = None; + self.connection = None; + let state = self.state.lock().await; + info!( + "Closed {CONNECTOR_NAME} connector with ID: {}, total messages produced: {}", + self.id, state.messages_produced + ); + Ok(()) + } +} + +/// Buckets are per-table and offsets are per-bucket, so the pair is unique within a table and +/// stable across restarts. A consumer can use it to spot a record replayed after an +/// at-least-once redelivery. The Apache Iggy server does not dedupe on it. +fn message_id(bucket: i32, offset: i64) -> u128 { + ((bucket as u32 as u128) << 64) | (offset as u64 as u128) +} + +fn origin_timestamp_nanos(timestamp_millis: i64) -> Option { + u64::try_from(timestamp_millis) + .ok() + .and_then(|millis| millis.checked_mul(NANOS_PER_MILLI)) +} + +/// `new()` cannot fail (the FFI macro fixes its signature), so an unparsable duration falls +/// back to the default. It is logged at warn so a typo does not silently change the cadence. +fn parse_duration(value: Option<&str>, default: Duration, field: &str, id: u32) -> Duration { + let Some(raw) = value else { + return default; + }; + match humantime::Duration::from_str(raw) { + Ok(duration) => *duration, + Err(error) => { + warn!( + "Invalid {field} '{raw}' for {CONNECTOR_NAME} connector with ID: {id}, \ + falling back to {default:?}. {error}" + ); + default + } + } +} + +fn connection_error(error: fluss::error::Error) -> Error { + Error::Connection(format!("Apache Fluss client failure: {error}")) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn test_config() -> FlussSourceConfig { + FlussSourceConfig { + bootstrap_servers: "localhost:9123".to_owned(), + database: "analytics".to_owned(), + table: "events".to_owned(), + table_type: None, + starting_offset: None, + columns: None, + poll_interval: Some("100ms".to_owned()), + poll_timeout: Some("1s".to_owned()), + batch_size: Some(500), + payload_format: None, + include_metadata: None, + sasl_username: None, + sasl_password: None, + verbose_logging: None, + } + } + + fn state_with(offsets: &[(i32, i64)], produced: u64) -> State { + State { + bucket_offsets: offsets.iter().copied().collect(), + messages_produced: produced, + } + } + + #[test] + fn given_persisted_state_should_restore_bucket_offsets() { + let serialized = rmp_serde::to_vec(&state_with(&[(0, 42), (1, 7)], 500)) + .expect("Failed to serialize state"); + + let source = FlussSource::new(1, test_config(), Some(ConnectorState(serialized))); + + let runtime = tokio::runtime::Runtime::new().expect("Failed to build runtime"); + runtime.block_on(async { + let restored = source.state.lock().await; + assert_eq!(restored.messages_produced, 500); + assert_eq!(restored.bucket_offsets.get(&0), Some(&42)); + assert_eq!(restored.bucket_offsets.get(&1), Some(&7)); + }); + } + + #[test] + fn given_no_state_should_start_fresh() { + let source = FlussSource::new(1, test_config(), None); + + let runtime = tokio::runtime::Runtime::new().expect("Failed to build runtime"); + runtime.block_on(async { + let state = source.state.lock().await; + assert_eq!(state.messages_produced, 0); + assert!(state.bucket_offsets.is_empty()); + }); + } + + /// The state is only a per-bucket cursor. An unreadable one falls back to + /// `starting_offset`, which with the default `earliest` re-reads the table: duplicates that + /// at-least-once delivery already allows, not lost rows. + #[test] + fn given_invalid_state_should_start_fresh() { + let invalid = ConnectorState(b"not valid msgpack".to_vec()); + + let source = FlussSource::new(1, test_config(), Some(invalid)); + + let runtime = tokio::runtime::Runtime::new().expect("Failed to build runtime"); + runtime.block_on(async { + let state = source.state.lock().await; + assert_eq!(state.messages_produced, 0); + assert!(state.bucket_offsets.is_empty()); + }); + } + + #[test] + fn state_should_be_serializable_and_deserializable() { + let original = state_with(&[(0, 100), (3, 250)], 1000); + + let serialized = rmp_serde::to_vec(&original).expect("Failed to serialize"); + let deserialized: State = + rmp_serde::from_slice(&serialized).expect("Failed to deserialize"); + + assert_eq!(original.messages_produced, deserialized.messages_produced); + assert_eq!(original.bucket_offsets, deserialized.bucket_offsets); + } + + #[test] + fn given_default_config_should_accept_log_table_and_earliest_offset() { + let source = FlussSource::new(1, test_config(), None); + + let start = source + .validate_config() + .expect("Default config should be valid"); + + assert!(matches!(start, StartingOffset::Earliest)); + } + + #[test] + fn given_primary_key_table_type_should_be_rejected() { + let mut config = test_config(); + config.table_type = Some("primary_key".to_owned()); + let source = FlussSource::new(1, config, None); + + let error = source + .validate_config() + .expect_err("Primary key tables are not supported yet"); + + assert!(matches!(error, Error::InitError(message) if message.contains("primary_key"))); + } + + #[test] + fn given_arrow_ipc_payload_format_should_be_rejected() { + let mut config = test_config(); + config.payload_format = Some("arrow_ipc".to_owned()); + let source = FlussSource::new(1, config, None); + + let error = source + .validate_config() + .expect_err("arrow_ipc is not supported yet"); + + assert!(matches!(error, Error::InitError(message) if message.contains("arrow_ipc"))); + } + + #[test] + fn given_latest_starting_offset_should_be_parsed() { + let mut config = test_config(); + config.starting_offset = Some("latest".to_owned()); + let source = FlussSource::new(1, config, None); + + let start = source.validate_config().expect("latest should be accepted"); + + assert!(matches!(start, StartingOffset::Latest)); + } + + #[test] + fn given_explicit_starting_offset_should_be_parsed() { + let mut config = test_config(); + config.starting_offset = Some("128".to_owned()); + let source = FlussSource::new(1, config, None); + + let start = source + .validate_config() + .expect("Explicit offset should parse"); + + assert!(matches!(start, StartingOffset::Explicit(128))); + } + + #[test] + fn given_unparsable_starting_offset_should_be_rejected() { + let mut config = test_config(); + config.starting_offset = Some("beginning".to_owned()); + let source = FlussSource::new(1, config, None); + + assert!(source.validate_config().is_err()); + } + + #[test] + fn given_negative_starting_offset_should_be_rejected() { + let mut config = test_config(); + config.starting_offset = Some("-1".to_owned()); + let source = FlussSource::new(1, config, None); + + let error = source + .validate_config() + .expect_err("A negative offset should be rejected"); + + assert!(matches!(error, Error::InitError(message) if message.contains("-1"))); + } + + #[test] + fn given_zero_batch_size_should_be_rejected() { + let mut config = test_config(); + config.batch_size = Some(0); + let source = FlussSource::new(1, config, None); + + let error = source + .validate_config() + .expect_err("A zero batch size should be rejected"); + + assert!(matches!(error, Error::InitError(message) if message.contains("batch_size"))); + } + + #[test] + fn given_sasl_username_without_password_should_be_rejected() { + let mut config = test_config(); + config.sasl_username = Some("user".to_owned()); + let source = FlussSource::new(1, config, None); + + let error = source + .validate_config() + .expect_err("Half of the SASL credentials should be rejected"); + + assert!(matches!(error, Error::InitError(message) if message.contains("sasl_password"))); + } + + #[test] + fn given_sasl_password_without_username_should_be_rejected() { + let mut config = test_config(); + config.sasl_password = Some(SecretString::from("secret")); + let source = FlussSource::new(1, config, None); + + assert!(source.validate_config().is_err()); + } + + #[test] + fn given_sasl_credentials_should_build_a_client_config_the_client_accepts() { + let mut config = test_config(); + config.sasl_username = Some("user".to_owned()); + config.sasl_password = Some(SecretString::from("secret")); + let source = FlussSource::new(1, config, None); + + let client_config = source.client_config(); + + assert!(client_config.is_sasl_enabled()); + assert_eq!(client_config.validate_security(), Ok(())); + assert_eq!(client_config.security_sasl_username, "user"); + assert_eq!(client_config.scanner_log_max_poll_records, 500); + } + + #[test] + fn given_no_sasl_credentials_should_build_a_plaintext_client_config() { + let source = FlussSource::new(1, test_config(), None); + + assert!(!source.client_config().is_sasl_enabled()); + } + + #[test] + fn given_tracked_buckets_should_keep_their_offsets_and_fill_the_rest() { + let tracked = HashMap::from([(0, 42)]); + + let offsets = FlussSource::resolve_start_offsets(3, EARLIEST_OFFSET, &tracked); + + assert_eq!(offsets.len(), 3); + assert_eq!(offsets[&0], 42); + assert_eq!(offsets[&1], EARLIEST_OFFSET); + assert_eq!(offsets[&2], EARLIEST_OFFSET); + } + + #[test] + fn given_explicit_start_should_apply_to_untracked_buckets_only() { + let tracked = HashMap::from([(1, 900)]); + + let offsets = FlussSource::resolve_start_offsets(2, 50, &tracked); + + assert_eq!(offsets[&0], 50); + assert_eq!(offsets[&1], 900); + } + + #[test] + fn message_id_should_be_unique_per_bucket_and_offset() { + assert_ne!(message_id(0, 1), message_id(1, 0)); + assert_ne!(message_id(0, 1), message_id(0, 2)); + assert_eq!(message_id(2, 5), message_id(2, 5)); + } + + #[test] + fn origin_timestamp_should_convert_milliseconds_to_nanoseconds() { + assert_eq!( + origin_timestamp_nanos(1_785_655_133_842), + Some(1_785_655_133_842_000_000) + ); + assert_eq!(origin_timestamp_nanos(0), Some(0)); + assert_eq!(origin_timestamp_nanos(-1), None); + } + + #[test] + fn given_invalid_duration_should_fall_back_to_default() { + assert_eq!( + parse_duration(Some("nonsense"), DEFAULT_POLL_INTERVAL, "poll_interval", 1), + DEFAULT_POLL_INTERVAL + ); + assert_eq!( + parse_duration(None, DEFAULT_POLL_TIMEOUT, "poll_timeout", 1), + DEFAULT_POLL_TIMEOUT + ); + assert_eq!( + parse_duration(Some("250ms"), DEFAULT_POLL_INTERVAL, "poll_interval", 1), + Duration::from_millis(250) + ); + } + + #[test] + fn given_ack_when_batch_is_staged_should_commit_candidate_state() { + let source = FlussSource::new(1, test_config(), None); + let runtime = tokio::runtime::Runtime::new().expect("failed to create test runtime"); + runtime.block_on(async { + *source.pending_state.lock().await = Some(state_with(&[(0, 42)], 42)); + + source + .on_batch_result(SourceBatchResult::Ack) + .await + .expect("ACK should be applied"); + + let state = source.state.lock().await; + assert_eq!(state.messages_produced, 42); + assert_eq!(state.bucket_offsets.get(&0), Some(&42)); + assert!(source.pending_state.lock().await.is_none()); + }); + } + + #[test] + fn given_nack_when_batch_is_staged_should_keep_committed_state() { + let source = FlussSource::new(1, test_config(), None); + let runtime = tokio::runtime::Runtime::new().expect("failed to create test runtime"); + runtime.block_on(async { + *source.pending_state.lock().await = Some(state_with(&[(0, 42)], 42)); + + source + .on_batch_result(SourceBatchResult::Nack) + .await + .expect("NACK should be applied"); + + let state = source.state.lock().await; + assert_eq!(state.messages_produced, 0); + assert!(state.bucket_offsets.is_empty()); + assert!(source.pending_state.lock().await.is_none()); + }); + } + + #[test] + fn given_empty_poll_when_start_offsets_are_saved_should_carry_no_state() { + let source = FlussSource::new(1, test_config(), None); + let runtime = tokio::runtime::Runtime::new().expect("failed to create test runtime"); + runtime.block_on(async { + *source.state.lock().await = state_with(&[(0, 42)], 42); + + let state = source + .stage_batch_state(HashMap::new(), 0) + .await + .expect("An empty batch should stage"); + + assert!(state.is_none()); + assert!(source.pending_state.lock().await.is_none()); + }); + } + + #[test] + fn given_empty_poll_when_start_offsets_are_unsaved_should_stage_them() { + let source = FlussSource::new(1, test_config(), None); + let runtime = tokio::runtime::Runtime::new().expect("failed to create test runtime"); + runtime.block_on(async { + *source.state.lock().await = state_with(&[(0, 42)], 0); + source.start_offsets_unsaved.store(true, Ordering::Release); + + let state = source + .stage_batch_state(HashMap::new(), 0) + .await + .expect("An empty batch should stage"); + + let persisted = state + .and_then(|state| state.deserialize::(CONNECTOR_NAME, 1)) + .expect("Unsaved start offsets should ride the empty batch"); + assert_eq!(persisted.bucket_offsets.get(&0), Some(&42)); + assert!(source.pending_state.lock().await.is_some()); + }); + } + + #[test] + fn given_polled_records_should_stage_advanced_offsets_without_committing_them() { + let source = FlussSource::new(1, test_config(), None); + let runtime = tokio::runtime::Runtime::new().expect("failed to create test runtime"); + runtime.block_on(async { + *source.state.lock().await = state_with(&[(0, 42), (1, 7)], 42); + + let state = source + .stage_batch_state(HashMap::from([(0, 45)]), 3) + .await + .expect("A polled batch should stage"); + + let persisted = state + .and_then(|state| state.deserialize::(CONNECTOR_NAME, 1)) + .expect("A batch with rows should carry state"); + assert_eq!(persisted.bucket_offsets.get(&0), Some(&45)); + assert_eq!(persisted.bucket_offsets.get(&1), Some(&7)); + assert_eq!(persisted.messages_produced, 45); + + let committed = source.state.lock().await; + assert_eq!(committed.bucket_offsets.get(&0), Some(&42)); + assert_eq!(committed.messages_produced, 42); + }); + } + + #[test] + fn given_unsaved_start_offsets_should_stay_unsaved_until_a_batch_is_acked() { + let source = FlussSource::new(1, test_config(), None); + let runtime = tokio::runtime::Runtime::new().expect("failed to create test runtime"); + runtime.block_on(async { + source.start_offsets_unsaved.store(true, Ordering::Release); + + *source.pending_state.lock().await = Some(state_with(&[(0, 42)], 0)); + source + .on_batch_result(SourceBatchResult::Nack) + .await + .expect("NACK should be applied"); + assert!(source.start_offsets_unsaved.load(Ordering::Acquire)); + + *source.pending_state.lock().await = Some(state_with(&[(0, 42)], 0)); + source + .on_batch_result(SourceBatchResult::Ack) + .await + .expect("ACK should be applied"); + assert!(!source.start_offsets_unsaved.load(Ordering::Acquire)); + }); + } +} diff --git a/core/connectors/sources/fluss_source/src/mapping.rs b/core/connectors/sources/fluss_source/src/mapping.rs new file mode 100644 index 0000000000..414caa38e1 --- /dev/null +++ b/core/connectors/sources/fluss_source/src/mapping.rs @@ -0,0 +1,355 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use base64::Engine; +use base64::engine::general_purpose::STANDARD as BASE64; +use chrono::{DateTime, NaiveDate, NaiveTime, Utc}; +use fluss::metadata::{DataField, DataType}; +use fluss::row::InternalRow; +use iggy_connector_sdk::Error; +use serde_json::{Map, Number, Value}; + +const MILLIS_PER_SECOND: i64 = 1_000; +const NANOS_PER_MILLI: i64 = 1_000_000; + +/// Temporal values are formatted with their full fractional precision, the way the +/// PostgreSQL source formats them. A number of milliseconds would drop the microseconds of +/// the default `TIMESTAMP(6)`. `TIMESTAMP` carries no timezone and is written without one, +/// while `TIMESTAMP_LTZ` is an instant and is written in UTC. +pub(crate) fn row_to_json( + row: &dyn InternalRow, + fields: &[DataField], +) -> Result, Error> { + let mut object = Map::with_capacity(fields.len()); + for (position, field) in fields.iter().enumerate() { + let value = read_field(row, position, field.data_type())?; + object.insert(field.name().to_owned(), value); + } + Ok(object) +} + +/// Rejects column types with no JSON representation before the first poll, so a table with +/// an unsupported column fails at startup instead of once per batch. +pub(crate) fn ensure_supported_types(fields: &[DataField]) -> Result<(), Error> { + for field in fields { + if !is_supported(field.data_type()) { + return Err(Error::SchemaMismatch(format!( + "column '{}' has type {:?}, which the Apache Fluss source cannot map to JSON", + field.name(), + field.data_type() + ))); + } + } + Ok(()) +} + +fn is_supported(data_type: &DataType) -> bool { + !matches!( + data_type, + DataType::Array(_) | DataType::Map(_) | DataType::Row(_) + ) +} + +fn read_field( + row: &dyn InternalRow, + position: usize, + data_type: &DataType, +) -> Result { + if row.is_null_at(position).map_err(read_error)? { + return Ok(Value::Null); + } + + let value = match data_type { + DataType::Boolean(_) => Value::Bool(row.get_boolean(position).map_err(read_error)?), + DataType::TinyInt(_) => Value::from(row.get_byte(position).map_err(read_error)?), + DataType::SmallInt(_) => Value::from(row.get_short(position).map_err(read_error)?), + DataType::Int(_) => Value::from(row.get_int(position).map_err(read_error)?), + DataType::BigInt(_) => Value::from(row.get_long(position).map_err(read_error)?), + DataType::Float(_) => float_value(f64::from(row.get_float(position).map_err(read_error)?)), + DataType::Double(_) => float_value(row.get_double(position).map_err(read_error)?), + DataType::Char(inner) => Value::String( + row.get_char(position, inner.length() as usize) + .map_err(read_error)? + .to_owned(), + ), + DataType::String(_) => { + Value::String(row.get_string(position).map_err(read_error)?.to_owned()) + } + DataType::Decimal(inner) => { + let decimal = row + .get_decimal(position, inner.precision() as usize, inner.scale() as usize) + .map_err(read_error)?; + Value::String(decimal.to_big_decimal().to_string()) + } + DataType::Date(_) => { + let days = row.get_date(position).map_err(read_error)?.get_inner(); + let date = NaiveDate::from_epoch_days(days).ok_or_else(|| out_of_range(data_type))?; + Value::String(date.to_string()) + } + DataType::Time(_) => { + let millis = row.get_time(position).map_err(read_error)?.get_inner(); + let time = time_of_day(millis).ok_or_else(|| out_of_range(data_type))?; + Value::String(time.to_string()) + } + DataType::Timestamp(inner) => { + let timestamp = row + .get_timestamp_ntz(position, inner.precision()) + .map_err(read_error)?; + let instant = instant( + timestamp.get_millisecond(), + timestamp.get_nano_of_millisecond(), + ) + .ok_or_else(|| out_of_range(data_type))?; + Value::String(instant.naive_utc().to_string()) + } + DataType::TimestampLTz(inner) => { + let timestamp = row + .get_timestamp_ltz(position, inner.precision()) + .map_err(read_error)?; + let instant = instant( + timestamp.get_epoch_millisecond(), + timestamp.get_nano_of_millisecond(), + ) + .ok_or_else(|| out_of_range(data_type))?; + Value::String(instant.to_rfc3339()) + } + DataType::Bytes(_) => { + Value::String(BASE64.encode(row.get_bytes(position).map_err(read_error)?)) + } + DataType::Binary(inner) => Value::String( + BASE64.encode( + row.get_binary(position, inner.length()) + .map_err(read_error)?, + ), + ), + DataType::Array(_) | DataType::Map(_) | DataType::Row(_) => { + return Err(Error::SchemaMismatch(format!( + "nested type {data_type:?} is not supported by the Apache Fluss source" + ))); + } + }; + Ok(value) +} + +/// Fluss keeps a timestamp as epoch milliseconds plus the nanoseconds within that +/// millisecond. Euclidean division keeps a pre-epoch value's fraction positive, so -1 ms is +/// 23:59:59.999 of the previous day rather than an invalid negative fraction. +fn instant(epoch_millis: i64, nano_of_millisecond: i32) -> Option> { + let seconds = epoch_millis.div_euclid(MILLIS_PER_SECOND); + let nanos = epoch_millis.rem_euclid(MILLIS_PER_SECOND) * NANOS_PER_MILLI + + i64::from(nano_of_millisecond); + DateTime::from_timestamp(seconds, u32::try_from(nanos).ok()?) +} + +/// Fluss keeps `TIME` as milliseconds since midnight. +fn time_of_day(millis: i32) -> Option { + let millis = i64::from(millis); + NaiveTime::from_num_seconds_from_midnight_opt( + u32::try_from(millis.div_euclid(MILLIS_PER_SECOND)).ok()?, + u32::try_from(millis.rem_euclid(MILLIS_PER_SECOND) * NANOS_PER_MILLI).ok()?, + ) +} + +/// JSON has no encoding for NaN or infinity, so those collapse to null rather than +/// failing the whole batch over one degenerate float. +fn float_value(value: f64) -> Value { + Number::from_f64(value).map_or(Value::Null, Value::Number) +} + +fn read_error(error: fluss::error::Error) -> Error { + Error::InvalidRecordValue(format!("failed to read Apache Fluss column: {error}")) +} + +fn out_of_range(data_type: &DataType) -> Error { + Error::InvalidRecordValue(format!( + "Apache Fluss {data_type:?} value is outside the range that can be formatted" + )) +} + +#[cfg(test)] +mod tests { + use super::*; + use fluss::metadata::DataTypes; + use fluss::row::{Date, Decimal, GenericRow, Time, TimestampLtz, TimestampNtz}; + + fn field(name: &str, data_type: DataType) -> DataField { + DataField::new(name, data_type, None) + } + + #[test] + fn given_temporal_columns_when_mapped_should_keep_full_precision() { + let fields = vec![ + field("date", DataTypes::date()), + field("time", DataTypes::time_with_precision(3)), + field("timestamp", DataTypes::timestamp()), + field("timestamp_nanos", DataTypes::timestamp_with_precision(9)), + field("timestamp_ltz", DataTypes::timestamp_ltz()), + ]; + let mut row = GenericRow::new(5); + row.set_field(0, Date::new(19_782)); + row.set_field(1, Time::new(45_296_789)); + row.set_field( + 2, + TimestampNtz::from_millis_nanos(1_700_000_000_123, 456_000) + .expect("Failed to build timestamp"), + ); + row.set_field( + 3, + TimestampNtz::from_millis_nanos(1_700_000_000_123, 456_789) + .expect("Failed to build timestamp"), + ); + row.set_field( + 4, + TimestampLtz::from_millis_nanos(1_700_000_000_123, 456_000) + .expect("Failed to build timestamp"), + ); + + let object = row_to_json(&row, &fields).expect("Failed to map row"); + + assert_eq!(object["date"], Value::from("2024-02-29")); + assert_eq!(object["time"], Value::from("12:34:56.789")); + assert_eq!( + object["timestamp"], + Value::from("2023-11-14 22:13:20.123456") + ); + assert_eq!( + object["timestamp_nanos"], + Value::from("2023-11-14 22:13:20.123456789") + ); + assert_eq!( + object["timestamp_ltz"], + Value::from("2023-11-14T22:13:20.123456+00:00") + ); + } + + #[test] + fn given_timestamp_before_the_epoch_when_mapped_should_borrow_from_the_previous_second() { + let fields = vec![field("timestamp", DataTypes::timestamp_with_precision(3))]; + let mut row = GenericRow::new(1); + row.set_field( + 0, + TimestampNtz::from_millis_nanos(-1, 0).expect("Failed to build timestamp"), + ); + + let object = row_to_json(&row, &fields).expect("Failed to map row"); + + assert_eq!(object["timestamp"], Value::from("1969-12-31 23:59:59.999")); + } + + #[test] + fn given_decimal_and_fixed_width_columns_when_mapped_should_produce_strings() { + let fields = vec![ + field("amount", DataTypes::decimal(10, 2)), + field("code", DataTypes::char(5)), + field("digest", DataTypes::binary(3)), + ]; + let mut row = GenericRow::new(3); + row.set_field( + 0, + Decimal::from_unscaled_long(12_345, 10, 2).expect("Failed to build decimal"), + ); + row.set_field(1, "abcde"); + row.set_field(2, [4u8, 5, 6].as_slice()); + + let object = row_to_json(&row, &fields).expect("Failed to map row"); + + assert_eq!(object["amount"], Value::from("123.45")); + assert_eq!(object["code"], Value::from("abcde")); + assert_eq!(object["digest"], Value::from(BASE64.encode([4u8, 5, 6]))); + } + + #[test] + fn given_scalar_columns_when_mapped_should_produce_json_object() { + let fields = vec![ + field("id", DataTypes::int()), + field("name", DataTypes::string()), + field("active", DataTypes::boolean()), + field("ratio", DataTypes::double()), + field("total", DataTypes::bigint()), + ]; + let mut row = GenericRow::new(5); + row.set_field(0, 7i32); + row.set_field(1, "alice"); + row.set_field(2, true); + row.set_field(3, 1.5f64); + row.set_field(4, 90i64); + + let object = row_to_json(&row, &fields).expect("Failed to map row"); + + assert_eq!(object["id"], Value::from(7)); + assert_eq!(object["name"], Value::from("alice")); + assert_eq!(object["active"], Value::from(true)); + assert_eq!(object["ratio"], Value::from(1.5)); + assert_eq!(object["total"], Value::from(90)); + } + + #[test] + fn given_unset_column_when_mapped_should_produce_null() { + let fields = vec![ + field("id", DataTypes::int()), + field("name", DataTypes::string()), + ]; + let mut row = GenericRow::new(2); + row.set_field(0, 1i32); + + let object = row_to_json(&row, &fields).expect("Failed to map row"); + + assert_eq!(object["id"], Value::from(1)); + assert_eq!(object["name"], Value::Null); + } + + #[test] + fn given_binary_column_when_mapped_should_produce_base64() { + let fields = vec![field("blob", DataTypes::bytes())]; + let mut row = GenericRow::new(1); + row.set_field(0, [1u8, 2, 3].as_slice()); + + let object = row_to_json(&row, &fields).expect("Failed to map row"); + + assert_eq!(object["blob"], Value::from(BASE64.encode([1u8, 2, 3]))); + } + + #[test] + fn given_scalar_columns_when_validated_should_be_accepted() { + let fields = vec![ + field("a", DataTypes::string()), + field("b", DataTypes::timestamp()), + field("c", DataTypes::decimal(10, 2)), + ]; + + assert!(ensure_supported_types(&fields).is_ok()); + } + + #[test] + fn given_nested_column_when_validated_should_be_rejected() { + let fields = vec![ + field("id", DataTypes::int()), + field("tags", DataTypes::array(DataTypes::string())), + ]; + + let error = ensure_supported_types(&fields).expect_err("Nested column should be rejected"); + + assert!(matches!(error, Error::SchemaMismatch(message) if message.contains("tags"))); + } + + #[test] + fn given_non_finite_float_should_map_to_null() { + assert_eq!(float_value(f64::NAN), Value::Null); + assert_eq!(float_value(f64::INFINITY), Value::Null); + assert_eq!(float_value(2.5), Value::from(2.5)); + } +} From a76e6992684afe837fafeea7e7cd2083c233fe15 Mon Sep 17 00:00:00 2001 From: seokjin0414 Date: Thu, 24 Sep 2026 23:53:19 +0900 Subject: [PATCH 2/6] test(integration): cover the Apache Fluss source end to end Exercises the whole path rather than the connector in isolation: rows are appended to a real Fluss log table, and the tests assert on what comes back out of the Apache Iggy topic, including the offsets and bucket the connector attached as metadata. Two cases cover the ways a row could be skipped. One kills the Apache Iggy server so a batch is rejected after the scanner has already handed its rows over, then restarts it and expects every row, which only holds if the source rewound the scanner. The other starts from `latest` over a table that already holds rows, restarts the runtime before any new row arrives and expects the rows written while it was down, which only holds if the tail resolved at startup reached disk. Another writes a row holding every supported column type, plus a row of nulls, and checks the JSON each one becomes. Those rows are decoded from Arrow by a real server, the path production reads, where the unit tests build rows in memory. The cluster runs as a single container. The image ships local-cluster.sh, which starts the same embedded ZooKeeper, coordinator server and tablet server, but it rewrites the tablet server's bind port to 0 so it cannot collide with the coordinator, and a random port inside the container cannot be published to the host. Giving the tablet server an explicit second port instead keeps both reachable, and lets both servers advertise localhost, which then resolves the same way inside the container and from the test process. The table is created during fixture setup because the connectors runtime starts before the test body and the connector resolves the table schema while opening. Writes retry because bucket leadership is assigned shortly after the tablet server registers, so the first attempts can still be rejected. Signed-off-by: seokjin0414 --- core/integration/Cargo.toml | 1 + .../connectors/fixtures/fluss/container.rs | 139 +++++++ .../tests/connectors/fixtures/fluss/mod.rs | 24 ++ .../tests/connectors/fixtures/fluss/source.rs | 383 ++++++++++++++++++ .../tests/connectors/fixtures/mod.rs | 5 + .../tests/connectors/fluss/fluss_source.rs | 362 +++++++++++++++++ .../integration/tests/connectors/fluss/mod.rs | 25 ++ .../tests/connectors/fluss/source.toml | 20 + core/integration/tests/connectors/mod.rs | 1 + 9 files changed, 960 insertions(+) create mode 100644 core/integration/tests/connectors/fixtures/fluss/container.rs create mode 100644 core/integration/tests/connectors/fixtures/fluss/mod.rs create mode 100644 core/integration/tests/connectors/fixtures/fluss/source.rs create mode 100644 core/integration/tests/connectors/fluss/fluss_source.rs create mode 100644 core/integration/tests/connectors/fluss/mod.rs create mode 100644 core/integration/tests/connectors/fluss/source.toml diff --git a/core/integration/Cargo.toml b/core/integration/Cargo.toml index ba6ff18a41..7761b7f7e6 100644 --- a/core/integration/Cargo.toml +++ b/core/integration/Cargo.toml @@ -51,6 +51,7 @@ ctor = { workspace = true } deltalake = { workspace = true } dtor = { workspace = true } figment = { workspace = true } +fluss-rs = { workspace = true } futures = { workspace = true } harness_derive = { workspace = true } hex = { workspace = true } diff --git a/core/integration/tests/connectors/fixtures/fluss/container.rs b/core/integration/tests/connectors/fixtures/fluss/container.rs new file mode 100644 index 0000000000..4ef35d8bf4 --- /dev/null +++ b/core/integration/tests/connectors/fixtures/fluss/container.rs @@ -0,0 +1,139 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use crate::connectors::fixtures; +use integration::harness::TestBinaryError; +use std::net::TcpListener; +use testcontainers_modules::testcontainers::core::{IntoContainerPort, WaitFor}; +use testcontainers_modules::testcontainers::runners::AsyncRunner; +use testcontainers_modules::testcontainers::{ContainerAsync, GenericImage, ImageExt}; + +const FLUSS_IMAGE: &str = "apache/fluss"; +const FLUSS_VERSION: &str = "1.0.0"; +const READY_MESSAGE: &str = "Registered tablet server 0"; + +pub(super) const ENV_SOURCE_BOOTSTRAP_SERVERS: &str = + "IGGY_CONNECTORS_SOURCE_FLUSS_PLUGIN_CONFIG_BOOTSTRAP_SERVERS"; +pub(super) const ENV_SOURCE_DATABASE: &str = "IGGY_CONNECTORS_SOURCE_FLUSS_PLUGIN_CONFIG_DATABASE"; +pub(super) const ENV_SOURCE_TABLE: &str = "IGGY_CONNECTORS_SOURCE_FLUSS_PLUGIN_CONFIG_TABLE"; +pub(super) const ENV_SOURCE_STARTING_OFFSET: &str = + "IGGY_CONNECTORS_SOURCE_FLUSS_PLUGIN_CONFIG_STARTING_OFFSET"; +pub(super) const ENV_SOURCE_POLL_INTERVAL: &str = + "IGGY_CONNECTORS_SOURCE_FLUSS_PLUGIN_CONFIG_POLL_INTERVAL"; +pub(super) const ENV_SOURCE_INCLUDE_METADATA: &str = + "IGGY_CONNECTORS_SOURCE_FLUSS_PLUGIN_CONFIG_INCLUDE_METADATA"; +pub(super) const ENV_SOURCE_STREAMS_0_STREAM: &str = + "IGGY_CONNECTORS_SOURCE_FLUSS_STREAMS_0_STREAM"; +pub(super) const ENV_SOURCE_STREAMS_0_TOPIC: &str = "IGGY_CONNECTORS_SOURCE_FLUSS_STREAMS_0_TOPIC"; +pub(super) const ENV_SOURCE_STREAMS_0_SCHEMA: &str = + "IGGY_CONNECTORS_SOURCE_FLUSS_STREAMS_0_SCHEMA"; +pub(super) const ENV_SOURCE_PATH: &str = "IGGY_CONNECTORS_SOURCE_FLUSS_PATH"; + +/// A whole Fluss cluster (embedded ZooKeeper, coordinator server, tablet server) inside one +/// container. +/// +/// The image ships `local-cluster.sh`, which starts the same three processes, but it rewrites +/// the tablet server's bind port to 0 so it cannot collide with the coordinator. A random port +/// inside the container cannot be published to the host, so the tablet server is given an +/// explicit second port here instead. +/// +/// Both servers advertise `localhost`, which resolves to the coordinator/tablet pair inside the +/// container and to the published ports from the test process. That only holds while the host +/// and container port numbers match, so free host ports are reserved up front and mapped +/// one-to-one rather than letting Docker assign them. +pub(super) struct FlussContainer { + #[allow(dead_code)] + container: ContainerAsync, + pub(super) bootstrap_servers: String, +} + +impl FlussContainer { + pub(super) async fn start() -> Result { + let coordinator_port = reserve_host_port()?; + let tablet_port = reserve_host_port()?; + + let container = GenericImage::new(FLUSS_IMAGE, FLUSS_VERSION) + .with_wait_for(WaitFor::message_on_stdout(READY_MESSAGE)) + .with_entrypoint("/bin/bash") + .with_container_name(fixtures::unique_container_name("fluss")) + .with_mapped_port(coordinator_port, coordinator_port.tcp()) + .with_mapped_port(tablet_port, tablet_port.tcp()) + .with_env_var("FLUSS_PROPERTIES", server_properties(coordinator_port)) + .with_cmd(["-c", &startup_script(tablet_port)]) + .start() + .await + .map_err(|error| TestBinaryError::FixtureSetup { + fixture_type: "FlussContainer".to_string(), + message: format!("Failed to start container: {error}"), + })?; + + Ok(Self { + container, + bootstrap_servers: format!("localhost:{coordinator_port}"), + }) + } +} + +/// The coordinator settings land in `server.yaml`. The tablet server inherits them and +/// overrides only its listeners on the command line. +fn server_properties(coordinator_port: u16) -> String { + format!( + "zookeeper.address: localhost:2181\n\ + bind.listeners: CLIENT://0.0.0.0:{coordinator_port}\n\ + advertised.listeners: CLIENT://localhost:{coordinator_port}\n\ + internal.listener.name: CLIENT\n\ + default.bucket.number: 1\n\ + default.replication.factor: 1\n\ + data.dir: /tmp/fluss/data\n\ + remote.data.dir: /tmp/fluss/remote-data\n\ + tablet-server.id: 0\n" + ) +} + +/// `/docker-entrypoint.sh true` only runs the image's configuration step, which appends +/// `FLUSS_PROPERTIES` to `server.yaml`. The tablet server runs in the foreground so the +/// container stays alive with it. +fn startup_script(tablet_port: u16) -> String { + format!( + "set -e\n\ + /docker-entrypoint.sh true\n\ + /opt/fluss/bin/fluss-daemon.sh start zookeeper /opt/fluss/conf/zookeeper.properties\n\ + /opt/fluss/bin/coordinator-server.sh start\n\ + exec /opt/fluss/bin/tablet-server.sh start-foreground \ + -Dbind.listeners=CLIENT://0.0.0.0:{tablet_port} \ + -Dadvertised.listeners=CLIENT://localhost:{tablet_port}\n" + ) +} + +/// Binds port 0, reads back what the kernel picked, then releases it. The port is only +/// reserved by convention until the container claims it, which is the same trade every +/// fixture that needs a known port ahead of time makes. +fn reserve_host_port() -> Result { + let listener = + TcpListener::bind("127.0.0.1:0").map_err(|error| TestBinaryError::FixtureSetup { + fixture_type: "FlussContainer".to_string(), + message: format!("Failed to reserve a host port: {error}"), + })?; + let port = listener + .local_addr() + .map_err(|error| TestBinaryError::FixtureSetup { + fixture_type: "FlussContainer".to_string(), + message: format!("Failed to read the reserved host port: {error}"), + })? + .port(); + Ok(port) +} diff --git a/core/integration/tests/connectors/fixtures/fluss/mod.rs b/core/integration/tests/connectors/fixtures/fluss/mod.rs new file mode 100644 index 0000000000..57b33719cb --- /dev/null +++ b/core/integration/tests/connectors/fixtures/fluss/mod.rs @@ -0,0 +1,24 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +mod container; +mod source; + +pub use source::{ + FlussSourceAllTypesFixture, FlussSourceFixture, FlussSourceLatestFixture, + FlussSourceSlowPollFixture, +}; diff --git a/core/integration/tests/connectors/fixtures/fluss/source.rs b/core/integration/tests/connectors/fixtures/fluss/source.rs new file mode 100644 index 0000000000..4da9b98c20 --- /dev/null +++ b/core/integration/tests/connectors/fixtures/fluss/source.rs @@ -0,0 +1,383 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use super::container::{ + ENV_SOURCE_BOOTSTRAP_SERVERS, ENV_SOURCE_DATABASE, ENV_SOURCE_INCLUDE_METADATA, + ENV_SOURCE_PATH, ENV_SOURCE_POLL_INTERVAL, ENV_SOURCE_STARTING_OFFSET, + ENV_SOURCE_STREAMS_0_SCHEMA, ENV_SOURCE_STREAMS_0_STREAM, ENV_SOURCE_STREAMS_0_TOPIC, + ENV_SOURCE_TABLE, FlussContainer, +}; +use async_trait::async_trait; +use fluss::client::FlussConnection; +use fluss::config::Config; +use fluss::metadata::{DataTypes, Schema, TableDescriptor, TablePath}; +use fluss::row::{Date, Decimal, GenericRow, Time, TimestampLtz, TimestampNtz}; +use integration::harness::seeds; +use integration::harness::{TestBinaryError, TestFixture}; +use std::collections::HashMap; +use std::ops::Deref; +use std::time::Duration; +use tokio::time::sleep; + +const DATABASE: &str = "iggy_test"; +const TABLE: &str = "events"; +const POLL_INTERVAL: &str = "100ms"; +const SLOW_POLL_INTERVAL: &str = "5s"; +const EXISTING_ROW_COUNT: usize = 3; +const ALL_TYPES_COLUMN_COUNT: usize = 17; +const READY_ATTEMPTS: usize = 20; +const READY_RETRY_DELAY: Duration = Duration::from_millis(500); + +/// Fluss log table read by the source connector under test. +pub struct FlussSourceFixture { + container: FlussContainer, +} + +impl FlussSourceFixture { + /// Appends one row per payload and flushes, so the rows are readable once this returns. + /// + /// Bucket leadership is assigned shortly after the tablet server registers, so the first + /// writes can still be rejected with `NotLeaderOrFollower`. The client retries internally + /// but gives up before leadership settles, hence the retry here. + pub async fn append_rows(&self, payloads: &[String]) -> Result<(), TestBinaryError> { + let rows: Vec = payloads + .iter() + .enumerate() + .map(|(index, payload)| { + let mut row = GenericRow::new(2); + row.set_field(0, index as i32); + row.set_field(1, payload.as_str()); + row + }) + .collect(); + self.append(&rows).await + } + + /// Starts a cluster and creates the table the connector reads, with the given schema. + async fn start(schema: fn() -> fluss::error::Result) -> Result { + let fixture = Self { + container: FlussContainer::start().await?, + }; + fixture.create_table_when_ready(schema).await?; + Ok(fixture) + } + + async fn append(&self, rows: &[GenericRow<'_>]) -> Result<(), TestBinaryError> { + let mut last_error = None; + for _ in 0..READY_ATTEMPTS { + match self.try_append(rows).await { + Ok(()) => return Ok(()), + Err(error) => last_error = Some(error), + } + sleep(READY_RETRY_DELAY).await; + } + Err(last_error.unwrap_or_else(|| TestBinaryError::FixtureSetup { + fixture_type: "FlussSourceFixture".to_string(), + message: "Failed to append rows".to_string(), + })) + } + + /// Creates the database and an append-only log table. + /// + /// Runs during `setup()`, before the harness starts the connectors runtime, because the + /// source connector resolves the table schema in `open()` and fails initialization when + /// the table is missing. + async fn create_table_when_ready( + &self, + schema: fn() -> fluss::error::Result, + ) -> Result<(), TestBinaryError> { + let mut last_error = None; + for _ in 0..READY_ATTEMPTS { + match self.try_create_table(schema).await { + Ok(()) => return Ok(()), + Err(error) => last_error = Some(error), + } + sleep(READY_RETRY_DELAY).await; + } + Err(last_error.unwrap_or_else(|| TestBinaryError::FixtureSetup { + fixture_type: "FlussSourceFixture".to_string(), + message: "Failed to create table".to_string(), + })) + } + + async fn try_create_table( + &self, + schema: fn() -> fluss::error::Result, + ) -> Result<(), TestBinaryError> { + let connection = self.connect().await?; + let admin = connection.get_admin().map_err(|error| self.error(error))?; + admin + .create_database(DATABASE, None, true) + .await + .map_err(|error| self.error(error))?; + + let schema = schema().map_err(|error| self.error(error))?; + let descriptor = TableDescriptor::builder() + .schema(schema) + .build() + .map_err(|error| self.error(error))?; + admin + .create_table(&Self::table_path(), &descriptor, true) + .await + .map_err(|error| self.error(error))?; + Ok(()) + } + + async fn try_append(&self, rows: &[GenericRow<'_>]) -> Result<(), TestBinaryError> { + let connection = self.connect().await?; + let table = connection + .get_table(&Self::table_path()) + .await + .map_err(|error| self.error(error))?; + let writer = table + .new_append() + .map_err(|error| self.error(error))? + .create_writer() + .map_err(|error| self.error(error))?; + + for row in rows { + writer.append(row).map_err(|error| self.error(error))?; + } + writer.flush().await.map_err(|error| self.error(error))?; + Ok(()) + } + + async fn connect(&self) -> Result { + let config = Config { + bootstrap_servers: self.container.bootstrap_servers.clone(), + ..Config::default() + }; + FlussConnection::new(config) + .await + .map_err(|error| self.error(error)) + } + + fn table_path() -> TablePath { + TablePath::new(DATABASE, TABLE) + } + + fn error(&self, error: fluss::error::Error) -> TestBinaryError { + TestBinaryError::FixtureSetup { + fixture_type: "FlussSourceFixture".to_string(), + message: format!("Apache Fluss client failure: {error}"), + } + } +} + +#[async_trait] +impl TestFixture for FlussSourceFixture { + async fn setup() -> Result { + Self::start(events_schema).await + } + + fn connectors_runtime_envs(&self) -> HashMap { + HashMap::from([ + ( + ENV_SOURCE_BOOTSTRAP_SERVERS.to_string(), + self.container.bootstrap_servers.clone(), + ), + (ENV_SOURCE_DATABASE.to_string(), DATABASE.to_string()), + (ENV_SOURCE_TABLE.to_string(), TABLE.to_string()), + ( + ENV_SOURCE_POLL_INTERVAL.to_string(), + POLL_INTERVAL.to_string(), + ), + (ENV_SOURCE_INCLUDE_METADATA.to_string(), "true".to_string()), + ( + ENV_SOURCE_STREAMS_0_STREAM.to_string(), + seeds::names::STREAM.to_string(), + ), + ( + ENV_SOURCE_STREAMS_0_TOPIC.to_string(), + seeds::names::TOPIC.to_string(), + ), + (ENV_SOURCE_STREAMS_0_SCHEMA.to_string(), "json".to_string()), + ( + ENV_SOURCE_PATH.to_string(), + "../../target/debug/libiggy_connector_fluss_source".to_string(), + ), + ]) + } +} + +/// Polls slowly enough to leave time for restarting Apache Iggy before the SDK's +/// consecutive-NACK limit stops the source. +pub struct FlussSourceSlowPollFixture { + inner: FlussSourceFixture, +} + +impl Deref for FlussSourceSlowPollFixture { + type Target = FlussSourceFixture; + + fn deref(&self) -> &Self::Target { + &self.inner + } +} + +#[async_trait] +impl TestFixture for FlussSourceSlowPollFixture { + async fn setup() -> Result { + Ok(Self { + inner: FlussSourceFixture::setup().await?, + }) + } + + fn connectors_runtime_envs(&self) -> HashMap { + let mut envs = self.inner.connectors_runtime_envs(); + envs.insert( + ENV_SOURCE_POLL_INTERVAL.to_string(), + SLOW_POLL_INTERVAL.to_string(), + ); + envs + } +} + +/// Starts from `latest` over a table that already holds rows, so none of those rows reaching +/// Apache Iggy shows the source began at the tail. +pub struct FlussSourceLatestFixture { + inner: FlussSourceFixture, + existing_payloads: Vec, +} + +impl FlussSourceLatestFixture { + /// Rows appended before the connectors runtime started. + pub fn existing_payloads(&self) -> &[String] { + &self.existing_payloads + } +} + +impl Deref for FlussSourceLatestFixture { + type Target = FlussSourceFixture; + + fn deref(&self) -> &Self::Target { + &self.inner + } +} + +#[async_trait] +impl TestFixture for FlussSourceLatestFixture { + async fn setup() -> Result { + let inner = FlussSourceFixture::setup().await?; + let existing_payloads: Vec = (0..EXISTING_ROW_COUNT) + .map(|index| format!("before-start-{index}")) + .collect(); + inner.append_rows(&existing_payloads).await?; + Ok(Self { + inner, + existing_payloads, + }) + } + + fn connectors_runtime_envs(&self) -> HashMap { + let mut envs = self.inner.connectors_runtime_envs(); + envs.insert(ENV_SOURCE_STARTING_OFFSET.to_string(), "latest".to_string()); + envs + } +} + +/// A log table with a column of every scalar type the source maps. The rows come back +/// decoded from Arrow by a real server, which is the path production reads, rather than +/// from a row built in memory. +pub struct FlussSourceAllTypesFixture { + inner: FlussSourceFixture, +} + +impl FlussSourceAllTypesFixture { + /// Appends one row with a value in every column, then one with every column null. + pub async fn append_sample_rows(&self) -> Result<(), TestBinaryError> { + let error = |error| self.inner.error(error); + let mut filled = GenericRow::new(ALL_TYPES_COLUMN_COUNT); + filled.set_field(0, true); + filled.set_field(1, 7i8); + filled.set_field(2, 300i16); + filled.set_field(3, 70_000i32); + filled.set_field(4, 9_000_000_000i64); + filled.set_field(5, 1.5f32); + filled.set_field(6, 2.25f64); + filled.set_field(7, "abcde"); + filled.set_field(8, "hello"); + filled.set_field( + 9, + Decimal::from_unscaled_long(12_345, 10, 2).map_err(error)?, + ); + filled.set_field(10, Date::new(19_782)); + filled.set_field(11, Time::new(45_296_789)); + filled.set_field( + 12, + TimestampNtz::from_millis_nanos(1_700_000_000_123, 456_000).map_err(error)?, + ); + filled.set_field( + 13, + TimestampNtz::from_millis_nanos(1_700_000_000_123, 456_789).map_err(error)?, + ); + filled.set_field( + 14, + TimestampLtz::from_millis_nanos(1_700_000_000_123, 456_000).map_err(error)?, + ); + filled.set_field(15, [1u8, 2, 3].as_slice()); + filled.set_field(16, [4u8, 5, 6].as_slice()); + + let empty = GenericRow::new(ALL_TYPES_COLUMN_COUNT); + self.inner.append(&[filled, empty]).await + } +} + +#[async_trait] +impl TestFixture for FlussSourceAllTypesFixture { + async fn setup() -> Result { + Ok(Self { + inner: FlussSourceFixture::start(all_types_schema).await?, + }) + } + + fn connectors_runtime_envs(&self) -> HashMap { + self.inner.connectors_runtime_envs() + } +} + +fn events_schema() -> fluss::error::Result { + Schema::builder() + .column("id", DataTypes::int()) + .column("payload", DataTypes::string()) + .build() +} + +/// Column order matches the positions `append_sample_rows` writes. +fn all_types_schema() -> fluss::error::Result { + Schema::builder() + .column("bool_col", DataTypes::boolean()) + .column("tinyint_col", DataTypes::tinyint()) + .column("smallint_col", DataTypes::smallint()) + .column("int_col", DataTypes::int()) + .column("bigint_col", DataTypes::bigint()) + .column("float_col", DataTypes::float()) + .column("double_col", DataTypes::double()) + .column("char_col", DataTypes::char(5)) + .column("string_col", DataTypes::string()) + .column("decimal_col", DataTypes::decimal(10, 2)) + .column("date_col", DataTypes::date()) + .column("time_col", DataTypes::time_with_precision(3)) + .column("timestamp_col", DataTypes::timestamp()) + .column( + "timestamp_nanos_col", + DataTypes::timestamp_with_precision(9), + ) + .column("timestamp_ltz_col", DataTypes::timestamp_ltz()) + .column("bytes_col", DataTypes::bytes()) + .column("binary_col", DataTypes::binary(3)) + .build() +} diff --git a/core/integration/tests/connectors/fixtures/mod.rs b/core/integration/tests/connectors/fixtures/mod.rs index cddee53ab1..58316baf96 100644 --- a/core/integration/tests/connectors/fixtures/mod.rs +++ b/core/integration/tests/connectors/fixtures/mod.rs @@ -22,6 +22,7 @@ mod delta; mod doris; mod elasticsearch; mod floci; +mod fluss; mod http; mod iceberg; mod influxdb; @@ -60,6 +61,10 @@ pub use doris::{ DorisSinkMaxFilterRatioFixture, DorisSinkPreCreatedFixture, }; pub use elasticsearch::{ElasticsearchSinkFixture, ElasticsearchSourcePreCreatedFixture}; +pub use fluss::{ + FlussSourceAllTypesFixture, FlussSourceFixture, FlussSourceLatestFixture, + FlussSourceSlowPollFixture, +}; pub use http::{ GITHUB_ENDPOINT_ID, GITHUB_HMAC_HEADER, GITHUB_INSTANCE, HttpSinkIndividualFixture, HttpSinkJsonArrayFixture, HttpSinkMultiTopicFixture, HttpSinkNdjsonFixture, diff --git a/core/integration/tests/connectors/fluss/fluss_source.rs b/core/integration/tests/connectors/fluss/fluss_source.rs new file mode 100644 index 0000000000..1d8c4d469e --- /dev/null +++ b/core/integration/tests/connectors/fluss/fluss_source.rs @@ -0,0 +1,362 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +use super::{API_KEY, POLL_ATTEMPTS, POLL_INTERVAL_MS, SOURCE_KEY, STATE_FILE, TEST_ROW_COUNT}; +use crate::connectors::fixtures::{ + FlussSourceAllTypesFixture, FlussSourceFixture, FlussSourceLatestFixture, + FlussSourceSlowPollFixture, +}; +use iggy::prelude::IggyClient; +use iggy_common::MessageClient; +use iggy_common::{Consumer, Identifier, PollingStrategy}; +use iggy_connector_sdk::api::{ConnectorRuntimeStats, ConnectorStats, ConnectorStatus}; +use integration::harness::seeds; +use integration::iggy_harness; +use reqwest::Client; +use serde::Deserialize; +use serde_json::{Value, json}; +use std::collections::HashSet; +use std::path::Path; +use std::time::Duration; +use tokio::time::{sleep, timeout}; + +/// Redelivery waits out the slow poll interval, the rejected batch and the Apache Iggy restart. +const REDELIVERY_ATTEMPTS: usize = POLL_ATTEMPTS * 3; +const SEND_FAILURE_TIMEOUT: Duration = Duration::from_secs(30); +const STATE_FILE_TIMEOUT: Duration = Duration::from_secs(15); + +#[derive(Debug, Deserialize)] +struct FlussRecord { + id: i32, + payload: String, + #[serde(rename = "_fluss_bucket")] + bucket: i32, + #[serde(rename = "_fluss_offset")] + offset: i64, + #[serde(rename = "_fluss_timestamp")] + timestamp: i64, +} + +#[iggy_harness( + server(connectors_runtime(config_path = "tests/connectors/fluss/source.toml")), + seed = seeds::connector_stream +)] +async fn log_table_rows_are_produced_to_iggy(harness: &TestHarness, fixture: FlussSourceFixture) { + let client = harness.root_client().await.unwrap(); + + let payloads = test_payloads(); + fixture + .append_rows(&payloads) + .await + .expect("Failed to append rows"); + + let received = poll_records(&client, TEST_ROW_COUNT, POLL_ATTEMPTS).await; + + assert!( + received.len() >= TEST_ROW_COUNT, + "Expected at least {TEST_ROW_COUNT} messages, got {}", + received.len() + ); + + for (index, record) in received.iter().take(TEST_ROW_COUNT).enumerate() { + assert_eq!(record.id, index as i32, "Column `id` mismatch at {index}"); + assert_eq!( + record.payload, payloads[index], + "Column `payload` mismatch at {index}" + ); + assert_eq!(record.bucket, 0, "Bucket mismatch at {index}"); + assert_eq!( + record.offset, index as i64, + "Fluss offset should be preserved and sequential at {index}" + ); + assert!( + record.timestamp > 0, + "Fluss timestamp should be populated at {index}" + ); + } +} + +#[iggy_harness( + cluster_nodes = 1, + server(connectors_runtime(config_path = "tests/connectors/fluss/source.toml")), + seed = seeds::connector_stream +)] +async fn given_rejected_batch_when_iggy_restarts_should_replay_rows_from_fluss( + harness: &mut TestHarness, + fixture: FlussSourceSlowPollFixture, +) { + // Without reconnection retries a send to a stopped server fails at once, so the runtime + // rejects the batch instead of blocking on it. + harness + .server_mut() + .stop_dependents() + .expect("Failed to stop connectors runtime"); + harness + .server_mut() + .connectors_runtime_mut() + .expect("connectors runtime") + .set_iggy_connection_options("reconnection_retries=0"); + harness + .server_mut() + .start_dependents() + .await + .expect("Failed to restart connectors runtime"); + + let api_url = harness + .connectors_runtime() + .expect("connectors runtime") + .http_url(); + let http = Client::new(); + let errors_before_failure = source_stats(&http, &api_url) + .await + .expect("Apache Fluss source stats should be present") + .errors; + + harness.kill_node(0).expect("Failed to kill Iggy server"); + let payloads = test_payloads(); + fixture + .append_rows(&payloads) + .await + .expect("Failed to append rows"); + wait_for_source_errors(&http, &api_url, errors_before_failure + 1).await; + + harness + .restart_node(0) + .expect("Failed to restart the Iggy server"); + + // The scanner handed these rows over before the batch was rejected, so they only arrive if + // the source rewound it instead of moving on. + let client = harness.root_client().await.unwrap(); + let received = poll_records(&client, payloads.len(), REDELIVERY_ATTEMPTS).await; + assert_rows_delivered(&received, &payloads, 0); +} + +#[iggy_harness( + server(connectors_runtime(config_path = "tests/connectors/fluss/source.toml")), + seed = seeds::connector_stream +)] +async fn given_latest_start_when_runtime_restarts_before_any_row_should_deliver_rows_written_meanwhile( + harness: &mut TestHarness, + fixture: FlussSourceLatestFixture, +) { + let state_path = harness + .connectors_runtime() + .expect("connectors runtime") + .state_path() + .join(STATE_FILE); + // No row has arrived yet, so only the tail offsets resolved for `latest` can have written it. + wait_for_file(&state_path).await; + + harness + .server_mut() + .stop_dependents() + .expect("Failed to stop connectors runtime"); + let payloads = test_payloads(); + fixture + .append_rows(&payloads) + .await + .expect("Failed to append rows"); + harness + .server_mut() + .start_dependents() + .await + .expect("Failed to restart connectors runtime"); + + let client = harness.root_client().await.unwrap(); + let received = poll_records(&client, payloads.len(), POLL_ATTEMPTS).await; + assert!( + received + .iter() + .all(|record| !fixture.existing_payloads().contains(&record.payload)), + "Rows written before the first start should be skipped by `latest`" + ); + assert_rows_delivered( + &received, + &payloads, + fixture.existing_payloads().len() as i64, + ); +} + +#[iggy_harness( + server(connectors_runtime(config_path = "tests/connectors/fluss/source.toml")), + seed = seeds::connector_stream +)] +async fn given_every_supported_column_type_should_map_each_to_json( + harness: &TestHarness, + fixture: FlussSourceAllTypesFixture, +) { + fixture + .append_sample_rows() + .await + .expect("Failed to append rows"); + + let client = harness.root_client().await.unwrap(); + let received = poll_payloads(&client, 2, POLL_ATTEMPTS).await; + let row_at = |offset: i64| { + received + .iter() + .find(|payload| payload["_fluss_offset"] == offset) + .unwrap_or_else(|| panic!("Row at Fluss offset {offset} was never delivered")) + }; + let (filled, empty) = (row_at(0), row_at(1)); + + let expected = json!({ + "bool_col": true, + "tinyint_col": 7, + "smallint_col": 300, + "int_col": 70_000, + "bigint_col": 9_000_000_000i64, + "float_col": 1.5, + "double_col": 2.25, + "char_col": "abcde", + "string_col": "hello", + "decimal_col": "123.45", + "date_col": "2024-02-29", + "time_col": "12:34:56.789", + "timestamp_col": "2023-11-14 22:13:20.123456", + "timestamp_nanos_col": "2023-11-14 22:13:20.123456789", + "timestamp_ltz_col": "2023-11-14T22:13:20.123456+00:00", + "bytes_col": "AQID", + "binary_col": "BAUG", + }); + for (column, value) in expected + .as_object() + .expect("Expected row should be an object") + { + assert_eq!( + &filled[column], value, + "Column `{column}` did not map as expected" + ); + assert_eq!( + empty[column], + Value::Null, + "Column `{column}` should be null in the empty row" + ); + } +} + +fn test_payloads() -> Vec { + (0..TEST_ROW_COUNT) + .map(|index| format!("fluss-payload-{index}")) + .collect() +} + +/// At-least-once delivery may repeat a row, so this checks that every row arrived, not how +/// many times. +fn assert_rows_delivered(received: &[FlussRecord], payloads: &[String], first_offset: i64) { + for (index, payload) in payloads.iter().enumerate() { + let offset = first_offset + index as i64; + assert!( + received + .iter() + .any(|record| record.offset == offset && &record.payload == payload), + "Row at Fluss offset {offset} ({payload}) was never delivered, got {} messages", + received.len() + ); + } +} + +async fn poll_records( + client: &IggyClient, + expected_rows: usize, + attempts: usize, +) -> Vec { + poll_payloads(client, expected_rows, attempts) + .await + .into_iter() + .filter_map(|payload| serde_json::from_value(payload).ok()) + .collect() +} + +/// Polls until messages for `expected_rows` distinct Fluss offsets have arrived, keeping +/// any repeats an at-least-once redelivery produced. +async fn poll_payloads(client: &IggyClient, expected_rows: usize, attempts: usize) -> Vec { + let stream_id: Identifier = seeds::names::STREAM.try_into().unwrap(); + let topic_id: Identifier = seeds::names::TOPIC.try_into().unwrap(); + let consumer_id: Identifier = "fluss_test_consumer".try_into().unwrap(); + + let mut received: Vec = Vec::new(); + for _ in 0..attempts { + if let Ok(polled) = client + .poll_messages( + &stream_id, + &topic_id, + None, + &Consumer::new(consumer_id.clone()), + &PollingStrategy::next(), + 10, + true, + ) + .await + { + for message in polled.messages { + if let Ok(payload) = serde_json::from_slice(&message.payload) { + received.push(payload); + } + } + let distinct_rows: HashSet = received + .iter() + .filter_map(|payload| payload["_fluss_offset"].as_i64()) + .collect(); + if distinct_rows.len() >= expected_rows { + break; + } + } + sleep(Duration::from_millis(POLL_INTERVAL_MS)).await; + } + received +} + +async fn source_stats(http: &Client, api_url: &str) -> Option { + let response = http + .get(format!("{api_url}/stats")) + .header("api-key", API_KEY) + .send() + .await + .ok()?; + let stats = response.json::().await.ok()?; + stats + .connectors + .into_iter() + .find(|connector| connector.key == SOURCE_KEY) +} + +async fn wait_for_source_errors(http: &Client, api_url: &str, minimum_errors: u64) { + timeout(SEND_FAILURE_TIMEOUT, async { + loop { + if let Some(source) = source_stats(http, api_url).await + && source.status == ConnectorStatus::Error + && source.errors >= minimum_errors + { + return; + } + sleep(Duration::from_millis(POLL_INTERVAL_MS)).await; + } + }) + .await + .expect("Apache Fluss source did not report the rejected batch"); +} + +async fn wait_for_file(path: &Path) { + timeout(STATE_FILE_TIMEOUT, async { + while !path.exists() { + sleep(Duration::from_millis(POLL_INTERVAL_MS)).await; + } + }) + .await + .expect("Apache Fluss source did not persist its start offsets"); +} diff --git a/core/integration/tests/connectors/fluss/mod.rs b/core/integration/tests/connectors/fluss/mod.rs new file mode 100644 index 0000000000..cca40043a5 --- /dev/null +++ b/core/integration/tests/connectors/fluss/mod.rs @@ -0,0 +1,25 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +mod fluss_source; + +const API_KEY: &str = "test-api-key"; +const SOURCE_KEY: &str = "fluss"; +const STATE_FILE: &str = "source_fluss.state"; +const TEST_ROW_COUNT: usize = 10; +const POLL_ATTEMPTS: usize = 30; +const POLL_INTERVAL_MS: u64 = 500; diff --git a/core/integration/tests/connectors/fluss/source.toml b/core/integration/tests/connectors/fluss/source.toml new file mode 100644 index 0000000000..57cb8a6b57 --- /dev/null +++ b/core/integration/tests/connectors/fluss/source.toml @@ -0,0 +1,20 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +[connectors] +config_type = "local" +config_dir = "../connectors/sources/fluss_source" diff --git a/core/integration/tests/connectors/mod.rs b/core/integration/tests/connectors/mod.rs index 54dd1437b9..11b374bc8e 100644 --- a/core/integration/tests/connectors/mod.rs +++ b/core/integration/tests/connectors/mod.rs @@ -21,6 +21,7 @@ mod delta; mod doris; mod elasticsearch; mod fixtures; +mod fluss; mod http; mod http_config_provider; mod iceberg; From fd0904251788def8e31625f2156f346fa85edc06 Mon Sep 17 00:00:00 2001 From: seokjin0414 Date: Thu, 24 Sep 2026 23:53:19 +0900 Subject: [PATCH 3/6] build: register the Apache Fluss source in the workspace Registers the connector as a regular workspace member so it inherits the shared dependency versions and stays inside cargo sort, the version bump script and the DAG-based test scoping, the way every other connector does. fluss-rs 1.0 ships its generated protobuf code, so building the workspace needs no protoc. It does pull in arrow 59 while the workspace stays on 58 for iceberg and deltalake, so the lockfile carries a second arrow line until those catch up. Signed-off-by: seokjin0414 --- Cargo.lock | 743 ++++++++++++++++++++++++++++++++++------ Cargo.toml | 2 + scripts/bump-version.sh | 2 +- 3 files changed, 642 insertions(+), 105 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index d0b272a665..6e2cbef34f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -491,7 +491,7 @@ dependencies = [ "digest 0.10.7", "log", "miniz_oxide 0.8.9", - "num-bigint", + "num-bigint 0.4.8", "quad-rand", "rand 0.9.5", "regex-lite", @@ -517,7 +517,7 @@ dependencies = [ "digest 0.11.3", "log", "miniz_oxide 0.9.1", - "num-bigint", + "num-bigint 0.4.8", "ouroboros", "quad-rand", "rand 0.10.2", @@ -612,19 +612,40 @@ version = "58.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6cfdd0833e32a9874d2b55089333ad310c0be208aafa277385ce2461dec90be3" dependencies = [ - "arrow-arith", - "arrow-array", - "arrow-buffer", - "arrow-cast", - "arrow-csv", - "arrow-data", - "arrow-ipc", - "arrow-json", - "arrow-ord", - "arrow-row", - "arrow-schema", - "arrow-select", - "arrow-string", + "arrow-arith 58.4.0", + "arrow-array 58.4.0", + "arrow-buffer 58.4.0", + "arrow-cast 58.4.0", + "arrow-csv 58.4.0", + "arrow-data 58.4.0", + "arrow-ipc 58.4.0", + "arrow-json 58.4.0", + "arrow-ord 58.4.0", + "arrow-row 58.4.0", + "arrow-schema 58.4.0", + "arrow-select 58.4.0", + "arrow-string 58.4.0", +] + +[[package]] +name = "arrow" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c14b3d39f306bc28fd639d59f06e17a0f377d0021e1b7e9054e4d6fedc98774" +dependencies = [ + "arrow-arith 59.3.0", + "arrow-array 59.3.0", + "arrow-buffer 59.3.0", + "arrow-cast 59.3.0", + "arrow-csv 59.3.0", + "arrow-data 59.3.0", + "arrow-ipc 59.3.0", + "arrow-json 59.3.0", + "arrow-ord 59.3.0", + "arrow-row 59.3.0", + "arrow-schema 59.3.0", + "arrow-select 59.3.0", + "arrow-string 59.3.0", ] [[package]] @@ -633,10 +654,24 @@ version = "58.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0a41203398f0eaa6f7ec8e62c0da742a21abf282c148fc157f6c35c90e29981a" dependencies = [ - "arrow-array", - "arrow-buffer", - "arrow-data", - "arrow-schema", + "arrow-array 58.4.0", + "arrow-buffer 58.4.0", + "arrow-data 58.4.0", + "arrow-schema 58.4.0", + "chrono", + "num-traits", +] + +[[package]] +name = "arrow-arith" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce2961626677665b2195eb59242af4c7befe7b8737ca2050295389362380104e" +dependencies = [ + "arrow-array 59.3.0", + "arrow-buffer 59.3.0", + "arrow-data 59.3.0", + "arrow-schema 59.3.0", "chrono", "num-traits", ] @@ -648,9 +683,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ae33dad492b7df00a217563a7b0ef2874df68a0deea1b1a3acf628152f7f7a69" dependencies = [ "ahash", - "arrow-buffer", - "arrow-data", - "arrow-schema", + "arrow-buffer 58.4.0", + "arrow-data 58.4.0", + "arrow-schema 58.4.0", "chrono", "chrono-tz", "half", @@ -660,6 +695,25 @@ dependencies = [ "num-traits", ] +[[package]] +name = "arrow-array" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e5f6adeffdf587d7a31db5d2266189624b526730cd3627f9ff9fedae97ad584" +dependencies = [ + "ahash", + "arrow-buffer 59.3.0", + "arrow-data 59.3.0", + "arrow-schema 59.3.0", + "chrono", + "half", + "hashbrown 0.17.1", + "libc", + "num-complex", + "num-integer", + "num-traits", +] + [[package]] name = "arrow-buffer" version = "58.4.0" @@ -668,7 +722,19 @@ checksum = "b9552f96391c005e6ab449fa941420935e7e062489b12b8b1b08879b2163f5b5" dependencies = [ "bytes", "half", - "num-bigint", + "num-bigint 0.4.8", + "num-traits", +] + +[[package]] +name = "arrow-buffer" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "097d193003ce7995d5d087089069ec2a6e0187faf5a6f8c9f38af2645d987182" +dependencies = [ + "bytes", + "half", + "num-bigint 0.5.1", "num-traits", ] @@ -678,12 +744,12 @@ version = "58.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3a8a327c9649f30d8406995f27642b68df354713cca3baaaf100f076f18d5f34" dependencies = [ - "arrow-array", - "arrow-buffer", - "arrow-data", - "arrow-ord", - "arrow-schema", - "arrow-select", + "arrow-array 58.4.0", + "arrow-buffer 58.4.0", + "arrow-data 58.4.0", + "arrow-ord 58.4.0", + "arrow-schema 58.4.0", + "arrow-select 58.4.0", "atoi", "base64 0.22.1", "chrono", @@ -693,15 +759,51 @@ dependencies = [ "ryu", ] +[[package]] +name = "arrow-cast" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "635c9c635668ad26adf76cce8fb276c4be7cf06e63bd516de7da514f9680ee53" +dependencies = [ + "arrow-array 59.3.0", + "arrow-buffer 59.3.0", + "arrow-data 59.3.0", + "arrow-ord 59.3.0", + "arrow-schema 59.3.0", + "arrow-select 59.3.0", + "atoi", + "base64 0.23.1", + "chrono", + "half", + "lexical-core", + "num-traits", + "ryu", +] + [[package]] name = "arrow-csv" version = "58.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "af0dd6d90d1955e9f9a014c1e563ee8aeffc21909085d25623e1da44d96eca26" dependencies = [ - "arrow-array", - "arrow-cast", - "arrow-schema", + "arrow-array 58.4.0", + "arrow-cast 58.4.0", + "arrow-schema 58.4.0", + "chrono", + "csv", + "csv-core", + "regex", +] + +[[package]] +name = "arrow-csv" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c2ebf8d631e79b02c16cf5ae860561272c26024ec88fce389a56aaddd558e86" +dependencies = [ + "arrow-array 59.3.0", + "arrow-cast 59.3.0", + "arrow-schema 59.3.0", "chrono", "csv", "csv-core", @@ -714,8 +816,21 @@ version = "58.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2b24852db04738907e06c04ea61e42fe7fda962a34513022dc0d0e754fb7976b" dependencies = [ - "arrow-buffer", - "arrow-schema", + "arrow-buffer 58.4.0", + "arrow-schema 58.4.0", + "half", + "num-integer", + "num-traits", +] + +[[package]] +name = "arrow-data" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ba2f832eaeca24b8f26143dba750e42ee4ab51cf7d65e701ca9607cfda9f358" +dependencies = [ + "arrow-buffer 59.3.0", + "arrow-schema 59.3.0", "half", "num-integer", "num-traits", @@ -727,26 +842,67 @@ version = "58.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "29a908a11fcfb3fb2f6730f4ac15e367bc644e419155e96238f68cf3adde572b" dependencies = [ - "arrow-array", - "arrow-buffer", - "arrow-data", - "arrow-schema", - "arrow-select", + "arrow-array 58.4.0", + "arrow-buffer 58.4.0", + "arrow-data 58.4.0", + "arrow-schema 58.4.0", + "arrow-select 58.4.0", "flatbuffers", ] +[[package]] +name = "arrow-ipc" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dcc41681ea80f521df14c36725b74d4c60702c47f0793af2be469c04527e2599" +dependencies = [ + "arrow-array 59.3.0", + "arrow-buffer 59.3.0", + "arrow-data 59.3.0", + "arrow-schema 59.3.0", + "arrow-select 59.3.0", + "flatbuffers", + "lz4_flex 0.14.0", + "zstd 0.13.3", +] + [[package]] name = "arrow-json" version = "58.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b8a96aed3931c076adee39ec2a40d8219fc7f09e79bcdaca1df16272993e1e14" dependencies = [ - "arrow-array", - "arrow-buffer", - "arrow-cast", - "arrow-ord", - "arrow-schema", - "arrow-select", + "arrow-array 58.4.0", + "arrow-buffer 58.4.0", + "arrow-cast 58.4.0", + "arrow-ord 58.4.0", + "arrow-schema 58.4.0", + "arrow-select 58.4.0", + "chrono", + "half", + "indexmap 2.14.2", + "itoa", + "lexical-core", + "memchr", + "num-traits", + "ryu", + "serde_core", + "serde_json", + "simdutf8", +] + +[[package]] +name = "arrow-json" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2f57d7a81969f24ccf80809587b76c09897e6f829d2d65a5976bfb3218851f1" +dependencies = [ + "arrow-array 59.3.0", + "arrow-buffer 59.3.0", + "arrow-cast 59.3.0", + "arrow-ord 59.3.0", + "arrow-schema 59.3.0", + "arrow-select 59.3.0", "chrono", "half", "indexmap 2.14.2", @@ -766,11 +922,24 @@ version = "58.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "63a083ec750f5c043f02946b4baf05fcdbb55f4560a3277055caca5cc99f3eb0" dependencies = [ - "arrow-array", - "arrow-buffer", - "arrow-data", - "arrow-schema", - "arrow-select", + "arrow-array 58.4.0", + "arrow-buffer 58.4.0", + "arrow-data 58.4.0", + "arrow-schema 58.4.0", + "arrow-select 58.4.0", +] + +[[package]] +name = "arrow-ord" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c900759f3bd8354fd4196bc4403eee846894dc2adf66b4225472006a0bf18c5" +dependencies = [ + "arrow-array 59.3.0", + "arrow-buffer 59.3.0", + "arrow-data 59.3.0", + "arrow-schema 59.3.0", + "arrow-select 59.3.0", ] [[package]] @@ -779,10 +948,23 @@ version = "58.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "514ba0ef0d4c5896202dae736251ce415abb43a950bed570fb7981b8716c0e4c" dependencies = [ - "arrow-array", - "arrow-buffer", - "arrow-data", - "arrow-schema", + "arrow-array 58.4.0", + "arrow-buffer 58.4.0", + "arrow-data 58.4.0", + "arrow-schema 58.4.0", + "half", +] + +[[package]] +name = "arrow-row" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f4c6425032e28266e3fc4ff680805e57e670d6ea92473043f3e65b7ed6ac79f2" +dependencies = [ + "arrow-array 59.3.0", + "arrow-buffer 59.3.0", + "arrow-data 59.3.0", + "arrow-schema 59.3.0", "half", ] @@ -797,6 +979,15 @@ dependencies = [ "serde_core", ] +[[package]] +name = "arrow-schema" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10fab8d4563491417ba801fab29d205104d20d4bdf37bda6cd1cf425cff598cd" +dependencies = [ + "bitflags 2.13.2", +] + [[package]] name = "arrow-select" version = "58.4.0" @@ -804,10 +995,24 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c58da39eb3d8350ad4a549e5c2bc49284dac554016c69829310350f1731b0aad" dependencies = [ "ahash", - "arrow-array", - "arrow-buffer", - "arrow-data", - "arrow-schema", + "arrow-array 58.4.0", + "arrow-buffer 58.4.0", + "arrow-data 58.4.0", + "arrow-schema 58.4.0", + "num-traits", +] + +[[package]] +name = "arrow-select" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc58569193c2525915f3cc6310edba3792f1200f65d6e9ed330aa33e691493b8" +dependencies = [ + "ahash", + "arrow-array 59.3.0", + "arrow-buffer 59.3.0", + "arrow-data 59.3.0", + "arrow-schema 59.3.0", "num-traits", ] @@ -817,11 +1022,28 @@ version = "58.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6789b388467525e3271326b6b4915666ecfdf5142aef09779445c954b67543c" dependencies = [ - "arrow-array", - "arrow-buffer", - "arrow-data", - "arrow-schema", - "arrow-select", + "arrow-array 58.4.0", + "arrow-buffer 58.4.0", + "arrow-data 58.4.0", + "arrow-schema 58.4.0", + "arrow-select 58.4.0", + "memchr", + "num-traits", + "regex", + "regex-syntax", +] + +[[package]] +name = "arrow-string" +version = "59.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2e0813f3c35c1cfea65e14c20a953440f7783c088b7ad2d0db162ccdeefcec14" +dependencies = [ + "arrow-array 59.3.0", + "arrow-buffer 59.3.0", + "arrow-data 59.3.0", + "arrow-schema 59.3.0", + "arrow-select 59.3.0", "memchr", "num-traits", "regex", @@ -2032,7 +2254,7 @@ checksum = "4d6867f1565b3aad85681f1015055b087fcfd840d6aeee6eee7f2da317603695" dependencies = [ "autocfg", "libm", - "num-bigint", + "num-bigint 0.4.8", "num-integer", "num-traits", "serde", @@ -2475,7 +2697,7 @@ version = "0.22.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2235eb320cd7178862a32dd111bd0c0f71a368e393add4914c50129add478eab" dependencies = [ - "arrow", + "arrow 58.4.0", "buoyant_kernel_derive", "bytes", "chrono", @@ -4035,6 +4257,17 @@ dependencies = [ "thiserror 2.0.20", ] +[[package]] +name = "delegate" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "780eb241654bf097afb00fc5f054a09b687dad862e485fdcf8399bb056565370" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + [[package]] name = "deltalake" version = "0.32.4" @@ -4096,17 +4329,17 @@ version = "0.32.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4588e95ff3b2ccdba56d9ec262bd3467c0593000f729402528706f62be8be1ca" dependencies = [ - "arrow", - "arrow-arith", - "arrow-array", - "arrow-buffer", - "arrow-cast", - "arrow-ipc", - "arrow-json", - "arrow-ord", - "arrow-row", - "arrow-schema", - "arrow-select", + "arrow 58.4.0", + "arrow-arith 58.4.0", + "arrow-array 58.4.0", + "arrow-buffer 58.4.0", + "arrow-cast 58.4.0", + "arrow-ipc 58.4.0", + "arrow-json 58.4.0", + "arrow-ord 58.4.0", + "arrow-row 58.4.0", + "arrow-schema 58.4.0", + "arrow-select 58.4.0", "async-trait", "buoyant_kernel", "bytes", @@ -4303,7 +4536,7 @@ dependencies = [ "asn1-rs", "displaydoc", "nom 7.1.3", - "num-bigint", + "num-bigint 0.4.8", "num-traits", "rusticata-macros", ] @@ -5171,6 +5404,46 @@ dependencies = [ "spin", ] +[[package]] +name = "fluss-rs" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a496c796a0e7831d4e1d5679f5e7b5973948f612108d70461143994bdff2a848" +dependencies = [ + "arrow 59.3.0", + "arrow-schema 59.3.0", + "bigdecimal", + "bitvec", + "byteorder", + "bytes", + "clap", + "crc32c", + "dashmap", + "delegate", + "futures", + "jiff", + "linked-hash-map", + "log", + "metrics", + "opendal 0.55.0", + "ordered-float 5.5.0", + "parking_lot", + "parse-display 0.10.0", + "prost", + "rand 0.9.5", + "scopeguard", + "serde", + "serde_json", + "snafu", + "strum 0.26.3", + "strum_macros 0.26.4", + "tempfile", + "thiserror 1.0.69", + "tokio", + "url", + "uuid", +] + [[package]] name = "fnv" version = "1.0.7" @@ -6579,14 +6852,14 @@ dependencies = [ "anyhow", "apache-avro 0.21.0", "array-init", - "arrow-arith", - "arrow-array", - "arrow-buffer", - "arrow-cast", - "arrow-ord", - "arrow-schema", - "arrow-select", - "arrow-string", + "arrow-arith 58.4.0", + "arrow-array 58.4.0", + "arrow-buffer 58.4.0", + "arrow-cast 58.4.0", + "arrow-ord 58.4.0", + "arrow-schema 58.4.0", + "arrow-select 58.4.0", + "arrow-string 58.4.0", "as-any", "async-trait", "backon", @@ -6658,7 +6931,7 @@ dependencies = [ "cfg-if", "futures", "iceberg", - "opendal", + "opendal 0.57.0", "reqsign-aws-v4", "reqsign-core", "serde", @@ -7219,6 +7492,27 @@ dependencies = [ "wiremock", ] +[[package]] +name = "iggy_connector_fluss_source" +version = "0.5.0" +dependencies = [ + "async-trait", + "base64 0.23.1", + "chrono", + "dashmap", + "fluss-rs", + "humantime", + "iggy_common", + "iggy_connector_sdk", + "rmp-serde", + "secrecy", + "serde", + "serde_json", + "simd-json", + "tokio", + "tracing", +] + [[package]] name = "iggy_connector_http_sink" version = "0.5.0" @@ -7271,8 +7565,8 @@ dependencies = [ name = "iggy_connector_iceberg_sink" version = "0.5.0" dependencies = [ - "arrow-array", - "arrow-json", + "arrow-array 58.4.0", + "arrow-json 58.4.0", "async-trait", "dashmap", "iceberg", @@ -7466,7 +7760,7 @@ dependencies = [ name = "iggy_connector_redshift_sink" version = "0.4.1" dependencies = [ - "arrow", + "arrow 58.4.0", "async-trait", "chrono", "humantime", @@ -7777,7 +8071,7 @@ name = "integration" version = "0.0.1" dependencies = [ "apache-avro 0.22.0", - "arrow", + "arrow 58.4.0", "assert_cmd", "async-trait", "base64 0.23.1", @@ -7793,6 +8087,7 @@ dependencies = [ "deltalake", "dtor 1.0.6", "figment", + "fluss-rs", "futures", "harness_derive", "hex", @@ -8562,6 +8857,9 @@ name = "log" version = "0.4.34" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6" +dependencies = [ + "value-bag", +] [[package]] name = "logos" @@ -8940,6 +9238,16 @@ dependencies = [ "tracing", ] +[[package]] +name = "metrics" +version = "0.24.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89550ee9f79e88fef3119de263694973a8adb26c21d75322164fb8c493039fe2" +dependencies = [ + "portable-atomic", + "rapidhash", +] + [[package]] name = "miette" version = "7.6.0" @@ -9343,7 +9651,7 @@ version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "35bd024e8b2ff75562e5f34e7f4905839deb4b22955ef5e73d2fea1b9813cb23" dependencies = [ - "num-bigint", + "num-bigint 0.4.8", "num-complex", "num-integer", "num-iter", @@ -9363,6 +9671,16 @@ dependencies = [ "serde", ] +[[package]] +name = "num-bigint" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93e7820bc0a80a0238e650327316f929ba18d5be054b647490a3a6a339f3e7c0" +dependencies = [ + "num-integer", + "num-traits", +] + [[package]] name = "num-bigint-dig" version = "0.8.6" @@ -9446,7 +9764,7 @@ version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" dependencies = [ - "num-bigint", + "num-bigint 0.4.8", "num-integer", "num-traits", ] @@ -9667,6 +9985,33 @@ version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381" +[[package]] +name = "opendal" +version = "0.55.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d075ab8a203a6ab4bc1bce0a4b9fe486a72bf8b939037f4b78d95386384bc80a" +dependencies = [ + "anyhow", + "backon", + "base64 0.22.1", + "bytes", + "futures", + "getrandom 0.2.17", + "http 1.5.0", + "http-body 1.1.0", + "jiff", + "log", + "md-5 0.10.6", + "percent-encoding", + "quick-xml 0.38.4", + "reqwest 0.12.28", + "serde", + "serde_json", + "tokio", + "url", + "uuid", +] + [[package]] name = "opendal" version = "0.57.0" @@ -9921,6 +10266,17 @@ dependencies = [ "num-traits", ] +[[package]] +name = "ordered-float" +version = "5.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8c7c9e0d9b23589f26070720bac724174bfec1083e82f7854cdd0267518343c0" +dependencies = [ + "num-traits", + "rand 0.8.8", + "serde", +] + [[package]] name = "ordered-multimap" version = "0.7.3" @@ -10069,12 +10425,12 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d298093b2dec60289dce0684c986d0f7679e9dd15771c2c65406e1aaf604a704" dependencies = [ "ahash", - "arrow-array", - "arrow-buffer", - "arrow-data", - "arrow-ipc", - "arrow-schema", - "arrow-select", + "arrow-array 58.4.0", + "arrow-buffer 58.4.0", + "arrow-data 58.4.0", + "arrow-ipc 58.4.0", + "arrow-schema 58.4.0", + "arrow-select 58.4.0", "base64 0.22.1", "brotli", "bytes", @@ -10084,7 +10440,7 @@ dependencies = [ "half", "hashbrown 0.17.1", "lz4_flex 0.13.1", - "num-bigint", + "num-bigint 0.4.8", "num-integer", "num-traits", "object_store", @@ -10105,7 +10461,18 @@ version = "0.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "914a1c2265c98e2446911282c6ac86d8524f495792c38c5bd884f80499c7538a" dependencies = [ - "parse-display-derive", + "parse-display-derive 0.9.1", + "regex", + "regex-syntax", +] + +[[package]] +name = "parse-display" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "287d8d3ebdce117b8539f59411e4ed9ec226e0a4153c7f55495c6070d68e6f72" +dependencies = [ + "parse-display-derive 0.10.0", "regex", "regex-syntax", ] @@ -10124,6 +10491,20 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "parse-display-derive" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7fc048687be30d79502dea2f623d052f3a074012c6eac41726b7ab17213616b1" +dependencies = [ + "proc-macro2", + "quote", + "regex", + "regex-syntax", + "structmeta", + "syn 2.0.119", +] + [[package]] name = "partitions" version = "0.1.0" @@ -11224,6 +11605,7 @@ dependencies = [ "libc", "rand_chacha 0.3.1", "rand_core 0.6.4", + "serde", ] [[package]] @@ -11274,6 +11656,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" dependencies = [ "getrandom 0.2.17", + "serde", ] [[package]] @@ -11309,6 +11692,15 @@ dependencies = [ "rand_core 0.10.1", ] +[[package]] +name = "rapidhash" +version = "4.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5da7e78a036ce858e8d55b7e7dc8ba3a88b78350fd2155d3591bbd966b58589e" +dependencies = [ + "rustversion", +] + [[package]] name = "rav1e" version = "0.8.1" @@ -12545,6 +12937,15 @@ dependencies = [ "syn 3.0.5", ] +[[package]] +name = "serde_fmt" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e497af288b3b95d067a23a4f749f2861121ffcb2f6d8379310dcda040c345ed" +dependencies = [ + "serde_core", +] + [[package]] name = "serde_json" version = "1.0.151" @@ -12618,7 +13019,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d4f284b4d521591b17ddee01aff830dd005a04476f7862aca9298c038d00fb7e" dependencies = [ "deno_error", - "num-bigint", + "num-bigint 0.4.8", "serde", "smallvec", "thiserror 2.0.20", @@ -12995,7 +13396,7 @@ version = "0.6.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0d585997b0ac10be3c5ee635f1bab02d512760d14b7c468801ac8a01d9ae5f1d" dependencies = [ - "num-bigint", + "num-bigint 0.4.8", "num-traits", "thiserror 2.0.20", "time", @@ -13375,7 +13776,7 @@ dependencies = [ "log", "md-5 0.11.0", "memchr", - "num-bigint", + "num-bigint 0.4.8", "rand 0.10.2", "serde", "serde_json", @@ -13522,6 +13923,12 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "strum" +version = "0.26.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06" + [[package]] name = "strum" version = "0.27.2" @@ -13540,6 +13947,19 @@ dependencies = [ "strum_macros 0.28.0", ] +[[package]] +name = "strum_macros" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c6bee85a5a24955dc440386795aa378cd9cf82acd5f764469152d2270e581be" +dependencies = [ + "heck 0.5.0", + "proc-macro2", + "quote", + "rustversion", + "syn 2.0.119", +] + [[package]] name = "strum_macros" version = "0.27.2" @@ -13570,6 +13990,85 @@ version = "2.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" +[[package]] +name = "sval" +version = "2.22.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b81b254da21fe1fcc4e3a74fe39b46e25e3a863078f8b71c954d47f84889dbc6" + +[[package]] +name = "sval_buffer" +version = "2.22.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "50be352d2822ffafb59e3e2ddac9d5ee60f2eeadbb7b5a2a951b9f3651e87a6f" +dependencies = [ + "sval", + "sval_ref", + "zerocopy", +] + +[[package]] +name = "sval_dynamic" +version = "2.22.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b048ca293b998d9a45659159f94a64063791e74cdc670164943dbb434405573d" +dependencies = [ + "sval", +] + +[[package]] +name = "sval_fmt" +version = "2.22.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6b5888e40f80568733217f27b7317b845f463400ced36c424b1a804730e53b2" +dependencies = [ + "itoa", + "ryu", + "sval", +] + +[[package]] +name = "sval_json" +version = "2.22.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e17664d6bb6b74947afaab9d7c991caa9bf5638d4dee16fcbef637f440796049" +dependencies = [ + "itoa", + "ryu", + "sval", +] + +[[package]] +name = "sval_nested" +version = "2.22.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "07c059969ca5ca163ea7fef6c9661758973d17691aba92abdcf5c428f4ec122c" +dependencies = [ + "sval", + "sval_buffer", + "sval_ref", +] + +[[package]] +name = "sval_ref" +version = "2.22.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42d6b29ff568c85c87561807f51d2adfff4b6016c6363133f7cd1652a12548f3" +dependencies = [ + "sval", +] + +[[package]] +name = "sval_serde" +version = "2.22.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f33ec9edc42b12764d5c90ca0a1d84189c6bde81ed27507f1e661c6e4e05853" +dependencies = [ + "serde_core", + "sval", + "sval_nested", +] + [[package]] name = "svgtypes" version = "0.15.3" @@ -13884,7 +14383,7 @@ dependencies = [ "itertools 0.14.0", "log", "memchr", - "parse-display", + "parse-display 0.9.1", "pin-project-lite", "reqwest 0.13.5", "serde", @@ -15064,6 +15563,42 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" +[[package]] +name = "value-bag" +version = "1.14.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2799ffb329a792ecfd902b71306c8a815a6ef1c0470fa9953a6aa4d4cecbe511" +dependencies = [ + "value-bag-serde1", + "value-bag-sval2", +] + +[[package]] +name = "value-bag-serde1" +version = "1.14.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0941feceafbe7a8f59ea1096d45b97002884a41306315ad797b3684b63a81d8c" +dependencies = [ + "erased-serde", + "serde_core", + "serde_fmt", +] + +[[package]] +name = "value-bag-sval2" +version = "1.14.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "839752af8179287d27eb2b94164641b1ede9e60ab7424163388dc21ebd0508cd" +dependencies = [ + "sval", + "sval_buffer", + "sval_dynamic", + "sval_fmt", + "sval_json", + "sval_ref", + "sval_serde", +] + [[package]] name = "value-trait" version = "0.12.2" diff --git a/Cargo.toml b/Cargo.toml index 7956ce5587..281ab58ede 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -50,6 +50,7 @@ members = [ "core/connectors/sinks/stdout_sink", "core/connectors/sinks/surrealdb_sink", "core/connectors/sources/elasticsearch_source", + "core/connectors/sources/fluss_source", "core/connectors/sources/http_source", "core/connectors/sources/influxdb_source", "core/connectors/sources/postgres_source", @@ -180,6 +181,7 @@ file-operation = "0.8.32" flatbuffers = "25.12.19" flate2 = "1.1.10" flume = "0.12.0" +fluss-rs = "1.0.0" fs2 = "0.4.3" futures = "0.3.34" futures-core = { version = "0.3.34", default-features = false } diff --git a/scripts/bump-version.sh b/scripts/bump-version.sh index d64ffe8c98..f31b5a7ed9 100755 --- a/scripts/bump-version.sh +++ b/scripts/bump-version.sh @@ -88,7 +88,7 @@ EOF RUST_COMPONENTS="rust-sdk rust-common rust-binary-protocol rust-server rust-cli rust-connector-sdk rust-mcp rust-bench rust-bench-dashboard-frontend rust-bench-dashboard-server rust-bench-report" CONNECTOR_SINK_COMPONENTS="rust-connector-clickhouse-sink rust-connector-delta-sink rust-connector-doris-sink rust-connector-elasticsearch-sink rust-connector-http-sink rust-connector-iceberg-sink rust-connector-influxdb-sink rust-connector-meilisearch-sink rust-connector-mongodb-sink rust-connector-postgres-sink rust-connector-quickwit-sink rust-connector-rabbitmq-sink rust-connector-redshift-sink rust-connector-s3-sink rust-connector-stdout-sink rust-connector-surrealdb-sink" -CONNECTOR_SOURCE_COMPONENTS="rust-connector-elasticsearch-source rust-connector-http-source rust-connector-influxdb-source rust-connector-postgres-source rust-connector-random-source" +CONNECTOR_SOURCE_COMPONENTS="rust-connector-elasticsearch-source rust-connector-fluss-source rust-connector-http-source rust-connector-influxdb-source rust-connector-postgres-source rust-connector-random-source" CONNECTOR_COMPONENTS="rust-connector-runtime ${CONNECTOR_SINK_COMPONENTS} ${CONNECTOR_SOURCE_COMPONENTS}" SDK_COMPONENTS="sdk-python sdk-node sdk-go sdk-csharp sdk-java" ALL_COMPONENTS="${RUST_COMPONENTS} ${CONNECTOR_COMPONENTS} ${SDK_COMPONENTS} web-ui" From 828f4d4247843fef3b273e5aa1534a8c5f2ef9a5 Mon Sep 17 00:00:00 2001 From: seokjin0414 Date: Sun, 27 Sep 2026 14:32:15 +0900 Subject: [PATCH 4/6] fix(connectors): stop Fluss source stalling on rows it cannot convert A row the JSON mapping cannot hold used to fail its whole batch. The failure repeats on every read and the SDK only logs a poll error, so the source stopped making progress while still reporting itself as running. Such a row is now dropped with an error naming its bucket and offset, the offsets move past it, and close() logs how many rows were skipped, as the runtime and the HTTP sink do with messages they cannot decode. Bad settings now fail before the connection is made: empty connection settings, an empty or repeated column list, and a poll interval and poll timeout that are both zero. With include_metadata on, a column under the _fluss_ prefix is rejected instead of being overwritten. Saved offsets for buckets the table no longer has are ignored, and the projection is resolved once so the row decoder and the scanner agree on column positions. Signed-off-by: seokjin0414 --- .../sources/fluss_source/Cargo.toml | 4 +- .../connectors/sources/fluss_source/README.md | 13 +- .../sources/fluss_source/src/lib.rs | 482 +++++++++++++++--- .../connectors/fixtures/fluss/container.rs | 1 + .../tests/connectors/fixtures/fluss/mod.rs | 3 +- .../tests/connectors/fixtures/fluss/source.rs | 117 ++++- .../tests/connectors/fixtures/mod.rs | 3 +- .../tests/connectors/fluss/fluss_source.rs | 142 +++++- 8 files changed, 679 insertions(+), 86 deletions(-) diff --git a/core/connectors/sources/fluss_source/Cargo.toml b/core/connectors/sources/fluss_source/Cargo.toml index eb08c0189a..a1b9a29c8d 100644 --- a/core/connectors/sources/fluss_source/Cargo.toml +++ b/core/connectors/sources/fluss_source/Cargo.toml @@ -44,7 +44,6 @@ fluss-rs = { workspace = true } humantime = { workspace = true } iggy_common = { workspace = true } iggy_connector_sdk = { workspace = true } -rmp-serde = { workspace = true } secrecy = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } @@ -52,5 +51,8 @@ simd-json = { workspace = true } tokio = { workspace = true } tracing = { workspace = true } +[dev-dependencies] +rmp-serde = { workspace = true } + [lints] workspace = true diff --git a/core/connectors/sources/fluss_source/README.md b/core/connectors/sources/fluss_source/README.md index 8ff5fa601f..e3acdfdbd2 100644 --- a/core/connectors/sources/fluss_source/README.md +++ b/core/connectors/sources/fluss_source/README.md @@ -12,13 +12,13 @@ Each Fluss row becomes one Apache Iggy message. Offsets are tracked per bucket a | `database` | yes | | Apache Fluss database name. | | `table` | yes | | Apache Fluss table name. | | `table_type` | no | `log` | Only `log` is accepted. See [Limitations](#limitations). | -| `starting_offset` | no | `earliest` | `earliest`, `latest` (each bucket's tail, resolved at startup), or an explicit non-negative offset. Applies only to buckets absent from the persisted state. | -| `columns` | no | all columns | Column projection pushed down to the server. | +| `starting_offset` | no | `earliest` | `earliest`, `latest` (each bucket's tail, resolved at startup), or an explicit non-negative offset. Applies only to buckets absent from the persisted state. An explicit offset is used as is for every such bucket, so it suits a table whose bucket offsets you know, such as one with a single bucket. | +| `columns` | no | all columns | Column projection pushed down to the server. Must name at least one column, and no column twice. | | `poll_interval` | no | `1s` | Delay before each poll. | -| `poll_timeout` | no | `5s` | How long a single server poll waits for records. | -| `batch_size` | no | client default | Maximum records returned per poll (`scanner.log.max-poll-records`). Must be greater than 0. | +| `poll_timeout` | no | `5s` | How long a single server poll waits for records. It and `poll_interval` cannot both be zero. | +| `batch_size` | no | `500` | Maximum records returned per poll (`scanner.log.max-poll-records`). Must be greater than 0. | | `payload_format` | no | `json` | Only `json` is accepted. See [Limitations](#limitations). | -| `include_metadata` | no | `false` | Adds `_fluss_bucket`, `_fluss_offset` and `_fluss_timestamp` to each JSON object. | +| `include_metadata` | no | `false` | Adds `_fluss_bucket`, `_fluss_offset` and `_fluss_timestamp` to each JSON object. While it is on, a column whose name starts with `_fluss_` is rejected at startup, since it would be overwritten. | | `sasl_username` | no | | Enables SASL/PLAIN together with `sasl_password`. Set both or neither. | | `sasl_password` | no | | Stored as a secret and redacted from logs and the `/stats` endpoint. | | `verbose_logging` | no | `false` | Logs per-batch counts at info instead of debug. | @@ -78,6 +78,9 @@ These are limits of this connector, not of the `fluss-rs` client. - **Primary-key tables are not supported yet.** `fluss-rs` 1.0 reads their changelog, but mapping its insert, update and delete records onto messages is left for a follow-up, so `table_type = "primary_key"` is rejected at startup. - **`payload_format = "arrow_ipc"` is not implemented yet.** The client does expose an Arrow `RecordBatch` scanner, but it uses a different offset-tracking path, so it is left for a follow-up. - **Partitioned tables are not supported.** They are detected and rejected at startup. +- **Rows that cannot be converted are skipped.** A value the JSON mapping cannot hold, such as a date beyond what `chrono` represents, fails the same way on every read. Such a row is dropped and logged at error level with its bucket and offset, the offsets move past it, and the number of skipped rows is logged when the connector closes. +- **A crash right after starting from `latest` can miss rows.** The resolved start offsets stay in memory until the first batch is acknowledged, which normally takes one poll. If the connector crashes in that window, the next start resolves `latest` again and misses the rows written in between. +- **An offset outside a bucket's range is not reset.** If a configured offset, or one restored from state after retention removed it, falls outside what the bucket still holds, polls keep failing until the offset is changed. The connector does not move to the earliest or latest offset on its own. ## Build and test diff --git a/core/connectors/sources/fluss_source/src/lib.rs b/core/connectors/sources/fluss_source/src/lib.rs index db47946dc8..c8ed8ecec2 100644 --- a/core/connectors/sources/fluss_source/src/lib.rs +++ b/core/connectors/sources/fluss_source/src/lib.rs @@ -20,7 +20,7 @@ mod mapping; use async_trait::async_trait; use fluss::client::{EARLIEST_OFFSET, FlussConnection, LogScanner}; use fluss::config::Config; -use fluss::metadata::{DataField, TablePath}; +use fluss::metadata::{DataField, RowType, TablePath}; use fluss::record::ScanRecords; use fluss::rpc::message::OffsetSpec; use iggy_connector_sdk::retry::{RetryPolicy, retry_async}; @@ -31,13 +31,13 @@ use iggy_connector_sdk::{ use secrecy::{ExposeSecret, SecretString}; use serde::{Deserialize, Serialize}; use serde_json::Value; -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; use std::str::FromStr; -use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::time::Duration; use tokio::sync::Mutex; use tokio::time::sleep; -use tracing::{debug, info, warn}; +use tracing::{debug, error, info, warn}; source_connector!(FlussSource); @@ -49,6 +49,7 @@ const JSON_PAYLOAD_FORMAT: &str = "json"; const METADATA_BUCKET: &str = "_fluss_bucket"; const METADATA_OFFSET: &str = "_fluss_offset"; const METADATA_TIMESTAMP: &str = "_fluss_timestamp"; +const METADATA_PREFIX: &str = "_fluss_"; const NANOS_PER_MILLI: u64 = 1_000_000; /// A rewind is local scanner bookkeeping unless the client has to refresh the table's /// metadata first. An error from `on_batch_result` stops the source, so that refresh gets a @@ -70,7 +71,8 @@ pub struct FlussSourceConfig { pub table_type: Option, /// `earliest` (default), `latest`, or an explicit numeric offset applied to every bucket. pub starting_offset: Option, - /// Column projection pushed down to the server. Omit to read every column. + /// Column projection pushed down to the server. Omit to read every column. An empty list or + /// a repeated name is rejected. pub columns: Option>, pub poll_interval: Option, pub poll_timeout: Option, @@ -78,6 +80,8 @@ pub struct FlussSourceConfig { /// Only `json` is accepted today. `arrow_ipc` needs the batch scanner and a different /// offset-tracking path, so it is rejected rather than quietly downgraded. pub payload_format: Option, + /// Adds the bucket, offset and timestamp of each row under the `_fluss_` prefix, so a column + /// under that prefix is rejected while it is on. pub include_metadata: Option, pub sasl_username: Option, #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")] @@ -134,6 +138,7 @@ pub struct FlussSource { /// them through the ACK handshake even without rows, so a restart before the first row does /// not resolve `latest` again and skip what was written in between. start_offsets_unsaved: AtomicBool, + rows_skipped: AtomicU64, } /// `FlussConnection` and `LogScanner` do not implement `Debug`, so the derive is replaced by @@ -196,6 +201,7 @@ impl FlussSource { state: Mutex::new(restored_state.unwrap_or_default()), pending_state: Mutex::new(None), start_offsets_unsaved: AtomicBool::new(false), + rows_skipped: AtomicU64::new(0), } } @@ -223,6 +229,18 @@ impl FlussSource { } fn validate_config(&self) -> Result { + for (field, value) in [ + ("bootstrap_servers", &self.config.bootstrap_servers), + ("database", &self.config.database), + ("table", &self.config.table), + ] { + if value.trim().is_empty() { + return Err(Error::InitError(format!( + "{field} for {CONNECTOR_NAME} must not be empty" + ))); + } + } + let table_type = self.config.table_type.as_deref().unwrap_or(LOG_TABLE_TYPE); if table_type != LOG_TABLE_TYPE { return Err(Error::InitError(format!( @@ -242,6 +260,21 @@ impl FlussSource { ))); } + if let Some(columns) = &self.config.columns { + if columns.is_empty() { + return Err(Error::InitError(format!( + "columns for {CONNECTOR_NAME} must name at least one column, or be left out to \ + read every column" + ))); + } + let mut seen = HashSet::with_capacity(columns.len()); + if let Some(duplicate) = columns.iter().find(|column| !seen.insert(column.as_str())) { + return Err(Error::InitError(format!( + "column '{duplicate}' is listed more than once in columns for {CONNECTOR_NAME}" + ))); + } + } + // The client accepts a zero limit, which makes every poll come back empty. if self.config.batch_size == Some(0) { return Err(Error::InitError(format!( @@ -254,6 +287,13 @@ impl FlussSource { "sasl_username and sasl_password for {CONNECTOR_NAME} must be set together" ))); } + // With neither a pause before the poll nor a server-side wait, an idle table would be + // polled in a tight loop. + if self.poll_interval.is_zero() && self.poll_timeout.is_zero() { + return Err(Error::InitError(format!( + "poll_interval and poll_timeout for {CONNECTOR_NAME} cannot both be zero" + ))); + } self.config .starting_offset @@ -262,40 +302,98 @@ impl FlussSource { .parse() } - /// Buckets already present in the restored state keep their offset. Everything else - /// starts at the given default, so a widened bucket count does not rewind buckets - /// that were already consumed. + /// Only the table's current buckets are subscribed. A bucket already present in the restored + /// state keeps its offset and every other one starts at the given default, so a widened + /// bucket count does not rewind buckets that were already consumed, and an offset saved for + /// a bucket the table no longer has is dropped. fn resolve_start_offsets( bucket_count: i32, start_offset: i64, tracked: &HashMap, ) -> HashMap { - let mut offsets = tracked.clone(); - for bucket in 0..bucket_count { - offsets.entry(bucket).or_insert(start_offset); - } - offsets + (0..bucket_count) + .map(|bucket| { + ( + bucket, + tracked.get(&bucket).copied().unwrap_or(start_offset), + ) + }) + .collect() } + /// Same rule as [`Self::resolve_start_offsets`], with each untracked bucket starting at its + /// current tail. + async fn resolve_latest_offsets( + &self, + connection: &FlussConnection, + bucket_count: i32, + tracked: &HashMap, + ) -> Result, Error> { + let missing: Vec = (0..bucket_count) + .filter(|bucket| !tracked.contains_key(bucket)) + .collect(); + let tails = if missing.is_empty() { + HashMap::new() + } else { + let admin = connection.get_admin().map_err(connection_error)?; + admin + .list_offsets(&self.table_path, &missing, OffsetSpec::Latest) + .await + .map_err(connection_error)? + }; + (0..bucket_count) + .map(|bucket| { + let offset = match tracked.get(&bucket) { + Some(offset) => *offset, + None => tails.get(&bucket).copied().ok_or_else(|| { + Error::InitError(format!( + "Apache Fluss returned no latest offset for bucket {bucket} of table '{}'", + self.table_path + )) + })?, + }; + Ok((bucket, offset)) + }) + .collect() + } + + /// A row that cannot be converted fails the same way on every read, so failing the batch + /// would read it again forever. It is dropped and logged with its position instead, and the + /// offsets still move past it. async fn build_batch( &self, records: &ScanRecords, ) -> Result<(Vec, Option), Error> { + let buckets = records.records_by_buckets(); let mut messages = Vec::with_capacity(records.count()); - let mut polled_offsets: HashMap = HashMap::new(); - for (bucket, bucket_records) in records.records_by_buckets() { + let mut polled_offsets = HashMap::with_capacity(buckets.len()); + let mut skipped = 0u64; + for (bucket, bucket_records) in buckets { let bucket_id = bucket.bucket_id(); for record in bucket_records { - let row = mapping::row_to_json(record.row(), &self.fields)?; - messages.push(self.build_message( - bucket_id, - record.offset(), - record.timestamp(), - row, - )?); - polled_offsets.insert(bucket_id, record.offset() + 1); + let message = mapping::row_to_json(record.row(), &self.fields).and_then(|row| { + self.build_message(bucket_id, record.offset(), record.timestamp(), row) + }); + match message { + Ok(message) => messages.push(message), + Err(error) => { + skipped += 1; + error!( + "{CONNECTOR_NAME} connector with ID: {} skipped the row at bucket \ + {bucket_id}, offset {}, which it cannot convert: {error}", + self.id, + record.offset() + ); + } + } + } + if let Some(last) = bucket_records.last() { + polled_offsets.insert(bucket_id, last.offset() + 1); } } + if skipped > 0 { + self.rows_skipped.fetch_add(skipped, Ordering::Relaxed); + } let state = self .stage_batch_state(polled_offsets, messages.len()) @@ -333,14 +431,15 @@ impl FlussSource { } /// Stages the state a polled batch would leave behind until the runtime reports its - /// result. A batch without rows changes no offset, so it carries no state unless the start - /// offsets from `open()` still have to reach disk. + /// result. A poll that read no rows changes no offset, so it carries no state unless the + /// start offsets from `open()` still have to reach disk. A poll whose rows were all skipped + /// still moves the offsets past them. async fn stage_batch_state( &self, polled_offsets: HashMap, produced: usize, ) -> Result, Error> { - if produced == 0 && !self.start_offsets_unsaved.load(Ordering::Acquire) { + if polled_offsets.is_empty() && !self.start_offsets_unsaved.load(Ordering::Acquire) { return Ok(None); } @@ -412,27 +511,33 @@ impl Source for FlussSource { ))); } - let row_type = match self.config.columns.as_ref() { - Some(columns) => table_info - .get_row_type() - .project_with_field_names(columns) - .map_err(|error| { - Error::InitError(format!( - "invalid columns projection {columns:?} for table '{}': {error}", - self.table_path - )) - })?, - None => table_info.get_row_type().clone(), + let table_row_type = table_info.get_row_type(); + let projection = self + .config + .columns + .as_deref() + .map(|columns| projection_indices(table_row_type, columns, &self.table_path)) + .transpose()?; + let projection_error = |error: fluss::error::Error| { + Error::InitError(format!( + "failed to project the columns of table '{}': {error}", + self.table_path + )) + }; + let row_type = match projection.as_deref() { + Some(indices) => table_row_type.project(indices).map_err(projection_error)?, + None => table_row_type.clone(), }; mapping::ensure_supported_types(row_type.fields())?; + if self.include_metadata { + ensure_no_metadata_collision(row_type.fields())?; + } - let scan = match self.config.columns.as_ref() { - Some(columns) => { - let names: Vec<&str> = columns.iter().map(String::as_str).collect(); - table.new_scan().project_by_name(&names).map_err(|error| { - Error::InitError(format!("failed to project columns: {error}")) - })? - } + let scan = match projection.as_deref() { + Some(indices) => table + .new_scan() + .project(indices) + .map_err(projection_error)?, None => table.new_scan(), }; let scanner = scan.create_log_scanner().map_err(|error| { @@ -446,6 +551,19 @@ impl Source for FlussSource { }; let restored = { self.state.lock().await.bucket_offsets.clone() }; + let mut stale: Vec = restored + .keys() + .copied() + .filter(|bucket| !(0..bucket_count).contains(bucket)) + .collect(); + if !stale.is_empty() { + stale.sort_unstable(); + warn!( + "{CONNECTOR_NAME} connector with ID: {} ignores the saved offsets of buckets \ + {stale:?}, which table '{}' no longer has", + self.id, self.table_path + ); + } let offsets = match start { StartingOffset::Earliest => { Self::resolve_start_offsets(bucket_count, EARLIEST_OFFSET, &restored) @@ -454,27 +572,8 @@ impl Source for FlussSource { Self::resolve_start_offsets(bucket_count, offset, &restored) } StartingOffset::Latest => { - let missing: Vec = (0..bucket_count) - .filter(|bucket| !restored.contains_key(bucket)) - .collect(); - let mut offsets = restored.clone(); - if !missing.is_empty() { - let admin = connection.get_admin().map_err(connection_error)?; - let tails = admin - .list_offsets(&self.table_path, &missing, OffsetSpec::Latest) - .await - .map_err(connection_error)?; - for bucket in missing { - let tail = tails.get(&bucket).copied().ok_or_else(|| { - Error::InitError(format!( - "Apache Fluss returned no latest offset for bucket {bucket} of table '{}'", - self.table_path - )) - })?; - offsets.insert(bucket, tail); - } - } - offsets + self.resolve_latest_offsets(&connection, bucket_count, &restored) + .await? } }; scanner @@ -587,8 +686,11 @@ impl Source for FlussSource { self.connection = None; let state = self.state.lock().await; info!( - "Closed {CONNECTOR_NAME} connector with ID: {}, total messages produced: {}", - self.id, state.messages_produced + "Closed {CONNECTOR_NAME} connector with ID: {}, total messages produced: {}, rows \ + skipped: {}", + self.id, + state.messages_produced, + self.rows_skipped.load(Ordering::Relaxed) ); Ok(()) } @@ -629,9 +731,45 @@ fn connection_error(error: fluss::error::Error) -> Error { Error::Connection(format!("Apache Fluss client failure: {error}")) } +/// Resolves the configured names once, so the row decoder and the scanner use the same +/// positions in the same order. +fn projection_indices( + row_type: &RowType, + columns: &[String], + table_path: &TablePath, +) -> Result, Error> { + columns + .iter() + .map(|column| { + row_type.get_field_index(column).ok_or_else(|| { + Error::InitError(format!( + "column '{column}' from columns does not exist in table '{table_path}'" + )) + }) + }) + .collect() +} + +/// `include_metadata` writes its fields into the same object as the columns, so a column under +/// the reserved prefix would be overwritten. +fn ensure_no_metadata_collision(fields: &[DataField]) -> Result<(), Error> { + match fields + .iter() + .find(|field| field.name().starts_with(METADATA_PREFIX)) + { + Some(field) => Err(Error::InitError(format!( + "column '{}' starts with '{METADATA_PREFIX}', which include_metadata reserves for its \ + own fields. Rename the column, leave it out of columns, or turn include_metadata off", + field.name() + ))), + None => Ok(()), + } +} + #[cfg(test)] mod tests { use super::*; + use fluss::metadata::DataTypes; fn test_config() -> FlussSourceConfig { FlussSourceConfig { @@ -1034,4 +1172,214 @@ mod tests { assert!(!source.start_offsets_unsaved.load(Ordering::Acquire)); }); } + + fn field(name: &str) -> DataField { + DataField::new(name, DataTypes::int(), None) + } + + fn json_row(columns: &[(&str, Value)]) -> serde_json::Map { + columns + .iter() + .map(|(name, value)| ((*name).to_owned(), value.clone())) + .collect() + } + + #[test] + fn given_empty_connection_setting_should_be_rejected() { + let mut blank_servers = test_config(); + blank_servers.bootstrap_servers.clear(); + let mut blank_database = test_config(); + blank_database.database = " ".to_owned(); + let mut blank_table = test_config(); + blank_table.table.clear(); + + for (setting, config) in [ + ("bootstrap_servers", blank_servers), + ("database", blank_database), + ("table", blank_table), + ] { + let source = FlussSource::new(1, config, None); + + let error = source + .validate_config() + .expect_err("An empty connection setting should be rejected"); + + assert!( + matches!(&error, Error::InitError(message) if message.contains(setting)), + "{setting}: {error:?}" + ); + } + } + + #[test] + fn given_empty_columns_should_be_rejected() { + let mut config = test_config(); + config.columns = Some(Vec::new()); + let source = FlussSource::new(1, config, None); + + let error = source + .validate_config() + .expect_err("An empty projection should be rejected"); + + assert!(matches!(error, Error::InitError(message) if message.contains("columns"))); + } + + #[test] + fn given_repeated_column_should_be_rejected() { + let mut config = test_config(); + config.columns = Some(vec!["id".to_owned(), "payload".to_owned(), "id".to_owned()]); + let source = FlussSource::new(1, config, None); + + let error = source + .validate_config() + .expect_err("A repeated column should be rejected"); + + assert!(matches!(error, Error::InitError(message) if message.contains("'id'"))); + } + + #[test] + fn given_zero_poll_interval_and_timeout_should_be_rejected() { + let mut config = test_config(); + config.poll_interval = Some("0s".to_owned()); + config.poll_timeout = Some("0s".to_owned()); + let source = FlussSource::new(1, config, None); + + let error = source + .validate_config() + .expect_err("A tight poll loop should be rejected"); + + assert!(matches!(error, Error::InitError(message) if message.contains("poll_timeout"))); + } + + #[test] + fn given_zero_poll_interval_with_a_server_wait_should_be_accepted() { + let mut config = test_config(); + config.poll_interval = Some("0s".to_owned()); + let source = FlussSource::new(1, config, None); + + assert!(source.validate_config().is_ok()); + } + + #[test] + fn given_saved_offset_for_a_bucket_the_table_no_longer_has_should_drop_it() { + let tracked = HashMap::from([(0, 42), (5, 9)]); + + let offsets = FlussSource::resolve_start_offsets(2, EARLIEST_OFFSET, &tracked); + + assert_eq!(offsets, HashMap::from([(0, 42), (1, EARLIEST_OFFSET)])); + } + + #[test] + fn given_projected_columns_should_resolve_positions_in_the_requested_order() { + let row_type = RowType::new(vec![field("id"), field("payload"), field("amount")]); + let columns = vec!["amount".to_owned(), "id".to_owned()]; + + let indices = projection_indices(&row_type, &columns, &TablePath::new("db", "events")) + .expect("Known columns should resolve"); + + assert_eq!(indices, vec![2, 0]); + } + + #[test] + fn given_unknown_projected_column_should_be_rejected() { + let row_type = RowType::new(vec![field("id")]); + let columns = vec!["missing".to_owned()]; + + let error = projection_indices(&row_type, &columns, &TablePath::new("db", "events")) + .expect_err("An unknown column should be rejected"); + + assert!(matches!(error, Error::InitError(message) if message.contains("'missing'"))); + } + + #[test] + fn given_include_metadata_should_add_bucket_offset_and_timestamp() { + let mut config = test_config(); + config.include_metadata = Some(true); + let source = FlussSource::new(1, config, None); + + let message = source + .build_message( + 2, + 41, + 1_700_000_000_123, + json_row(&[("id", Value::from(7))]), + ) + .expect("The row should build"); + + let payload: Value = + serde_json::from_slice(&message.payload).expect("The payload should be JSON"); + assert_eq!( + payload, + serde_json::json!({ + "id": 7, + "_fluss_bucket": 2, + "_fluss_offset": 41, + "_fluss_timestamp": 1_700_000_000_123_i64, + }) + ); + assert_eq!(message.id, Some(message_id(2, 41))); + assert_eq!(message.origin_timestamp, Some(1_700_000_000_123_000_000)); + } + + #[test] + fn given_metadata_left_off_should_emit_only_the_columns() { + let source = FlussSource::new(1, test_config(), None); + + let message = source + .build_message( + 2, + 41, + 1_700_000_000_123, + json_row(&[("id", Value::from(7))]), + ) + .expect("The row should build"); + + let payload: Value = + serde_json::from_slice(&message.payload).expect("The payload should be JSON"); + assert_eq!(payload, serde_json::json!({ "id": 7 })); + } + + #[test] + fn given_column_under_the_metadata_prefix_should_be_rejected() { + let fields = [field("id"), field("_fluss_offset")]; + + let error = ensure_no_metadata_collision(&fields) + .expect_err("A column under the reserved prefix should be rejected"); + + assert!(matches!(error, Error::InitError(message) if message.contains("_fluss_offset"))); + } + + #[test] + fn given_columns_outside_the_metadata_prefix_should_be_accepted() { + let fields = [field("id"), field("fluss_offset")]; + + assert!(ensure_no_metadata_collision(&fields).is_ok()); + } + + #[test] + fn metadata_fields_should_sit_under_the_reserved_prefix() { + for name in [METADATA_BUCKET, METADATA_OFFSET, METADATA_TIMESTAMP] { + assert!(name.starts_with(METADATA_PREFIX), "{name}"); + } + } + + #[test] + fn given_polled_rows_that_were_all_skipped_should_still_stage_the_offsets_past_them() { + let source = FlussSource::new(1, test_config(), None); + let runtime = tokio::runtime::Runtime::new().expect("failed to create test runtime"); + runtime.block_on(async { + *source.state.lock().await = state_with(&[(0, 42)], 42); + + let state = source + .stage_batch_state(HashMap::from([(0, 45)]), 0) + .await + .expect("A batch of skipped rows should stage"); + + let persisted = state + .and_then(|state| state.deserialize::(CONNECTOR_NAME, 1)) + .expect("Offsets past skipped rows should be staged"); + assert_eq!(persisted.bucket_offsets.get(&0), Some(&45)); + assert_eq!(persisted.messages_produced, 42); + }); + } } diff --git a/core/integration/tests/connectors/fixtures/fluss/container.rs b/core/integration/tests/connectors/fixtures/fluss/container.rs index 4ef35d8bf4..d62f727a09 100644 --- a/core/integration/tests/connectors/fixtures/fluss/container.rs +++ b/core/integration/tests/connectors/fixtures/fluss/container.rs @@ -30,6 +30,7 @@ pub(super) const ENV_SOURCE_BOOTSTRAP_SERVERS: &str = "IGGY_CONNECTORS_SOURCE_FLUSS_PLUGIN_CONFIG_BOOTSTRAP_SERVERS"; pub(super) const ENV_SOURCE_DATABASE: &str = "IGGY_CONNECTORS_SOURCE_FLUSS_PLUGIN_CONFIG_DATABASE"; pub(super) const ENV_SOURCE_TABLE: &str = "IGGY_CONNECTORS_SOURCE_FLUSS_PLUGIN_CONFIG_TABLE"; +pub(super) const ENV_SOURCE_COLUMNS: &str = "IGGY_CONNECTORS_SOURCE_FLUSS_PLUGIN_CONFIG_COLUMNS"; pub(super) const ENV_SOURCE_STARTING_OFFSET: &str = "IGGY_CONNECTORS_SOURCE_FLUSS_PLUGIN_CONFIG_STARTING_OFFSET"; pub(super) const ENV_SOURCE_POLL_INTERVAL: &str = diff --git a/core/integration/tests/connectors/fixtures/fluss/mod.rs b/core/integration/tests/connectors/fixtures/fluss/mod.rs index 57b33719cb..fb0a0f000b 100644 --- a/core/integration/tests/connectors/fixtures/fluss/mod.rs +++ b/core/integration/tests/connectors/fixtures/fluss/mod.rs @@ -20,5 +20,6 @@ mod source; pub use source::{ FlussSourceAllTypesFixture, FlussSourceFixture, FlussSourceLatestFixture, - FlussSourceSlowPollFixture, + FlussSourceProjectedFixture, FlussSourceReservedColumnFixture, FlussSourceSlowPollFixture, + FlussSourceUnconvertibleRowFixture, }; diff --git a/core/integration/tests/connectors/fixtures/fluss/source.rs b/core/integration/tests/connectors/fixtures/fluss/source.rs index 4da9b98c20..f2e3818942 100644 --- a/core/integration/tests/connectors/fixtures/fluss/source.rs +++ b/core/integration/tests/connectors/fixtures/fluss/source.rs @@ -16,10 +16,10 @@ // under the License. use super::container::{ - ENV_SOURCE_BOOTSTRAP_SERVERS, ENV_SOURCE_DATABASE, ENV_SOURCE_INCLUDE_METADATA, - ENV_SOURCE_PATH, ENV_SOURCE_POLL_INTERVAL, ENV_SOURCE_STARTING_OFFSET, - ENV_SOURCE_STREAMS_0_SCHEMA, ENV_SOURCE_STREAMS_0_STREAM, ENV_SOURCE_STREAMS_0_TOPIC, - ENV_SOURCE_TABLE, FlussContainer, + ENV_SOURCE_BOOTSTRAP_SERVERS, ENV_SOURCE_COLUMNS, ENV_SOURCE_DATABASE, + ENV_SOURCE_INCLUDE_METADATA, ENV_SOURCE_PATH, ENV_SOURCE_POLL_INTERVAL, + ENV_SOURCE_STARTING_OFFSET, ENV_SOURCE_STREAMS_0_SCHEMA, ENV_SOURCE_STREAMS_0_STREAM, + ENV_SOURCE_STREAMS_0_TOPIC, ENV_SOURCE_TABLE, FlussContainer, }; use async_trait::async_trait; use fluss::client::FlussConnection; @@ -349,6 +349,100 @@ impl TestFixture for FlussSourceAllTypesFixture { } } +/// A log table whose middle row holds a date `chrono` cannot represent, so the source can only +/// get past it by skipping that row. +pub struct FlussSourceUnconvertibleRowFixture { + inner: FlussSourceFixture, +} + +impl FlussSourceUnconvertibleRowFixture { + /// Appends rows with ids 0 to 2, where only the row with id 1 carries an unconvertible date. + pub async fn append_rows_around_an_unconvertible_date(&self) -> Result<(), TestBinaryError> { + let rows: Vec = [(0, 19_782), (1, i32::MAX), (2, 19_783)] + .into_iter() + .map(|(id, days)| { + let mut row = GenericRow::new(2); + row.set_field(0, id); + row.set_field(1, Date::new(days)); + row + }) + .collect(); + self.inner.append(&rows).await + } +} + +#[async_trait] +impl TestFixture for FlussSourceUnconvertibleRowFixture { + async fn setup() -> Result { + Ok(Self { + inner: FlussSourceFixture::start(dated_events_schema).await?, + }) + } + + fn connectors_runtime_envs(&self) -> HashMap { + self.inner.connectors_runtime_envs() + } +} + +/// A log table with a column under the `_fluss_` prefix that `include_metadata` reserves, read +/// without a projection, so the source has to refuse to start. +pub struct FlussSourceReservedColumnFixture { + _inner: FlussSourceFixture, +} + +#[async_trait] +impl TestFixture for FlussSourceReservedColumnFixture { + async fn setup() -> Result { + Ok(Self { + _inner: FlussSourceFixture::start(noted_events_schema).await?, + }) + } + + fn connectors_runtime_envs(&self) -> HashMap { + self._inner.connectors_runtime_envs() + } +} + +/// The same table read through a projection that leaves the reserved column out and names the +/// other two in reverse table order. +pub struct FlussSourceProjectedFixture { + inner: FlussSourceFixture, +} + +impl FlussSourceProjectedFixture { + /// Appends one row per payload, with the row index as its id and a note in the column the + /// projection leaves out. + pub async fn append_rows(&self, payloads: &[String]) -> Result<(), TestBinaryError> { + let rows: Vec = payloads + .iter() + .enumerate() + .map(|(index, payload)| { + let mut row = GenericRow::new(3); + row.set_field(0, index as i32); + row.set_field(1, payload.as_str()); + row.set_field(2, "left out by the projection"); + row + }) + .collect(); + self.inner.append(&rows).await + } +} + +#[async_trait] +impl TestFixture for FlussSourceProjectedFixture { + async fn setup() -> Result { + Ok(Self { + inner: FlussSourceFixture::start(noted_events_schema).await?, + }) + } + + fn connectors_runtime_envs(&self) -> HashMap { + let mut envs = self.inner.connectors_runtime_envs(); + envs.insert(ENV_SOURCE_COLUMNS.to_string(), "[payload,id]".to_string()); + envs + } +} + fn events_schema() -> fluss::error::Result { Schema::builder() .column("id", DataTypes::int()) @@ -356,6 +450,21 @@ fn events_schema() -> fluss::error::Result { .build() } +fn noted_events_schema() -> fluss::error::Result { + Schema::builder() + .column("id", DataTypes::int()) + .column("payload", DataTypes::string()) + .column("_fluss_note", DataTypes::string()) + .build() +} + +fn dated_events_schema() -> fluss::error::Result { + Schema::builder() + .column("id", DataTypes::int()) + .column("day", DataTypes::date()) + .build() +} + /// Column order matches the positions `append_sample_rows` writes. fn all_types_schema() -> fluss::error::Result { Schema::builder() diff --git a/core/integration/tests/connectors/fixtures/mod.rs b/core/integration/tests/connectors/fixtures/mod.rs index 58316baf96..dcef6e6bb0 100644 --- a/core/integration/tests/connectors/fixtures/mod.rs +++ b/core/integration/tests/connectors/fixtures/mod.rs @@ -63,7 +63,8 @@ pub use doris::{ pub use elasticsearch::{ElasticsearchSinkFixture, ElasticsearchSourcePreCreatedFixture}; pub use fluss::{ FlussSourceAllTypesFixture, FlussSourceFixture, FlussSourceLatestFixture, - FlussSourceSlowPollFixture, + FlussSourceProjectedFixture, FlussSourceReservedColumnFixture, FlussSourceSlowPollFixture, + FlussSourceUnconvertibleRowFixture, }; pub use http::{ GITHUB_ENDPOINT_ID, GITHUB_HMAC_HEADER, GITHUB_INSTANCE, HttpSinkIndividualFixture, diff --git a/core/integration/tests/connectors/fluss/fluss_source.rs b/core/integration/tests/connectors/fluss/fluss_source.rs index 1d8c4d469e..1b1f18528c 100644 --- a/core/integration/tests/connectors/fluss/fluss_source.rs +++ b/core/integration/tests/connectors/fluss/fluss_source.rs @@ -18,12 +18,15 @@ use super::{API_KEY, POLL_ATTEMPTS, POLL_INTERVAL_MS, SOURCE_KEY, STATE_FILE, TEST_ROW_COUNT}; use crate::connectors::fixtures::{ FlussSourceAllTypesFixture, FlussSourceFixture, FlussSourceLatestFixture, - FlussSourceSlowPollFixture, + FlussSourceProjectedFixture, FlussSourceReservedColumnFixture, FlussSourceSlowPollFixture, + FlussSourceUnconvertibleRowFixture, }; use iggy::prelude::IggyClient; use iggy_common::MessageClient; use iggy_common::{Consumer, Identifier, PollingStrategy}; -use iggy_connector_sdk::api::{ConnectorRuntimeStats, ConnectorStats, ConnectorStatus}; +use iggy_connector_sdk::api::{ + ConnectorRuntimeStats, ConnectorStats, ConnectorStatus, SourceInfoResponse, +}; use integration::harness::seeds; use integration::iggy_harness; use reqwest::Client; @@ -38,6 +41,7 @@ use tokio::time::{sleep, timeout}; const REDELIVERY_ATTEMPTS: usize = POLL_ATTEMPTS * 3; const SEND_FAILURE_TIMEOUT: Duration = Duration::from_secs(30); const STATE_FILE_TIMEOUT: Duration = Duration::from_secs(15); +const SOURCE_STATUS_TIMEOUT: Duration = Duration::from_secs(30); #[derive(Debug, Deserialize)] struct FlussRecord { @@ -249,6 +253,98 @@ async fn given_every_supported_column_type_should_map_each_to_json( } } +#[iggy_harness( + server(connectors_runtime(config_path = "tests/connectors/fluss/source.toml")), + seed = seeds::connector_stream +)] +async fn given_row_that_cannot_be_converted_should_skip_it_and_deliver_the_rest( + harness: &TestHarness, + fixture: FlussSourceUnconvertibleRowFixture, +) { + fixture + .append_rows_around_an_unconvertible_date() + .await + .expect("Failed to append rows"); + + let client = harness.root_client().await.unwrap(); + let received = poll_payloads(&client, 2, POLL_ATTEMPTS).await; + let delivered: HashSet = received + .iter() + .filter_map(|payload| payload["_fluss_offset"].as_i64()) + .collect(); + + assert_eq!( + delivered, + HashSet::from([0, 2]), + "Only the rows around the unconvertible one should be delivered" + ); + for payload in &received { + assert_eq!( + payload["id"], payload["_fluss_offset"], + "Each row should carry the id written at its offset" + ); + } +} + +#[iggy_harness( + server(connectors_runtime(config_path = "tests/connectors/fluss/source.toml")), + seed = seeds::connector_stream +)] +async fn given_projection_in_reverse_column_order_should_map_each_column_by_name( + harness: &TestHarness, + fixture: FlussSourceProjectedFixture, +) { + let payloads = test_payloads(); + fixture + .append_rows(&payloads) + .await + .expect("Failed to append rows"); + + let client = harness.root_client().await.unwrap(); + let received = poll_payloads(&client, payloads.len(), POLL_ATTEMPTS).await; + for (index, payload) in payloads.iter().enumerate() { + let offset = index as i64; + let row = received + .iter() + .find(|row| row["_fluss_offset"] == offset) + .unwrap_or_else(|| panic!("Row at Fluss offset {offset} was never delivered")); + assert_eq!( + row, + &json!({ + "id": index, + "payload": payload, + "_fluss_bucket": 0, + "_fluss_offset": offset, + "_fluss_timestamp": row["_fluss_timestamp"], + }), + "Only the projected columns should arrive, each under its own name" + ); + } +} + +/// The paired projection test reads the same table successfully, so the failure here comes from +/// the reserved column rather than from the table. +#[iggy_harness( + server(connectors_runtime(config_path = "tests/connectors/fluss/source.toml")), + seed = seeds::connector_stream +)] +async fn given_column_under_the_metadata_prefix_should_fail_to_start( + harness: &TestHarness, + _fixture: FlussSourceReservedColumnFixture, +) { + let api_url = harness + .connectors_runtime() + .expect("connector runtime should be available") + .http_url(); + + let source = wait_for_source_status(&Client::new(), &api_url, ConnectorStatus::Error).await; + + assert!( + source.last_error.is_some(), + "A source that failed to open should report why" + ); +} + fn test_payloads() -> Vec { (0..TEST_ROW_COUNT) .map(|index| format!("fluss-payload-{index}")) @@ -277,8 +373,11 @@ async fn poll_records( ) -> Vec { poll_payloads(client, expected_rows, attempts) .await - .into_iter() - .filter_map(|payload| serde_json::from_value(payload).ok()) + .iter() + .map(|payload| { + FlussRecord::deserialize(payload) + .unwrap_or_else(|error| panic!("Unexpected Fluss row {payload}: {error}")) + }) .collect() } @@ -304,9 +403,13 @@ async fn poll_payloads(client: &IggyClient, expected_rows: usize, attempts: usiz .await { for message in polled.messages { - if let Ok(payload) = serde_json::from_slice(&message.payload) { - received.push(payload); - } + let payload = serde_json::from_slice(&message.payload).unwrap_or_else(|error| { + panic!( + "Apache Iggy message is not JSON ({error}): {}", + String::from_utf8_lossy(&message.payload) + ) + }); + received.push(payload); } let distinct_rows: HashSet = received .iter() @@ -351,6 +454,31 @@ async fn wait_for_source_errors(http: &Client, api_url: &str, minimum_errors: u6 .expect("Apache Fluss source did not report the rejected batch"); } +async fn wait_for_source_status( + http: &Client, + api_url: &str, + status: ConnectorStatus, +) -> SourceInfoResponse { + timeout(SOURCE_STATUS_TIMEOUT, async { + loop { + if let Ok(response) = http + .get(format!("{api_url}/sources")) + .header("api-key", API_KEY) + .send() + .await + && let Ok(sources) = response.json::>().await + && let Some(source) = sources.into_iter().find(|source| source.key == SOURCE_KEY) + && source.status == status + { + return source; + } + sleep(Duration::from_millis(POLL_INTERVAL_MS)).await; + } + }) + .await + .unwrap_or_else(|_| panic!("Apache Fluss source never reached status {status:?}")) +} + async fn wait_for_file(path: &Path) { timeout(STATE_FILE_TIMEOUT, async { while !path.exists() { From f8151c17064eb88ee33ce01e91fc149a7319453c Mon Sep 17 00:00:00 2001 From: seokjin0414 Date: Fri, 2 Oct 2026 09:18:46 +0900 Subject: [PATCH 5/6] fix(connectors): stream Fluss rows to JSON and drop config Serialize The config struct derived Serialize through the exposing secret helper, so the annotation on sasl_password only looked like redaction. Nothing serializes a plugin config, so the derive is gone and serializing the password no longer compiles. Each row was collected into a serde_json map first, which copied every column name for every row. Rows are now written straight into the payload from the schema and the Arrow batch. The JSON is unchanged, checked byte for byte against the old mapping on random rows. The skipped-row count was bumped while the batch was built, so a row read again after a NACK was counted again. It now commits with the offsets on ACK. The README no longer says the runtime forwards origin_timestamp or that the password is redacted everywhere. The test fixture keeps both port listeners open until both ports are read, so the kernel cannot hand out the same port twice. Signed-off-by: seokjin0414 --- Cargo.lock | 2 - .../sources/fluss_source/Cargo.toml | 2 - .../connectors/sources/fluss_source/README.md | 6 +- .../sources/fluss_source/src/lib.rs | 139 ++++---- .../sources/fluss_source/src/mapping.rs | 307 +++++++++++------- .../connectors/fixtures/fluss/container.rs | 43 +-- 6 files changed, 308 insertions(+), 191 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 6e2cbef34f..6d116ea7ef 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -7502,13 +7502,11 @@ dependencies = [ "dashmap", "fluss-rs", "humantime", - "iggy_common", "iggy_connector_sdk", "rmp-serde", "secrecy", "serde", "serde_json", - "simd-json", "tokio", "tracing", ] diff --git a/core/connectors/sources/fluss_source/Cargo.toml b/core/connectors/sources/fluss_source/Cargo.toml index a1b9a29c8d..029a239df2 100644 --- a/core/connectors/sources/fluss_source/Cargo.toml +++ b/core/connectors/sources/fluss_source/Cargo.toml @@ -42,12 +42,10 @@ chrono = { workspace = true } dashmap = { workspace = true } fluss-rs = { workspace = true } humantime = { workspace = true } -iggy_common = { workspace = true } iggy_connector_sdk = { workspace = true } secrecy = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } -simd-json = { workspace = true } tokio = { workspace = true } tracing = { workspace = true } diff --git a/core/connectors/sources/fluss_source/README.md b/core/connectors/sources/fluss_source/README.md index e3acdfdbd2..cd67ef495e 100644 --- a/core/connectors/sources/fluss_source/README.md +++ b/core/connectors/sources/fluss_source/README.md @@ -20,7 +20,7 @@ Each Fluss row becomes one Apache Iggy message. Offsets are tracked per bucket a | `payload_format` | no | `json` | Only `json` is accepted. See [Limitations](#limitations). | | `include_metadata` | no | `false` | Adds `_fluss_bucket`, `_fluss_offset` and `_fluss_timestamp` to each JSON object. While it is on, a column whose name starts with `_fluss_` is rejected at startup, since it would be overwritten. | | `sasl_username` | no | | Enables SASL/PLAIN together with `sasl_password`. Set both or neither. | -| `sasl_password` | no | | Stored as a secret and redacted from logs and the `/stats` endpoint. | +| `sasl_password` | no | | The connector never logs it and `/stats` never shows it, but the runtime does not redact it: it logs the raw plugin config at trace level and returns it from its config API, such as `GET /sources/{key}/configs/plugin`. | | `verbose_logging` | no | `false` | Logs per-batch counts at info instead of debug. | ## Example @@ -69,7 +69,9 @@ Temporal values keep every fractional digit the column holds, so the default `TI Rows are decoded with the table schema read when the connector starts. A row written before a column was added carries `null` for it, and a column added after startup stays out of the output until the connector restarts. -Every message carries an `id` derived from its bucket and offset, so a consumer can spot a record replayed after an at-least-once redelivery (the Apache Iggy server does not deduplicate on it), and an `origin_timestamp` taken from the Fluss record timestamp. +Every message carries an `id` derived from its bucket and offset, so a consumer can spot a record replayed after an at-least-once redelivery (the Apache Iggy server does not deduplicate on it). + +The connector also hands the runtime the Fluss record timestamp as `origin_timestamp`, but the runtime does not pass it on, so Apache Iggy stamps each message with its own time. Turn on `include_metadata` to keep the record timestamp in the payload as `_fluss_timestamp`. ## Limitations diff --git a/core/connectors/sources/fluss_source/src/lib.rs b/core/connectors/sources/fluss_source/src/lib.rs index c8ed8ecec2..0d8f354a76 100644 --- a/core/connectors/sources/fluss_source/src/lib.rs +++ b/core/connectors/sources/fluss_source/src/lib.rs @@ -22,6 +22,7 @@ use fluss::client::{EARLIEST_OFFSET, FlussConnection, LogScanner}; use fluss::config::Config; use fluss::metadata::{DataField, RowType, TablePath}; use fluss::record::ScanRecords; +use fluss::row::InternalRow; use fluss::rpc::message::OffsetSpec; use iggy_connector_sdk::retry::{RetryPolicy, retry_async}; use iggy_connector_sdk::{ @@ -30,10 +31,9 @@ use iggy_connector_sdk::{ }; use secrecy::{ExposeSecret, SecretString}; use serde::{Deserialize, Serialize}; -use serde_json::Value; use std::collections::{HashMap, HashSet}; use std::str::FromStr; -use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::sync::atomic::{AtomicBool, Ordering}; use std::time::Duration; use tokio::sync::Mutex; use tokio::time::sleep; @@ -60,7 +60,9 @@ const REWIND_RETRY: RetryPolicy = RetryPolicy { max_delay: Duration::from_secs(1), }; -#[derive(Debug, Serialize, Deserialize)] +/// Deliberately not `Serialize`. The runtime hands plugin configuration over as raw JSON and +/// never serializes this struct, so leaving it out keeps `sasl_password` unserializable. +#[derive(Debug, Deserialize)] pub struct FlussSourceConfig { pub bootstrap_servers: String, pub database: String, @@ -84,7 +86,6 @@ pub struct FlussSourceConfig { /// under that prefix is rejected while it is on. pub include_metadata: Option, pub sasl_username: Option, - #[serde(serialize_with = "iggy_common::serde_secret::serialize_optional_secret")] pub sasl_password: Option, pub verbose_logging: Option, } @@ -94,6 +95,11 @@ struct State { /// Next offset to read per bucket. Absent buckets fall back to the configured start. bucket_offsets: HashMap, messages_produced: u64, + /// Unconvertible rows this run dropped, reported when the connector closes. It moves with + /// the offsets on ACK, so rows read again after a NACK are not counted twice, and it is + /// not persisted. + #[serde(skip)] + rows_skipped: u64, } #[derive(Debug, Clone, Copy)] @@ -138,7 +144,6 @@ pub struct FlussSource { /// them through the ACK handshake even without rows, so a restart before the first row does /// not resolve `latest` again and skip what was written in between. start_offsets_unsaved: AtomicBool, - rows_skipped: AtomicU64, } /// `FlussConnection` and `LogScanner` do not implement `Debug`, so the derive is replaced by @@ -201,7 +206,6 @@ impl FlussSource { state: Mutex::new(restored_state.unwrap_or_default()), pending_state: Mutex::new(None), start_offsets_unsaved: AtomicBool::new(false), - rows_skipped: AtomicU64::new(0), } } @@ -358,8 +362,9 @@ impl FlussSource { } /// A row that cannot be converted fails the same way on every read, so failing the batch - /// would read it again forever. It is dropped and logged with its position instead, and the - /// offsets still move past it. + /// would read it again forever. That includes a failed column read, since the scanner hands + /// rows over already decoded in memory. Such a row is dropped and logged with its position + /// instead, and the offsets still move past it. async fn build_batch( &self, records: &ScanRecords, @@ -367,14 +372,16 @@ impl FlussSource { let buckets = records.records_by_buckets(); let mut messages = Vec::with_capacity(records.count()); let mut polled_offsets = HashMap::with_capacity(buckets.len()); - let mut skipped = 0u64; + let mut skipped = 0; for (bucket, bucket_records) in buckets { let bucket_id = bucket.bucket_id(); for record in bucket_records { - let message = mapping::row_to_json(record.row(), &self.fields).and_then(|row| { - self.build_message(bucket_id, record.offset(), record.timestamp(), row) - }); - match message { + match self.build_message( + bucket_id, + record.offset(), + record.timestamp(), + record.row(), + ) { Ok(message) => messages.push(message), Err(error) => { skipped += 1; @@ -391,12 +398,9 @@ impl FlussSource { polled_offsets.insert(bucket_id, last.offset() + 1); } } - if skipped > 0 { - self.rows_skipped.fetch_add(skipped, Ordering::Relaxed); - } let state = self - .stage_batch_state(polled_offsets, messages.len()) + .stage_batch_state(polled_offsets, messages.len(), skipped) .await?; Ok((messages, state)) } @@ -406,19 +410,21 @@ impl FlussSource { bucket: i32, offset: i64, timestamp_millis: i64, - mut record: serde_json::Map, + row: &dyn InternalRow, ) -> Result { - if self.include_metadata { - record.insert(METADATA_BUCKET.to_owned(), Value::from(bucket)); - record.insert(METADATA_OFFSET.to_owned(), Value::from(offset)); - record.insert(METADATA_TIMESTAMP.to_owned(), Value::from(timestamp_millis)); - } + let record_metadata = [ + (METADATA_BUCKET, i64::from(bucket)), + (METADATA_OFFSET, offset), + (METADATA_TIMESTAMP, timestamp_millis), + ]; + let metadata: &[(&str, i64)] = if self.include_metadata { + &record_metadata + } else { + &[] + }; - let payload = simd_json::to_vec(&Value::Object(record)).map_err(|error| { - Error::Serialization(format!( - "failed to serialize Apache Fluss row at bucket {bucket}, offset {offset}: {error}" - )) - })?; + let payload = serde_json::to_vec(&mapping::JsonRow::new(row, &self.fields, metadata)) + .map_err(|error| Error::InvalidRecordValue(error.to_string()))?; Ok(ProducedMessage { id: Some(message_id(bucket, offset)), @@ -438,6 +444,7 @@ impl FlussSource { &self, polled_offsets: HashMap, produced: usize, + skipped: usize, ) -> Result, Error> { if polled_offsets.is_empty() && !self.start_offsets_unsaved.load(Ordering::Acquire) { return Ok(None); @@ -446,6 +453,7 @@ impl FlussSource { let mut candidate = self.state.lock().await.clone(); candidate.bucket_offsets.extend(polled_offsets); candidate.messages_produced += produced as u64; + candidate.rows_skipped += skipped as u64; let persisted = self.serialize_state(&candidate).ok_or_else(|| { Error::Serialization(format!( "failed to serialize state for {CONNECTOR_NAME} connector with ID: {}", @@ -688,9 +696,7 @@ impl Source for FlussSource { info!( "Closed {CONNECTOR_NAME} connector with ID: {}, total messages produced: {}, rows \ skipped: {}", - self.id, - state.messages_produced, - self.rows_skipped.load(Ordering::Relaxed) + self.id, state.messages_produced, state.rows_skipped ); Ok(()) } @@ -770,6 +776,8 @@ fn ensure_no_metadata_collision(fields: &[DataField]) -> Result<(), Error> { mod tests { use super::*; use fluss::metadata::DataTypes; + use fluss::row::GenericRow; + use serde_json::Value; fn test_config() -> FlussSourceConfig { FlussSourceConfig { @@ -794,6 +802,7 @@ mod tests { State { bucket_offsets: offsets.iter().copied().collect(), messages_produced: produced, + ..State::default() } } @@ -1095,7 +1104,7 @@ mod tests { *source.state.lock().await = state_with(&[(0, 42)], 42); let state = source - .stage_batch_state(HashMap::new(), 0) + .stage_batch_state(HashMap::new(), 0, 0) .await .expect("An empty batch should stage"); @@ -1113,7 +1122,7 @@ mod tests { source.start_offsets_unsaved.store(true, Ordering::Release); let state = source - .stage_batch_state(HashMap::new(), 0) + .stage_batch_state(HashMap::new(), 0, 0) .await .expect("An empty batch should stage"); @@ -1133,7 +1142,7 @@ mod tests { *source.state.lock().await = state_with(&[(0, 42), (1, 7)], 42); let state = source - .stage_batch_state(HashMap::from([(0, 45)]), 3) + .stage_batch_state(HashMap::from([(0, 45)]), 3, 0) .await .expect("A polled batch should stage"); @@ -1173,15 +1182,45 @@ mod tests { }); } + #[test] + fn given_skipped_rows_read_again_after_a_nack_should_count_them_once() { + let source = FlussSource::new(1, test_config(), None); + let runtime = tokio::runtime::Runtime::new().expect("failed to create test runtime"); + runtime.block_on(async { + source + .stage_batch_state(HashMap::from([(0, 3)]), 1, 2) + .await + .expect("A batch with skipped rows should stage"); + source + .on_batch_result(SourceBatchResult::Nack) + .await + .expect("NACK should be applied"); + assert_eq!(source.state.lock().await.rows_skipped, 0); + + source + .stage_batch_state(HashMap::from([(0, 3)]), 1, 2) + .await + .expect("The rewound batch should stage again"); + source + .on_batch_result(SourceBatchResult::Ack) + .await + .expect("ACK should be applied"); + assert_eq!(source.state.lock().await.rows_skipped, 2); + }); + } + fn field(name: &str) -> DataField { DataField::new(name, DataTypes::int(), None) } - fn json_row(columns: &[(&str, Value)]) -> serde_json::Map { - columns - .iter() - .map(|(name, value)| ((*name).to_owned(), value.clone())) - .collect() + fn source_with_id_column(include_metadata: bool) -> (FlussSource, GenericRow<'static>) { + let mut config = test_config(); + config.include_metadata = Some(include_metadata); + let mut source = FlussSource::new(1, config, None); + source.fields = vec![field("id")]; + let mut row = GenericRow::new(1); + row.set_field(0, 7i32); + (source, row) } #[test] @@ -1293,17 +1332,10 @@ mod tests { #[test] fn given_include_metadata_should_add_bucket_offset_and_timestamp() { - let mut config = test_config(); - config.include_metadata = Some(true); - let source = FlussSource::new(1, config, None); + let (source, row) = source_with_id_column(true); let message = source - .build_message( - 2, - 41, - 1_700_000_000_123, - json_row(&[("id", Value::from(7))]), - ) + .build_message(2, 41, 1_700_000_000_123, &row) .expect("The row should build"); let payload: Value = @@ -1323,15 +1355,10 @@ mod tests { #[test] fn given_metadata_left_off_should_emit_only_the_columns() { - let source = FlussSource::new(1, test_config(), None); + let (source, row) = source_with_id_column(false); let message = source - .build_message( - 2, - 41, - 1_700_000_000_123, - json_row(&[("id", Value::from(7))]), - ) + .build_message(2, 41, 1_700_000_000_123, &row) .expect("The row should build"); let payload: Value = @@ -1371,7 +1398,7 @@ mod tests { *source.state.lock().await = state_with(&[(0, 42)], 42); let state = source - .stage_batch_state(HashMap::from([(0, 45)]), 0) + .stage_batch_state(HashMap::from([(0, 45)]), 0, 0) .await .expect("A batch of skipped rows should stage"); diff --git a/core/connectors/sources/fluss_source/src/mapping.rs b/core/connectors/sources/fluss_source/src/mapping.rs index 414caa38e1..ab29004384 100644 --- a/core/connectors/sources/fluss_source/src/mapping.rs +++ b/core/connectors/sources/fluss_source/src/mapping.rs @@ -15,31 +15,62 @@ // specific language governing permissions and limitations // under the License. -use base64::Engine; +use base64::display::Base64Display; use base64::engine::general_purpose::STANDARD as BASE64; use chrono::{DateTime, NaiveDate, NaiveTime, Utc}; use fluss::metadata::{DataField, DataType}; use fluss::row::InternalRow; use iggy_connector_sdk::Error; -use serde_json::{Map, Number, Value}; +use serde::ser::{Error as _, SerializeMap}; +use serde::{Serialize, Serializer}; const MILLIS_PER_SECOND: i64 = 1_000; const NANOS_PER_MILLI: i64 = 1_000_000; +/// One row written as a JSON object straight into the payload, with the `metadata` entries +/// after the columns. Names and values are written from the schema and the Arrow batch as +/// they are read, without building a `serde_json::Value` or copying a column name per row. +/// /// Temporal values are formatted with their full fractional precision, the way the /// PostgreSQL source formats them. A number of milliseconds would drop the microseconds of /// the default `TIMESTAMP(6)`. `TIMESTAMP` carries no timezone and is written without one, /// while `TIMESTAMP_LTZ` is an instant and is written in UTC. -pub(crate) fn row_to_json( - row: &dyn InternalRow, - fields: &[DataField], -) -> Result, Error> { - let mut object = Map::with_capacity(fields.len()); - for (position, field) in fields.iter().enumerate() { - let value = read_field(row, position, field.data_type())?; - object.insert(field.name().to_owned(), value); +pub(crate) struct JsonRow<'a> { + row: &'a dyn InternalRow, + fields: &'a [DataField], + metadata: &'a [(&'a str, i64)], +} + +impl<'a> JsonRow<'a> { + pub(crate) fn new( + row: &'a dyn InternalRow, + fields: &'a [DataField], + metadata: &'a [(&'a str, i64)], + ) -> Self { + Self { + row, + fields, + metadata, + } + } +} + +impl Serialize for JsonRow<'_> { + fn serialize(&self, serializer: S) -> Result { + let mut object = serializer.serialize_map(Some(self.fields.len() + self.metadata.len()))?; + for (position, field) in self.fields.iter().enumerate() { + let value = JsonField { + row: self.row, + position, + field, + }; + object.serialize_entry(field.name(), &value)?; + } + for (name, value) in self.metadata { + object.serialize_entry(name, value)?; + } + object.end() } - Ok(object) } /// Rejects column types with no JSON representation before the first poll, so a table with @@ -64,85 +95,125 @@ fn is_supported(data_type: &DataType) -> bool { ) } -fn read_field( - row: &dyn InternalRow, +struct JsonField<'a> { + row: &'a dyn InternalRow, position: usize, - data_type: &DataType, -) -> Result { - if row.is_null_at(position).map_err(read_error)? { - return Ok(Value::Null); - } + field: &'a DataField, +} - let value = match data_type { - DataType::Boolean(_) => Value::Bool(row.get_boolean(position).map_err(read_error)?), - DataType::TinyInt(_) => Value::from(row.get_byte(position).map_err(read_error)?), - DataType::SmallInt(_) => Value::from(row.get_short(position).map_err(read_error)?), - DataType::Int(_) => Value::from(row.get_int(position).map_err(read_error)?), - DataType::BigInt(_) => Value::from(row.get_long(position).map_err(read_error)?), - DataType::Float(_) => float_value(f64::from(row.get_float(position).map_err(read_error)?)), - DataType::Double(_) => float_value(row.get_double(position).map_err(read_error)?), - DataType::Char(inner) => Value::String( - row.get_char(position, inner.length() as usize) - .map_err(read_error)? - .to_owned(), - ), - DataType::String(_) => { - Value::String(row.get_string(position).map_err(read_error)?.to_owned()) - } - DataType::Decimal(inner) => { - let decimal = row - .get_decimal(position, inner.precision() as usize, inner.scale() as usize) - .map_err(read_error)?; - Value::String(decimal.to_big_decimal().to_string()) - } - DataType::Date(_) => { - let days = row.get_date(position).map_err(read_error)?.get_inner(); - let date = NaiveDate::from_epoch_days(days).ok_or_else(|| out_of_range(data_type))?; - Value::String(date.to_string()) - } - DataType::Time(_) => { - let millis = row.get_time(position).map_err(read_error)?.get_inner(); - let time = time_of_day(millis).ok_or_else(|| out_of_range(data_type))?; - Value::String(time.to_string()) +impl Serialize for JsonField<'_> { + fn serialize(&self, serializer: S) -> Result { + let row = self.row; + let position = self.position; + let name = self.field.name(); + let data_type = self.field.data_type(); + let read_error = |error: fluss::error::Error| { + S::Error::custom(format_args!("failed to read column '{name}': {error}")) + }; + let out_of_range = || { + S::Error::custom(format_args!( + "column '{name}' holds a {data_type:?} value outside the range that can be \ + formatted" + )) + }; + + if row.is_null_at(position).map_err(read_error)? { + return serializer.serialize_none(); } - DataType::Timestamp(inner) => { - let timestamp = row - .get_timestamp_ntz(position, inner.precision()) - .map_err(read_error)?; - let instant = instant( - timestamp.get_millisecond(), - timestamp.get_nano_of_millisecond(), - ) - .ok_or_else(|| out_of_range(data_type))?; - Value::String(instant.naive_utc().to_string()) - } - DataType::TimestampLTz(inner) => { - let timestamp = row - .get_timestamp_ltz(position, inner.precision()) - .map_err(read_error)?; - let instant = instant( - timestamp.get_epoch_millisecond(), - timestamp.get_nano_of_millisecond(), - ) - .ok_or_else(|| out_of_range(data_type))?; - Value::String(instant.to_rfc3339()) - } - DataType::Bytes(_) => { - Value::String(BASE64.encode(row.get_bytes(position).map_err(read_error)?)) - } - DataType::Binary(inner) => Value::String( - BASE64.encode( - row.get_binary(position, inner.length()) + + match data_type { + DataType::Boolean(_) => { + serializer.serialize_bool(row.get_boolean(position).map_err(read_error)?) + } + DataType::TinyInt(_) => { + serializer.serialize_i8(row.get_byte(position).map_err(read_error)?) + } + DataType::SmallInt(_) => { + serializer.serialize_i16(row.get_short(position).map_err(read_error)?) + } + DataType::Int(_) => { + serializer.serialize_i32(row.get_int(position).map_err(read_error)?) + } + DataType::BigInt(_) => { + serializer.serialize_i64(row.get_long(position).map_err(read_error)?) + } + DataType::Float(_) => serialize_float( + serializer, + f64::from(row.get_float(position).map_err(read_error)?), + ), + DataType::Double(_) => { + serialize_float(serializer, row.get_double(position).map_err(read_error)?) + } + DataType::Char(inner) => serializer.serialize_str( + row.get_char(position, inner.length() as usize) .map_err(read_error)?, ), - ), - DataType::Array(_) | DataType::Map(_) | DataType::Row(_) => { - return Err(Error::SchemaMismatch(format!( - "nested type {data_type:?} is not supported by the Apache Fluss source" - ))); + DataType::String(_) => { + serializer.serialize_str(row.get_string(position).map_err(read_error)?) + } + DataType::Decimal(inner) => { + let decimal = row + .get_decimal(position, inner.precision() as usize, inner.scale() as usize) + .map_err(read_error)?; + serializer.collect_str(&decimal.to_big_decimal()) + } + DataType::Date(_) => { + let days = row.get_date(position).map_err(read_error)?.get_inner(); + serializer.collect_str(&NaiveDate::from_epoch_days(days).ok_or_else(out_of_range)?) + } + DataType::Time(_) => { + let millis = row.get_time(position).map_err(read_error)?.get_inner(); + serializer.collect_str(&time_of_day(millis).ok_or_else(out_of_range)?) + } + DataType::Timestamp(inner) => { + let timestamp = row + .get_timestamp_ntz(position, inner.precision()) + .map_err(read_error)?; + let instant = instant( + timestamp.get_millisecond(), + timestamp.get_nano_of_millisecond(), + ) + .ok_or_else(out_of_range)?; + serializer.collect_str(&instant.naive_utc()) + } + DataType::TimestampLTz(inner) => { + let timestamp = row + .get_timestamp_ltz(position, inner.precision()) + .map_err(read_error)?; + let instant = instant( + timestamp.get_epoch_millisecond(), + timestamp.get_nano_of_millisecond(), + ) + .ok_or_else(out_of_range)?; + serializer.serialize_str(&instant.to_rfc3339()) + } + DataType::Bytes(_) => serializer.collect_str(&Base64Display::new( + row.get_bytes(position).map_err(read_error)?, + &BASE64, + )), + DataType::Binary(inner) => serializer.collect_str(&Base64Display::new( + row.get_binary(position, inner.length()) + .map_err(read_error)?, + &BASE64, + )), + DataType::Array(_) | DataType::Map(_) | DataType::Row(_) => { + Err(S::Error::custom(format_args!( + "column '{name}' has nested type {data_type:?}, which the Apache Fluss \ + source does not support" + ))) + } } - }; - Ok(value) + } +} + +/// JSON has no encoding for NaN or infinity, so those collapse to null rather than +/// failing the whole batch over one degenerate float. +fn serialize_float(serializer: S, value: f64) -> Result { + if value.is_finite() { + serializer.serialize_f64(value) + } else { + serializer.serialize_none() + } } /// Fluss keeps a timestamp as epoch milliseconds plus the nanoseconds within that @@ -164,32 +235,24 @@ fn time_of_day(millis: i32) -> Option { ) } -/// JSON has no encoding for NaN or infinity, so those collapse to null rather than -/// failing the whole batch over one degenerate float. -fn float_value(value: f64) -> Value { - Number::from_f64(value).map_or(Value::Null, Value::Number) -} - -fn read_error(error: fluss::error::Error) -> Error { - Error::InvalidRecordValue(format!("failed to read Apache Fluss column: {error}")) -} - -fn out_of_range(data_type: &DataType) -> Error { - Error::InvalidRecordValue(format!( - "Apache Fluss {data_type:?} value is outside the range that can be formatted" - )) -} - #[cfg(test)] mod tests { use super::*; + use base64::Engine; use fluss::metadata::DataTypes; use fluss::row::{Date, Decimal, GenericRow, Time, TimestampLtz, TimestampNtz}; + use serde_json::Value; fn field(name: &str, data_type: DataType) -> DataField { DataField::new(name, data_type, None) } + fn to_json(row: &GenericRow, fields: &[DataField]) -> Value { + let payload = + serde_json::to_vec(&JsonRow::new(row, fields, &[])).expect("Failed to map row"); + serde_json::from_slice(&payload).expect("The payload should be JSON") + } + #[test] fn given_temporal_columns_when_mapped_should_keep_full_precision() { let fields = vec![ @@ -218,7 +281,7 @@ mod tests { .expect("Failed to build timestamp"), ); - let object = row_to_json(&row, &fields).expect("Failed to map row"); + let object = to_json(&row, &fields); assert_eq!(object["date"], Value::from("2024-02-29")); assert_eq!(object["time"], Value::from("12:34:56.789")); @@ -245,11 +308,23 @@ mod tests { TimestampNtz::from_millis_nanos(-1, 0).expect("Failed to build timestamp"), ); - let object = row_to_json(&row, &fields).expect("Failed to map row"); + let object = to_json(&row, &fields); assert_eq!(object["timestamp"], Value::from("1969-12-31 23:59:59.999")); } + #[test] + fn given_date_beyond_what_can_be_formatted_should_fail_naming_the_column() { + let fields = vec![field("shipped_on", DataTypes::date())]; + let mut row = GenericRow::new(1); + row.set_field(0, Date::new(i32::MAX)); + + let error = serde_json::to_vec(&JsonRow::new(&row, &fields, &[])) + .expect_err("A date chrono cannot hold should fail"); + + assert!(error.to_string().contains("'shipped_on'"), "{error}"); + } + #[test] fn given_decimal_and_fixed_width_columns_when_mapped_should_produce_strings() { let fields = vec![ @@ -265,7 +340,7 @@ mod tests { row.set_field(1, "abcde"); row.set_field(2, [4u8, 5, 6].as_slice()); - let object = row_to_json(&row, &fields).expect("Failed to map row"); + let object = to_json(&row, &fields); assert_eq!(object["amount"], Value::from("123.45")); assert_eq!(object["code"], Value::from("abcde")); @@ -288,7 +363,7 @@ mod tests { row.set_field(3, 1.5f64); row.set_field(4, 90i64); - let object = row_to_json(&row, &fields).expect("Failed to map row"); + let object = to_json(&row, &fields); assert_eq!(object["id"], Value::from(7)); assert_eq!(object["name"], Value::from("alice")); @@ -306,7 +381,7 @@ mod tests { let mut row = GenericRow::new(2); row.set_field(0, 1i32); - let object = row_to_json(&row, &fields).expect("Failed to map row"); + let object = to_json(&row, &fields); assert_eq!(object["id"], Value::from(1)); assert_eq!(object["name"], Value::Null); @@ -318,7 +393,7 @@ mod tests { let mut row = GenericRow::new(1); row.set_field(0, [1u8, 2, 3].as_slice()); - let object = row_to_json(&row, &fields).expect("Failed to map row"); + let object = to_json(&row, &fields); assert_eq!(object["blob"], Value::from(BASE64.encode([1u8, 2, 3]))); } @@ -347,9 +422,21 @@ mod tests { } #[test] - fn given_non_finite_float_should_map_to_null() { - assert_eq!(float_value(f64::NAN), Value::Null); - assert_eq!(float_value(f64::INFINITY), Value::Null); - assert_eq!(float_value(2.5), Value::from(2.5)); + fn given_non_finite_floats_should_map_to_null() { + let fields = vec![ + field("nan", DataTypes::double()), + field("infinite", DataTypes::float()), + field("finite", DataTypes::double()), + ]; + let mut row = GenericRow::new(3); + row.set_field(0, f64::NAN); + row.set_field(1, f32::INFINITY); + row.set_field(2, 2.5f64); + + let object = to_json(&row, &fields); + + assert_eq!(object["nan"], Value::Null); + assert_eq!(object["infinite"], Value::Null); + assert_eq!(object["finite"], Value::from(2.5)); } } diff --git a/core/integration/tests/connectors/fixtures/fluss/container.rs b/core/integration/tests/connectors/fixtures/fluss/container.rs index d62f727a09..36ca41d508 100644 --- a/core/integration/tests/connectors/fixtures/fluss/container.rs +++ b/core/integration/tests/connectors/fixtures/fluss/container.rs @@ -64,8 +64,7 @@ pub(super) struct FlussContainer { impl FlussContainer { pub(super) async fn start() -> Result { - let coordinator_port = reserve_host_port()?; - let tablet_port = reserve_host_port()?; + let [coordinator_port, tablet_port] = reserve_host_ports()?; let container = GenericImage::new(FLUSS_IMAGE, FLUSS_VERSION) .with_wait_for(WaitFor::message_on_stdout(READY_MESSAGE)) @@ -120,21 +119,27 @@ fn startup_script(tablet_port: u16) -> String { ) } -/// Binds port 0, reads back what the kernel picked, then releases it. The port is only -/// reserved by convention until the container claims it, which is the same trade every -/// fixture that needs a known port ahead of time makes. -fn reserve_host_port() -> Result { - let listener = - TcpListener::bind("127.0.0.1:0").map_err(|error| TestBinaryError::FixtureSetup { - fixture_type: "FlussContainer".to_string(), - message: format!("Failed to reserve a host port: {error}"), - })?; - let port = listener - .local_addr() - .map_err(|error| TestBinaryError::FixtureSetup { - fixture_type: "FlussContainer".to_string(), - message: format!("Failed to read the reserved host port: {error}"), - })? - .port(); - Ok(port) +/// Binds port 0 once per port, reads back what the kernel picked, then releases them all. +/// Every listener stays open until the last port is read, so the kernel cannot hand out the +/// same port twice. The ports are only reserved by convention until the container claims +/// them, which is the same trade every fixture that needs a known port ahead of time makes. +fn reserve_host_ports() -> Result<[u16; N], TestBinaryError> { + let mut listeners = Vec::with_capacity(N); + let mut ports = [0; N]; + for port in &mut ports { + let listener = + TcpListener::bind("127.0.0.1:0").map_err(|error| TestBinaryError::FixtureSetup { + fixture_type: "FlussContainer".to_string(), + message: format!("Failed to reserve a host port: {error}"), + })?; + *port = listener + .local_addr() + .map_err(|error| TestBinaryError::FixtureSetup { + fixture_type: "FlussContainer".to_string(), + message: format!("Failed to read the reserved host port: {error}"), + })? + .port(); + listeners.push(listener); + } + Ok(ports) } From 65eab7d94fb4403cbaac514d149ce7bf3afa6fc2 Mon Sep 17 00:00:00 2001 From: seokjin0414 Date: Sun, 4 Oct 2026 11:52:07 +0900 Subject: [PATCH 6/6] test(connectors): stage a skipped count in the Fluss all-skipped test The test was written before stage_batch_state took a skipped count, and the argument was later filled with 0, so it modelled three offsets advancing with nothing produced and nothing skipped. Pass the three skipped rows a real poll would report and check they reach the pending state, since rows_skipped is not serialized. Signed-off-by: seokjin0414 --- core/connectors/sources/fluss_source/src/lib.rs | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/core/connectors/sources/fluss_source/src/lib.rs b/core/connectors/sources/fluss_source/src/lib.rs index 0d8f354a76..9574202d98 100644 --- a/core/connectors/sources/fluss_source/src/lib.rs +++ b/core/connectors/sources/fluss_source/src/lib.rs @@ -1398,7 +1398,7 @@ mod tests { *source.state.lock().await = state_with(&[(0, 42)], 42); let state = source - .stage_batch_state(HashMap::from([(0, 45)]), 0, 0) + .stage_batch_state(HashMap::from([(0, 45)]), 0, 3) .await .expect("A batch of skipped rows should stage"); @@ -1407,6 +1407,13 @@ mod tests { .expect("Offsets past skipped rows should be staged"); assert_eq!(persisted.bucket_offsets.get(&0), Some(&45)); assert_eq!(persisted.messages_produced, 42); + let pending_skipped = source + .pending_state + .lock() + .await + .as_ref() + .map(|pending| pending.rows_skipped); + assert_eq!(pending_skipped, Some(3)); }); } }