Repository navigation
Conversation
|
@themis-blindfold review |
⚖️ Themis review: 🟠 Fix before mergeThe PostgreSQL sink is wired consistently with the existing batching flow, but it silently changes accepted NUL-containing input and cannot safely retry an ambiguous commit. The recorded CI status is passing, although the only completed check was a neutral PR-title validation rather than a test run.
🟠 Majors
📝 Walkthrough
🧪 How to verify
Automate: add an integration test for per-event NUL rejection and a writer-level test that proves ambiguous commit failures are idempotent. Product take: Direct PostgreSQL ingestion is a meaningful deployment simplification for self-hosted experimentation. The two integrity gaps would make analytics results unreliable under malformed input or ordinary network faults. 🧭 Assumptions & unverified claimsNo unverified assumptions or claims. PostgreSQL gets the fast lane; it still needs a ticket checker · reviewed at e40f074 |
* feat(ingest): reject events older than 10 days Bounds how many daily ClickHouse partitions one insert can touch, so a batch with scattered old timestamps cannot fail or overload the insert. Refs #3 * feat(sink): write events to ClickHouse in batches SINK=clickhouse buffers accepted events in memory and writes them with one plain INSERT per flush, every SINK_FLUSH_INTERVAL_MS or earlier at SINK_BATCH_MAX_ROWS. Requests return 202 once buffered; a full buffer (SINK_BUFFER_MAX_BYTES) returns 503. Failed writes are retried, then dropped. The batching is generic so other sinks can reuse it. Refs #3 * test(sink): ClickHouse integration tests Tests the ClickHouse sink against a real server: flushed rows land in the events table and read back intact, and shutdown writes buffered rows. CI runs a clickhouse-server service for them. Refs #3 * refactor(sink): buffer owned EventRecords EventRecord owns its fields, so BatchingSink buffers records directly and sizes them with one approx_size. BatchWriter shrinks to write(), and the ClickHouse writer builds borrowed rows at flush time. Kafka's wire bytes are unchanged. Refs #3 * refactor(sink): rename ClickHouseSinkConfig to ClickHouseConfig It configures the writer, not a sink, and reads oddly next to config::SinkConfig. Refs #3 * refactor(sink): flush the ClickHouse buffer by event count Byte estimates of parsed JSON were guesswork. The buffer is now unbounded: put never waits or fails, and the flusher takes the whole buffer every SINK_FLUSH_INTERVAL_MS, or once SINK_FLUSH_MAX_EVENTS (5000) events are waiting, as one INSERT. This drops SINK_BATCH_MAX_ROWS, SINK_BUFFER_MAX_BYTES and SinkError::BufferFull. Refs #3 * refactor(sink): trim comments and tidy test names Drop comments that restate the code, inline json_column, rename value_column to value_to_text, and prefix the sink and config tests with test_. Refs #3 * refactor(config): read positive env vars in positive_env_var Rename parse_positive to positive_env_var and have it read the variable itself, so callers name it once. Refs #3 * refactor(config): name env helpers after the config they build Refs #3 * feat(sink): retry ClickHouse writes for about 17 seconds Wait 1, 2, 4, 5, 5 s between attempts before dropping a batch, so writes survive a ClickHouse restart instead of giving up after about 1.4 s. The delays are a fixed list; jitter is dropped, since each process has a single flusher. Refs #3 * feat(sink): handle ClickHouse errors by kind and check the table at startup Classify write errors as transient, bad data or config. Transient errors are retried; bad data and config errors drop the batch at once. A config error also fails /health until restart. At startup, check that ClickHouse is reachable and the table matches the row, so bad credentials or a missing table fail before serving. Refs #3
Thanks for submitting a PR! Please check the boxes below:
docs/if required so people know about the feature.Changes
Contributes to Flagsmith/flagsmith#8857
Stacked on #5. Adds a
SINK=postgressink that writes events directly toflagsmith_exp.eventsin Flagsmith's own PostgreSQL (the table created by Flagsmith/flagsmith#8857), so self-hosted installs need neither Kafka nor ClickHouse.PostgresWriterplugs into feat(sink): write events directly to ClickHouse #5'sBatchingSink: same buffering, flush interval, retries and drain on shutdown.INSERT ... SELECT * FROM UNNEST(...)(one array parameter per column) in a transaction withSET LOCAL statement_timeout, plus a 30s client-side timeout for stalled connections.DATABASE_URLpool; no new env vars or crates.NULL(identifieris''), JSON values as compact text, NUL bytes stripped.DETAIL, so no event data reaches the logs.SINK=postgresand the grants a non-owner role needs.How did you test this code?
#[sqlx::test]integration test: buffers events, shuts the sink down, and compares every stored column, includingNULLs,'', NUL stripping and a non-stringvalue.SINK=postgresagainst a local Flagsmith database and confirmed ingested events landed inflagsmith_exp.events.