Skip to content

feat(sink): write events directly to PostgreSQL - #6

Open
Zaimwa9 wants to merge 3 commits into
clickhouse-sinkfrom
feat/postgres-sink
Open

Zaimwa9 wants to merge 3 commits into
clickhouse-sinkfrom
feat/postgres-sink

Conversation

@Zaimwa9

@Zaimwa9 Zaimwa9 commented Oct 29, 2026 •

Copy link
Copy Markdown

Thanks for submitting a PR! Please check the boxes below:

  • I have read the Contributing Guide.
  • I have added information to docs/ if required so people know about the feature.
  • I have filled in the "Changes" section below.
  • I have filled in the "How did you test this code" section below.

Changes

Contributes to Flagsmith/flagsmith#8857

Stacked on #5. Adds a SINK=postgres sink that writes events directly to flagsmith_exp.events in Flagsmith's own PostgreSQL (the table created by Flagsmith/flagsmith#8857), so self-hosted installs need neither Kafka nor ClickHouse.

  • PostgresWriter plugs into feat(sink): write events directly to ClickHouse #5's BatchingSink: same buffering, flush interval, retries and drain on shutdown.
  • Each batch is one INSERT ... SELECT * FROM UNNEST(...) (one array parameter per column) in a transaction with SET LOCAL statement_timeout, plus a 30s client-side timeout for stalled connections.
  • Reuses the existing DATABASE_URL pool; no new env vars or crates.
  • Row mapping matches warehouse-delivery: absent optionals are NULL (identifier is ''), JSON values as compact text, NUL bytes stripped.
  • Errors log only the server message, not DETAIL, so no event data reaches the logs.
  • README: SINK=postgres and the grants a non-owner role needs.

How did you test this code?

  • #[sqlx::test] integration test: buffers events, shuts the sink down, and compares every stored column, including NULLs, '', NUL stripping and a non-string value.
  • End-to-end: ran the server with SINK=postgres against a local Flagsmith database and confirmed ingested events landed in flagsmith_exp.events.

@Zaimwa9

Zaimwa9 commented Oct 29, 2026

Copy link
Copy Markdown
Author

@themis-blindfold review

Comment thread src/sink/postgres.rs Resolved
Comment thread src/sink/postgres.rs Resolved
@themis-blindfold

Copy link
Copy Markdown

⚖️ Themis review: 🟠 Fix before merge

The PostgreSQL sink is wired consistently with the existing batching flow, but it silently changes accepted NUL-containing input and cannot safely retry an ambiguous commit. The recorded CI status is passing, although the only completed check was a neutral PR-title validation rather than a test run.

Area Score
🎯 Correctness 2/5
🧪 Test coverage 3/5
📐 Code quality 3/5
🚀 Product impact 4/5

🟠 Majors

  • src/sink/postgres.rs:71 — accepted NUL bytes are removed before storage, collapsing distinct analytics dimensions.
  • src/sink/postgres.rs:127 — an ambiguous commit acknowledgement can retry the same batch without deduplication (unverified).
📝 Walkthrough
  • configuration - adds SINK=postgres and reuses the batching configuration.
  • application wiring - shares the existing PostgreSQL pool between environment-key lookups and the new sink.
  • PostgreSQL writer - converts event records into typed column arrays and inserts a batch in one transaction.
  • documentation and tests - documents setup/permissions and covers a successful buffered shutdown flush.
🧪 How to verify
  1. POST a batch containing \u0000 in event, feature_name, identifier, and a string value; verify only those events are rejected and no stored dimension is rewritten.
  2. Inject a connection loss after PostgreSQL applies COMMIT but before the client receives its acknowledgement; verify the retry cannot create duplicate rows.
  3. Run cargo fmt --all -- --check && cargo clippy --all-targets -- -D warnings && cargo test --all with the configured PostgreSQL and ClickHouse services.

Automate: add an integration test for per-event NUL rejection and a writer-level test that proves ambiguous commit failures are idempotent.

Product take: Direct PostgreSQL ingestion is a meaningful deployment simplification for self-hosted experimentation. The two integrity gaps would make analytics results unreliable under malformed input or ordinary network faults.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

PostgreSQL gets the fast lane; it still needs a ticket checker · reviewed at e40f074

@Zaimwa9
Zaimwa9 marked this pull request as ready for review October 29, 2026 16:25
gagantrivedi and others added 2 commits October 30, 2026 13:39
* feat(ingest): reject events older than 10 days

Bounds how many daily ClickHouse partitions one insert can touch, so a
batch with scattered old timestamps cannot fail or overload the insert.

Refs #3

* feat(sink): write events to ClickHouse in batches

SINK=clickhouse buffers accepted events in memory and writes them with
one plain INSERT per flush, every SINK_FLUSH_INTERVAL_MS or earlier at
SINK_BATCH_MAX_ROWS. Requests return 202 once buffered; a full buffer
(SINK_BUFFER_MAX_BYTES) returns 503. Failed writes are retried, then
dropped. The batching is generic so other sinks can reuse it.

Refs #3

* test(sink): ClickHouse integration tests

Tests the ClickHouse sink against a real server: flushed rows land in
the events table and read back intact, and shutdown writes buffered
rows. CI runs a clickhouse-server service for them.

Refs #3

* refactor(sink): buffer owned EventRecords

EventRecord owns its fields, so BatchingSink buffers records directly and
sizes them with one approx_size. BatchWriter shrinks to write(), and the
ClickHouse writer builds borrowed rows at flush time. Kafka's wire bytes
are unchanged.

Refs #3

* refactor(sink): rename ClickHouseSinkConfig to ClickHouseConfig

It configures the writer, not a sink, and reads oddly next to
config::SinkConfig.

Refs #3

* refactor(sink): flush the ClickHouse buffer by event count

Byte estimates of parsed JSON were guesswork. The buffer is now
unbounded: put never waits or fails, and the flusher takes the whole
buffer every SINK_FLUSH_INTERVAL_MS, or once SINK_FLUSH_MAX_EVENTS
(5000) events are waiting, as one INSERT. This drops
SINK_BATCH_MAX_ROWS, SINK_BUFFER_MAX_BYTES and SinkError::BufferFull.

Refs #3

* refactor(sink): trim comments and tidy test names

Drop comments that restate the code, inline json_column, rename value_column to value_to_text, and prefix the sink and config tests with test_.

Refs #3

* refactor(config): read positive env vars in positive_env_var

Rename parse_positive to positive_env_var and have it read the variable itself, so callers name it once.

Refs #3

* refactor(config): name env helpers after the config they build

Refs #3

* feat(sink): retry ClickHouse writes for about 17 seconds

Wait 1, 2, 4, 5, 5 s between attempts before dropping a batch, so
writes survive a ClickHouse restart instead of giving up after about
1.4 s. The delays are a fixed list; jitter is dropped, since each
process has a single flusher.

Refs #3

* feat(sink): handle ClickHouse errors by kind and check the table at startup

Classify write errors as transient, bad data or config. Transient errors
are retried; bad data and config errors drop the batch at once. A config
error also fails /health until restart. At startup, check that ClickHouse
is reachable and the table matches the row, so bad credentials or a
missing table fail before serving.

Refs #3

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants