Repository navigation
feat(sink): write events directly to ClickHouse - #5
Merged
Merged
Conversation
gagantrivedi
force-pushed
the
clickhouse-sink
branch
2 times, most recently
from
October 28, 2026 08:59
91ca8f6 to
828a26d
Compare
Bounds how many daily ClickHouse partitions one insert can touch, so a batch with scattered old timestamps cannot fail or overload the insert. Refs #3
SINK=clickhouse buffers accepted events in memory and writes them with one plain INSERT per flush, every SINK_FLUSH_INTERVAL_MS or earlier at SINK_BATCH_MAX_ROWS. Requests return 202 once buffered; a full buffer (SINK_BUFFER_MAX_BYTES) returns 503. Failed writes are retried, then dropped. The batching is generic so other sinks can reuse it. Refs #3
Tests the ClickHouse sink against a real server: flushed rows land in the events table and read back intact, and shutdown writes buffered rows. CI runs a clickhouse-server service for them. Refs #3
gagantrivedi
force-pushed
the
clickhouse-sink
branch
from
October 28, 2026 09:09
828a26d to
ac2af7a
Compare
EventRecord owns its fields, so BatchingSink buffers records directly and sizes them with one approx_size. BatchWriter shrinks to write(), and the ClickHouse writer builds borrowed rows at flush time. Kafka's wire bytes are unchanged. Refs #3
It configures the writer, not a sink, and reads oddly next to config::SinkConfig. Refs #3
Byte estimates of parsed JSON were guesswork. The buffer is now unbounded: put never waits or fails, and the flusher takes the whole buffer every SINK_FLUSH_INTERVAL_MS, or once SINK_FLUSH_MAX_EVENTS (5000) events are waiting, as one INSERT. This drops SINK_BATCH_MAX_ROWS, SINK_BUFFER_MAX_BYTES and SinkError::BufferFull. Refs #3
Drop comments that restate the code, inline json_column, rename value_column to value_to_text, and prefix the sink and config tests with test_. Refs #3
gagantrivedi
force-pushed
the
clickhouse-sink
branch
2 times, most recently
from
October 28, 2026 10:44
ba59c6b to
110323e
Compare
Rename parse_positive to positive_env_var and have it read the variable itself, so callers name it once. Refs #3
gagantrivedi
force-pushed
the
clickhouse-sink
branch
from
October 28, 2026 10:44
110323e to
4175491
Compare
gagantrivedi
marked this pull request as ready for review
October 28, 2026 10:52
Zaimwa9
reviewed
Oct 29, 2026
3 of 4 tasks
Wait 1, 2, 4, 5, 5 s between attempts before dropping a batch, so writes survive a ClickHouse restart instead of giving up after about 1.4 s. The delays are a fixed list; jitter is dropped, since each process has a single flusher. Refs #3
gagantrivedi
force-pushed
the
clickhouse-sink
branch
from
October 30, 2026 05:59
9cd8463 to
715df4d
Compare
…tartup Classify write errors as transient, bad data or config. Transient errors are retried; bad data and config errors drop the batch at once. A config error also fails /health until restart. At startup, check that ClickHouse is reachable and the table matches the row, so bad credentials or a missing table fail before serving. Refs #3
Zaimwa9
approved these changes
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writes events straight to ClickHouse, batched in memory and flushed on a configurable interval, instead of going through Kafka.
SinkErrorsink-agnosticSINK=clickhouse)