Announcing AeroStream: Ultra-Low Latency Dual-Engine (Rust + Go) Distributed Streaming Platform (Kafka Wire Compatible)

[Uttam-Mahata]

🚀 Announcing AeroStream

I was curious about how distributed event streaming works and started reading the design and architecture of Kafka 6 months ago...

What began as a deep dive into log segments, partition replication, zero-copy socket transfers, and Raft consensus quickly turned into an obsession:

What would an event streaming platform look like if built from scratch today, combining modern mechanical sympathy with the best systems languages available?

Kafka’s wire protocol is incredible, but running heavy JVM runtimes, unpredictable GC pauses, and page cache flush stalls can introduce significant tail latency under high write loads.

So over the past 6 months, I built AeroStream — an open-source, dual-engine distributed streaming platform designed for predictable sub-millisecond tail latencies and Kafka wire protocol compatibility.

---

🛠️ Architecture Overview

AeroStream separates concerns cleanly between a bare-metal storage engine and a distributed consensus control plane:

🦀 1. The Rust Data Plane (Port 9092 & 9091)

• Shard-per-Core Model: Physical CPU thread pinning (sched_setaffinity), NUMA-aware memory allocations, and lock-free actor messaging via bounded channels.
• Kernel DMA Zero-Copy: Linux sendfile(2) eliminates userspace memory copy overhead on consumer fetches.
• Paced Writeback: Memory-mapped log segments with asynchronous sync_file_range flushes, eliminating page-cache flush latency spikes.
• Native Protocol (0xAE 0x01 on Port 9091): Ultra-compact 7-byte framing with official client SDKs for Go, Rust, Java, .NET, and Node.js.

🦫 2. The Go Control Plane (Raft Consensus)

• Raft Consensus: Embedded Raft engine with sub-150ms leader elections.
• Go Runtime: Powered by Go's Green Tea GC (<1ms pauses) and Swiss Tables for high-speed metadata, quota, and ACL lookups.
• Ecosystem Compatibility: Built-in Confluent-compatible Schema Registry (Avro, Protobuf, JSON Schema) & Kafka Connect REST engine.

---

📊 Early Benchmark Results (AeroStream Native Protocol)

The figures below were measured on the AeroStream Native Protocol (Port 9091) incorporating the paced NVMe writeback engine (sync_file_range) on a single AWS c6id.2xlarge (8 vCPUs), 32 partitions, 8 producers and 8 consumers with 1 KB messages:

Workload	p50	p95	p99	p99.9	Broker CPU
100k msg/s	0.7 ms	1.2 ms	1.3 ms	1.8 ms	32%
200k msg/s	0.8 ms	1.3 ms	1.5 ms	3.8 ms	42%
Peak Throughput	—	—	149 ms	—	67% (287,428 msg/s)

📌 Note: These figures reflect the native binary protocol. Comprehensive comparative benchmarks evaluating AeroStream Native Protocol vs. AeroStream Kafka Wire Protocol will be published in upcoming documentation updates as edge-case testing concludes.

---

🔌 Kafka Wire Protocol Support (Port 9092)

Supports standard Kafka wire protocol for producing and consuming (tested with core client workflows):

• Python: confluent-kafka, kafka-python-ng
• Go: IBM/sarama, twmb/franz-go
• Java: org.apache.kafka:kafka-clients, Kafka Streams
• Node.js: kafkajs
• .NET: Confluent.Kafka

---

☁️ Multi-Cloud Tiered Storage

Seamlessly offload cold sealed segments to object storage with transparent client reads:

• AWS S3 & MinIO
• Google Cloud Storage (GCS)
• Azure Blob Storage

---

🚀 Quickstart (Run in 5 Seconds)

docker run -d \
  --name aerostream \
  -p 9092:9092 \
  -p 9091:9091 \
  -p 9001:9001 \
  quay.io/gradientgeeks/aerostream:latest

• Kafka Wire Port: 127.0.0.1:9092
• Native Protocol Port: 127.0.0.1:9091
• Web Console & REST API: http://127.0.0.1:9001/aerostream/console/

---

💬 Community, Feedback & Status

AeroStream is actively in development.

• 🐛 Issues & Bugs: If you run into any quirks or bugs, please open an Issue.
• 🤝 Contributions: PRs, feature ideas, and performance insights are warmly welcomed!
• ⭐️ Star the Repo: If you like what we're building, please consider starring the repository — it helps the project gain visibility.

🔗 Documentation & Showcase: https://aerostream.gradientgeeks.com
📦 SDK Repository: github.com/gradientgeeks/aerostream-sdk

We'd love to hear your thoughts, feedback, and architecture critiques!