[bwplotka]

Adds the otap-rust scenario under manifests/scenarios/otap-rust to benchmark the Rust OTAP Dataflow Engine (df_engine) scraping avalanche with urn:otel:receiver:prometheus (prometheus/scrape.rs, open-telemetry/otel-arrow#4630) and exporting via urn:otel:exporter:otlp_grpc + urn:otel:extension:gcp_auth (open-telemetry/otel-arrow#4650) to Google Cloud Telemetry (https://telemetry.googleapis.com:443).

[bwplotka]

Results

Initial results are spectacular. I couldn't believe when I saw it. Scalable metrics performance of like ~30x in memory usage. Logging, tracing and profiling pipelines could be improved as well (at least 10x).

If true, this changes a lot.

We don't need some caches, we can deprioritize various scalability work (e.g. complex VPA for DaemonSets), we can mitigate stockout problems, we effectively will save hundreds of millions of USD per year at Google, on collection alone around my space (yes, I calculated), easily $4+B across the world.

This is the evidence I and others in community needed that we should literally stop investing in Go implementations (Rust slow dev AI velocity is effectively solved) and row-based design for collections and backends (you will see some fun proposals in Prometheus and Thanos space soon!).

I am consulting those results with otel-arrow SIG. Huge thanks to otel-arrow folks and especially @lquerel for help. Thanks to @nicolastakashi for the collab on prometheus/scrape.rs#31 so far, and Saswata, Michael, Frederic, Matthias, Filip, and Jesus for inspiration for the columnar work.

Let's keep the good work 💪🏽

Disclaimer

Currently benchmarked version does NOT have a full parity. I am sending all metrics 1:1 across those scenarios, but:

• We didn't implement staleness yet. This alone takes ~101.54MB of heap in Go version, because you need scrape cache per series. There are ways to optimize this in both Go and Rust (e.g. per target staleness only) PLUS staleness is optional (we don't support it at Google now). WIP, but I think there are ways to maybe bump it +5MB max (WIP arrow: add Prometheus series staleness detection and StalenessTracker prometheus/scrape.rs#41)
• We scrape sequentially for 16 targets. This obviously is not great (not even intervals, no jitter, won't scale) - however initial concurrent versions only add max +2MB of heap too (WIP scrape-manager: add bounded concurrent and sequential scrape scheduler prometheus/scrape.rs#49)
• There are +-10 labels differences across scenarios, but I repeated with more labels and it's like +/-5MB heap difference.
• There is no relabelling yet (WIP in relabel: add Prometheus-compatible target and metric relabeling prometheus/scrape.rs#40 using DataFusion 💪🏽 )
• I am only using YOLO DNS discovery. We need to add some compute for Kubernetes API watcher, but should be negligible.
• I don't run metricstarttime processor needed for metrics without ST -- however in my setup all of them have it.
• I am not entirely sure if unit and help are parsed and passed long 🙃 type must be.

On the other hand, we are converting Otel Arrow to inefficient OTLP (row based), so with OTAP this pipeline should be even better (assuming backends adopt OTAP).

WHY SO GOOD?

• Go has GC (GOGC=100 2x heap multiplier) vs. Rust deterministic RAII deallocation in Rust
• We don't have/need scrape cache, fully stateless pipeline.
• We stream EVERYTHING into 200-point Arrow batches (20-30 KiB), including parsing (!). Scrapes are also sequential (lol):

• Zero-copy Arrow spans, everything reuses memory buffers as much as possible.
• Advanced interning (StringInterning).
• No WAL (Prometheus has persistent WAL). Again, often not needed (at Google we don't need it).
• We don't hold memory for rebuilding labels (labels.Builder) because allocs are cheap, no need for memory pooling (sync.Pool sucks so much in Go, I am so happy it's gone)

[lquerel]

Thanks, @bwplotka , for working on this integration. It's really exciting to see the impact otap/otel-arrow can have on this type of workload.