Document telemetry usage and configuration in DEVELOPER_GUIDE.md #5

Activity

1. syahra712 commented on Oct 17, 2026

   

   syahra712

   on Oct 17, 2026

   Contributor

   More actions

   Hi @janhoy, I started looking at this. While tracing how telemetry is wired, I found a few places where the code doesn't do what the docs say:

   • cluster-environment-info appears in solr-orbit list telemetry and in the docs, but nothing creates ClusterEnvironmentInfo. As a result, --telemetry cluster-environment-info does nothing.
   • The run pipelines (docker, from-distribution, from-sources) never pass telemetry_devices to SolrProvisioner / SolrDockerLauncher. That means jfr/gc/jit/heapdump/disk-io never turn on under run. Only the older builder/launcher.py path used by start/stop wires them up.
   • --telemetry-params collection-stats-sample-interval is documented, but nothing reads it. SolrCollectionStats always uses the 30 s default.

   A developer guide written today would have to describe this as it is. Would you prefer these fixed first, so the guide can describe one working path? If so, I'm happy to open an issue and small PRs for them, starting with cluster-environment-info. Or would you rather have the guide first, with these noted as known gaps?
2. janhoy commented on Oct 19, 2026

   

   janhoy

   on Oct 19, 2026

   ContributorAuthor

   More actions

   Yes there is some missing pieces regarding telemetry that must be cleaned up and then documented.

   Happy to get contributions. We should also solve how to get OS level telemetry (CPU, Disk IO, RAM, Network) and whether this is within the responsibility of solr-orbitd. Some of this could also be delegated to external provisioner like launchr (see other PR) but it is nice to let orbit pipeline do as much it can.
3. syahra712 commented on Oct 20, 2026

   

   syahra712

   on Oct 20, 2026

   Contributor

   More actions

   Thanks. #134 is merged, #137 is open for #136, and I'll work on #138 next. Once those land, the guide can describe one path that actually works.

   On OS-level telemetry, here's a split that I think keeps solr-orbit responsible only for what it can measure reliably:

   • Any cluster, via Solr itself. SolrNodeStats already reads /api/node/system, which exposes host CPU and memory, so this works for external clusters too with no agent. One small fix: cpu_usage_percent currently uses processCpuLoad and falls back to systemCpuLoad without saying so. Recording process_cpu_percent and system_cpu_percent as separate metrics would make runs comparable.
   • Provisioned on the same host (from-distribution / from-sources / docker). Orbit knows the Solr PID or container, so it can sample host and process CPU, RAM, disk I/O and network itself. psutil is already a dependency, and DiskIo already does the disk part per process. For Docker, the per-container numbers would come from the container stats API, so the host's other load doesn't pollute them.
   • Remote or orchestrated (launchr / EC2). Orbit can't sample a remote host well without running an agent there, so I'd delegate that to launchr. It could run node_exporter or something similar next to Solr and record the results alongside the run, and reportr would then join them by run id. solr-orbit would just document that boundary.

   If that split works for you, I'd open one issue for the separate CPU metrics plus the same-host sampler, after #138. The guide section for this issue can describe all three tiers. Does that match how you see the responsibility for solr-orbit vs. launchr?