--telemetry jfr/gc/jit/heapdump never reach Solr on provisioned pipelines #138

Activity

1. janhoy commented on Oct 19, 2026

   

   janhoy

   on Oct 19, 2026

   Contributor

   More actions

   Thanks for this, it’s complex stuff and as you describe the issue it makes sense in my head even if I have not spent lots of time on this other than obviously dropping the ball too early during the solr port work.

   If you can provide solution to this which is backed by tests, that would be fantastic.
2. syahra712 commented on Oct 20, 2026

   

   syahra712

   on Oct 20, 2026

   ContributorAuthor

   More actions

   Thanks, I'll take this one. Here's the plan, so you can redirect me before I write it:

   1. Build the JVM devices (jfr, gc, jit, heapdump, disk-io) once in test_run_orchestrator.py from telemetry.devices / telemetry.params, using the same factory the start/stop path in builder/launcher.py uses. Then pass them to SolrProvisioner and SolrDockerLauncher in all three pipelines.
   2. Docker: when a JVM device is enabled, bind-mount the log directory into the container at the same path, and make sure uid 8983 can write to it. Then gc.log / profile.jfr survive docker rm -f.
   3. heapdump: run the dump before the container or process is stopped, not after.
   4. Tests: unit tests that each pipeline passes the devices through and that SOLR_OPTS gets the flags. For Docker, a test that checks the -v mount and the paths in the flags. If you want, I can also add an it/ case that checks gc.log exists after a short docker run.

   I'll split it into two PRs: wiring plus from-distribution/from-sources first, then Docker mounts plus heapdump. That keeps each one reviewable. Does that work for you?
3. janhoy commented on Oct 21, 2026

   

   janhoy

   on Oct 21, 2026

   Contributor

   More actions

   Sounds like a plan.

   Wrt bind-mount, there can be all kinds of issues with write access to disk on various systems (Mac, Linux, Windows, CI system, WSL2 etc). Before we choose to go into that mine field, can we explore our options?

   One option is to create a named docker volume and ask the container to use it. Then the provisioner will have a chance to inspect the volume after container stops, and also dispose of it once done.
4. syahra712 commented on Oct 21, 2026

   

   syahra712

   on Oct 21, 2026

   ContributorAuthor

   More actions

   Good call. I tried the options locally (Docker Desktop on macOS, solr:9, JDK 17) instead of guessing:

   	How files get out	What broke
   Bind mount	Directly on the host	Worked on macOS. With a host dir owned by uid 1000 and Solr running as 8983, which is what native Linux/CI gives you, writes fail with Permission denied. On top of that, SELinux (:z) and Windows path translation each need their own handling.
   Named volume at /var/solr/logs	Still needs docker cp or a helper container, since on Mac/Windows the volume lives inside the VM	/var/solr/logs doesn't exist in the image, so Docker creates the empty volume root-owned. Solr then refuses to start with Logs directory /var/solr/logs is not writable. You can work around it, but it needs a chown helper plus volume cleanup, including after crashes.
   No volume: docker stop, then docker cp, then docker rm	docker cp from the stopped container	Nothing broke. Solr shuts down in about 1 s, and the JFR is flushed on exit (profile.jfr goes from 0 B to 1 MB after the stop). The copied files belong to the user running solr-orbit, not 8983, so there are no host permission issues.

   So I'd go with the last one. It never writes to a host path from inside the container, and it doesn't need any volume lifecycle. The current launcher uses docker run --rm plus docker rm -f, so both of those change: drop --rm, and make stop() do docker stop, docker cp <name>:/var/solr/logs <test-run>/telemetry/, docker rm. start() already does rm -f on a stale container with the same name, so a crashed run doesn't block the next one. A side benefit is that solr.log gets saved with the run as well.

   Two findings that affect heapdump on Docker:

   • solr:9 is a JRE image, with no jmap or jcmd, so the current docker exec ... jmap can't work there.
   • node.pid comes from docker inspect .State.Pid, which is the PID on the Docker host or VM (1324 in my run). Inside the container the JVM was PID 14.

   For Docker I'd look for jcmd in the container. If it's there, use it with the in-container PID. If not, log one clear warning instead of a silent failure.

   PR split:

   1. Pass jfr, gc and jit telemetry devices to the provisioned pipelines #143: jfr/gc/jit for from-distribution and from-sources, which doesn't touch Docker storage. I kept Docker out on purpose. Today the flags would point at host paths, and the JVM in the container exits on startup (Error opening log file ... Could not create the Java Virtual Machine).
   2. Next: Docker via stop/cp/rm as above, plus heapdump before stop for all three pipelines. I'll pick this up once you're OK with the approach.
5. janhoy commented on Oct 22, 2026

   

   janhoy

   on Oct 22, 2026

   Contributor

   More actions

   solr:9 is a JRE image, with no jmap or jcmd, so the current docker exec ... jmap can't work there.

   This is not correct, as all our docker images include jattach.

   Using docker cp to pull logs from the stopped container is probably a viable and portable solution.