Executor is OOM-killed while Comet's pool and allocation counter stay under the limit: glibc keeps several GiB of freed native memory #7108

Describe the bug

A TPC-H SF10 run on Iceberg tables with the native Iceberg scan gets the JVM killed by the cgroup OOM killer
(exit 137). This happens in a 14 GiB container with no swap, a 4g heap and spark.memory.offHeap.size=4g. Every
figure Comet reports stays well inside the limits:

• JVM Native Memory Tracking is flat at about 4.7 GiB.
• Comet's memory log peaks at allocated 3349 MiB, reserved 3318 MiB.
• Anonymous memory outside the JVM still reaches about 9.2 GiB.
• The kernel record shows anon-rss of about 14.6 GB, with page cache already reclaimed.

Most of the difference is memory that has already been freed. Between queries Comet's live figure drops to about
2 MiB while RSS stays near 12 GiB, and jcmd <pid> System.trim_native_heap gives back 5.1 to 6.0 GiB in one call.
The release build uses the system allocator (glibc). The jemalloc and mimalloc features exist, but neither is
in the default build.

On the full 22-query suite (two warm-up rounds and one timed round, 72 queries), changing only the allocator makes
the run pass:

This is not specific to Iceberg. I ran the same Comet build on Parquet and Delta copies of the same data:

• Parquet (CometNativeScanExec) finished the full suite at an 11.5 GiB peak.
• Delta (CometDeltaNativeScanExec) finished it at 12.0 GiB.
• On both, a trim releases 3.6 to 5.1 GiB while live native memory is a few MiB.

Iceberg sits 1.5 to 2 GiB higher than the other two and is the one that crosses the limit; I'll put that part on
#6126.

Steps to reproduce

This loop runs TPC-H q09 ten times in one JVM. I reproduced it three times on the setup below.

1. Image. Start from apache/spark:4.1.3-scala2.13-java17-ubuntu and add:

   • iceberg-spark-runtime-4.1_2.13-1.12.0.jar (from Maven Central);
   • a Comet jar built from main with make release PROFILES="-Pspark-4.1";
   • tpchgen-cli 3.0.0;
   • Ubuntu's libjemalloc2, which is only for the comparison run.

2. Data. Generate SF10 with tpchgen-cli: part, supplier, partsupp and nation as one file each, orders and
   lineitem in 10 parts each. Load the six tables into an Iceberg Hadoop catalog with default write settings. Run the
   load with spark.sql.files.maxPartitionBytes=16g and spark.sql.files.openCostInBytes=16g, so that each generator
   file becomes one Iceberg data file (lineitem ends up as 10 files of about 155 MB):

   CREATE NAMESPACE IF NOT EXISTS local.tpch;
   CREATE TABLE local.tpch.lineitem USING iceberg AS SELECT * FROM parquet.`/data/gen/lineitem`;
   -- same for orders, part, partsupp, supplier, nation

3. Run. Run q09 ten times in a single spark-sql session, inside
   docker run --rm --memory 14g --memory-swap 14g --cpus 8:

   spark-sql --master 'local[8]' --driver-memory 4g \
     --conf 'spark.driver.extraJavaOptions=-Xms4g -XX:+AlwaysPreTouch' \
     --conf spark.memory.offHeap.enabled=true \
     --conf spark.memory.offHeap.size=4g \
     --conf spark.driver.extraClassPath=/opt/comet/comet-spark.jar \
     --conf spark.plugins=org.apache.spark.CometPlugin \
     --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
     --conf spark.comet.shuffle.enabled=true \
     --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
     --conf spark.sql.catalog.local.type=hadoop \
     --conf spark.sql.catalog.local.warehouse=/data/iceberg \
     -f loop.sql
   

Results:

Two conditions are needed for the kill:

• The 10-file layout. With Spark's default read splits (lineitem as 20 files), the same loop peaks at 8.6 GiB and
  passes.
• The pre-touched heap. Without it the loop peaks about 2 GiB lower and passes. Pre-touching only makes the heap
  resident from the start, which it becomes anyway in a long-running executor.

The margin under jemalloc is also small in this loop. The clearer signal is the trim: a single
System.trim_native_heap releases 5 to 6 GiB.

The Dockerfile and scripts are small. I can attach them if that helps.

Expected behavior

The executor stays inside a container sized from the heap plus spark.memory.offHeap.size plus overhead,
or Comet's memory log shows that resident memory is about to cross the container size.

Additional context

Environment:

• Comet main at 4a8a949, default cargo features.
• Spark 4.1.3, Iceberg 1.12.0, Temurin 17.0.19, glibc 2.35 (Ubuntu 22.04).
• Docker Desktop on Apple silicon (linuxkit aarch64, cgroup v2), container --memory 14g --memory-swap 14g --cpus 8,
  local[8].

Related:

• Investigate adopting DataFusion's allocator-level memory accounting to replace manual memory tuning #4576 measured the resident versus allocated gap with jemalloc. This shows what that gap looks like with the
  default glibc allocator, and that a large part of it is freed memory glibc does not return.
• Report resident memory (RssAnon) in the executor memory usage log and warn on it against the container size #6167 would make this visible: the log would have shown about 12 GiB of RssAnon next to 2 MiB allocated. Its
  warning would be one way to catch this before the kernel does.
• Tuning guide does not explain that Comet's memory comes out of the executor container #6032: the tuning guide could mention the allocator, including MALLOC_ARENA_MAX or a jemalloc build for
  container deployments, until the default changes.

Questions for the maintainers:

• Should release builds use jemalloc or mimalloc by default?
• Failing that, should the deployment docs recommend MALLOC_ARENA_MAX or preloading jemalloc?

I have per-run memory.stat samples, NMT summaries, trim results and the kernel OOM records if they would help.