Skip to content

Native memory requests treat an error from Spark as a refusal #7244

Open

Description

@dwsmith1983

Describe the bug

Found in #6336 (review comment).

Two native memory paths treat an error from Spark during a memory request as if the pool had refused it, so the task carries on instead of failing.

  • The multi-partition shuffle repartitioner grows its batch reservation with self.reservation.try_grow(mem_growth).is_err() and spills on any error (multi_partition.rs#L577). When a Spark-backed pool fails because another consumer's spill() threw, the repartitioner spills and keeps going, where only a ResourcesExhausted refusal should lead to a spill.
  • An infallible MemoryReservation::grow or resize on the Spark-backed pools goes through SparkMemory::acquire, which logs a failed call to Spark as a warning and books the whole size as overcommit (spark_memory.rs#L135-L151). The next try_grow asks Spark for that debt too, so a repeated exception does surface, but a single one is only logged.

Steps to reproduce

Run a native shuffle with more than one output partition on a Spark-backed pool where another memory consumer's spill() throws while the repartitioner asks for memory. The shuffle spills and finishes rather than failing with that exception.

Expected behavior

An error other than ResourcesExhausted fails the task, the way #6336 handles the shuffle writer's own buffer requests. For the infallible grow path, the exception should at least reach the task instead of only the log.

Additional context

#6336 fixes the shuffle writer's buffer requests and leaves these two paths as they are on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions