Skip to content

Fix compile leaking multi-output arrays - #4183

Draft
AKnassa wants to merge 1 commit into
ml-explore:mainfrom
AKnassa:fix-compile-multi-output-leak
Draft

Fix compile leaking multi-output arrays#4183
AKnassa wants to merge 1 commit into
ml-explore:mainfrom
AKnassa:fix-compile-multi-output-leak

Conversation

@AKnassa

@AKnassa AKnassa commented Aug 11, 2026

Copy link
Copy Markdown

Proposed changes

Fixes #3932.

A compiled function containing a multi-output primitive leaks every array the
traced function captured, once per call:

def make_forward(weights):
    def forward(x):
        for w in weights:
            a, b = mx.split(x @ w, 2, axis=-1)
            x = mx.concatenate([b, a], axis=-1)
        return x
    return forward

# 8 x (256, 256) float32 weights, evaluated outside the compiled function
# before: 2097152 bytes retained per call, forever
# after:  0

Replacing the mx.split with plain slicing (a single-output op) leaks nothing,
which is what pointed at the multi-output path.

Cause

compile_dfs deep copies the traced tape, so the originals of any multi-output
node should die once the new tape and the parents map are in place. They cannot:

  • The outputs of a multi-output primitive hold each other through siblings
    (array::make_arrays), which is a strong reference cycle by construction.
  • array::~array() is the only thing that breaks that cycle, and only when the
    destructing array is the last external holder (use_count() == n + 1).
  • At tape = std::move(new_tape) the old arrays are destroyed, but the parents
    map still holds copies of them, so the count is too high and nothing is
    detached.
  • The parents map then drops its references through array::operator=, which
    reseats array_desc_ and has no cycle-breaking counterpart.

The group ends up orphaned at use_count() == 1 each, referenced only by each
other, still holding ArrayDesc::inputs — which pins the captured weights.

That also explains why all three ingredients are needed to see it: single-output
originals have no cycle, compile_dfs is the only place doing this deep copy
plus operator= rewiring, and an already-evaluated capture is shared into the
new tape rather than copied, so the orphan holds a real buffer.

Fix

Rather than adding a new liveness rule, keep one member of each replaced group
alive until both the tape and the parents map have been rewired, then drop it,
so the existing destructor test runs at a point where its arithmetic is valid.

This is deliberately conservative. If the caller still holds the group — for
example a lazy mx.split created before tracing and captured by the closure —
the reference count stays above the threshold and nothing is mutated at all. I
checked that case explicitly against main and against this branch, and both
behave identically. Nothing is force-detached, so the compile cache's record of
each node's primitive is untouched.

Also worth a line: compile_simplify's merge_one rewires parents through the
same array::operator= and split_one clones nodes, so a similar shape may
exist there. I have not investigated it and am not claiming simplify is clean.

Testing

New test_multi_output_leak in python/tests/test_compile.py, which loops so a
single-cycle fluke cannot pass it. It fails before this change and passes after.
The existing test_leaks gates on mx.metal.is_available(), so it is a no-op
on a CPU-only build; this one does not.

  • Full Python suite: 820 tests, no new failures.
  • Full C++ suite: 247 cases, all pass.

Built CPU-only (MLX_BUILD_METAL=OFF); the GPU suite was not run on my machine.

Checklist

  • I have read the CONTRIBUTING document
  • I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the necessary documentation (if needed) — no documentation change needed, this is an internal lifetime fix

compile_dfs deep copies the traced tape, so the originals of any
multi-output node are meant to die once the tape and the parents map
have been rewired. They did not: the outputs of a multi-output
primitive hold each other through siblings, and array::~array() only
breaks that cycle when it is the last external reference. Moving the
tape still left the parents map holding them, and the parents map then
released them through operator=, which has no such cycle breaking. The
group was left orphaned, holding its inputs, which pinned any array
the traced function had captured.

Keep one member of each replaced group alive until both have been
rewired, then drop it, so the destructor runs when its reference count
test is actually valid. A group the caller still holds stays above the
threshold and is left untouched.

Compiling a function with an mx.split over 8 captured 256x256 weights
retained 2 MB per call before this, and nothing after.

Fixes ml-explore#3932
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory leak when compiled function contains multi-output op

1 participant