Skip to content

vae: a decode graph with no room on the device is decoded again in smaller tiles - #29

Open
timoncool wants to merge 1 commit into
ServeurpersoCom:masterfrom
timoncool:upstream-vae-tile-retry
Open

timoncool wants to merge 1 commit into
ServeurpersoCom:masterfrom
timoncool:upstream-vae-tile-retry

Conversation

@timoncool

Copy link
Copy Markdown

On a card that is short of memory the VAE decode either failed the song ("graph alloc failed") or, on
Windows with the NVIDIA driver's default, spilled into system memory and ran many times slower without any
error.

Now, before a tile's graph is allocated, it is measured on a scheduler of its own
(ggml_backend_sched_reserve_size) and compared with the device's free memory; on a GPU it leaves 768 MiB for
what the desktop, a browser and the CUDA context hold, because the driver reports a process only its own room.
A graph that does not fit — or an allocation that still fails — returns VAE_GRAPH_NO_FIT, and
vae_ggml_decode_tiled starts the whole decode over in tiles half the size, down to 64 frames, before it
gives up with an out-of-memory message. Errors and cancels are not retried. Memory the scheduler already holds
for an earlier tile counts as available. The public signature of vae_ggml_decode_tiled is unchanged.

In the engine of YuE2 Studio 4.1.0 (our fork, commits 20364d8 and 67f59c1): on an RTX 4090 every song decodes through the fitting path. The retry itself was not exercised on a 4-6 GB card, we have none.

Ported onto master by hand from our fork, where it runs in YuE2 Studio 4.1.0; not built on master on our side.

…aller tiles

The graph of a tile is measured on a scheduler of its own before anything is
allocated (ggml_backend_sched_reserve_size) and set against the device's free
memory; on a card it leaves 768 MiB for what the desktop, a browser and the
CUDA context hold, since the driver tells a process only its own room. A
graph that does not fit, or an allocation that fails, makes the whole decode
start over in tiles half the size, down to 64 frames, instead of failing the
song or, on Windows, spilling into the computer's memory and running many
times slower. An error or a cancel is not retried.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant