Skip to content

fix: split large appendBuffer to stop QuotaExceededError on high-bitrate segments#7753

Open
alchemyyy wants to merge 2 commits intovideo-dev:masterfrom
alchemyyy:fix/chunked-append-large-segments
Open

fix: split large appendBuffer to stop QuotaExceededError on high-bitrate segments#7753
alchemyyy wants to merge 2 commits intovideo-dev:masterfrom
alchemyyy:fix/chunked-append-large-segments

Conversation

@alchemyyy
Copy link
Copy Markdown
Contributor

This PR will...

Split large HLS fMP4 segments into smaller chunks before appending to MSE SourceBuffer, preventing QuotaExceededError with high-bitrate streams.

Why is this Pull Request needed?

When an HLS fMP4 segment exceeds Chromium's per-SourceBuffer quota (~150MB), appendBuffer() throws QuotaExceededError. With high-bitrate remuxed streams (80-150+ Mbps), a single segment can easily reach 95-100MB. At playback start there is no back buffer to evict, so hls.js enters an unrecoverable loop: download → append → QuotaExceededError → flush → retry → repeat.

This PR adds splitAppendData() to buffer-controller.ts that splits large Uint8Array buffers into ≤16MB chunks before appending. When data.byteLength exceeds 16MB, onBufferAppending splits the data and recursively creates separate append operations for each chunk — each with full error handling including QuotaExceeded recovery.

Splitting is attempted first at fMP4 top-level box boundaries. If that can't produce small-enough chunks (e.g. a single giant mdat box, which is the typical FFmpeg HLS output: styp + moof + mdat), it falls back to naive byte splitting. MSE SourceBuffer natively handles reassembly of partial box data across sequential appendBuffer() calls.

Are there any points in the code the reviewer needs to double check?

  • The fMP4 box-boundary splitting logic in splitAppendData() — verify it correctly parses top-level box headers and falls back to naive splitting when boxes exceed the chunk size
  • The recursive chunk append flow in onBufferAppending — each chunk gets its own queued operation with the same frag/part/chunkMeta context and error handling

Resolves issues:

Fixes #6711, #6776. Related: #5587, #6529.

Checklist

  • changes have been done against master branch, and PR does not conflict
  • new unit / functional tests have been added (whenever applicable)
  • API or design changes are documented in API.md

@robwalch
Copy link
Copy Markdown
Collaborator

Can you provide a sample that reproduces the issue?

@robwalch robwalch added this to the 1.7.0 milestone Mar 19, 2026
@robwalch robwalch linked an issue Mar 19, 2026 that may be closed by this pull request
5 tasks
Comment thread src/controller/buffer-controller.ts Outdated
Comment thread src/controller/buffer-controller.ts Outdated
Move fMP4 box-boundary splitting functions (`readMp4BoxSize`,
`splitAtBoxBoundaries`, `splitAppendData`) from buffer-controller to
mp4-tools where they belong.

Replace hardcoded `MAX_APPEND_SIZE` constant with a new
`config.maxAppendSize` option (default: `Infinity`) so the feature is
opt-in and the chunk size is user-configurable. Splitting only activates
when `maxAppendSize` is finite.
@alchemyyy
Copy link
Copy Markdown
Contributor Author

Can you provide a sample that reproduces the issue?

Howdy! Apologies for the delay in reply, wanted to be able to sit down with this.

I wouldn't be able to provide the test file here directly. If you'd like to reach out via email rather than source it yourself, that may work.

I can at least provide an mkvinfo dump and some general context. I came across this issue when beginning at or moving into playback around the 11 minute mark in the test file through jellyfin-web 10.11 on Brave 1.88.134. I was able to reproduce this issue consistently.

Test File Info ``` mkvinfo "Chernobyl (2019) - S01E03 - Open Wide O Earth [Bluray-2160p Remux][DTS-HD MA 5.1][HDR10][x265].mkv" + EBML head |+ EBML version: 1 |+ EBML read version: 1 |+ Maximum EBML ID length: 4 |+ Maximum EBML size length: 8 |+ Document type: matroska |+ Document type version: 4 |+ Document type read version: 2 + Segment: size 18310660753 |+ Seek head (subentries will be skipped) |+ EBML void: size 19 |+ Tracks | + Track | + Track number: 1 (track ID for mkvmerge & mkvextract: 0) | + Track UID: 10143929875395923631 | + Track type: video | + "Lacing" flag: 0 | + Codec ID: V_MPEGH/ISO/HEVC | + Language: und | + Video track | + Pixel width: 3840 | + Pixel height: 1920 | + Display width: 3840 | + Display height: 1920 | + Codec's private data: size 2682 (HEVC profile: Main 10 @L5.1) | + Default duration: 00:00:00.041708333 (23.976 frames/fields per second for a video track) | + Track | + Track number: 2 (track ID for mkvmerge & mkvextract: 1) | + Track UID: 4534071386780693852 | + Track type: audio | + Codec ID: A_DTS | + Default duration: 00:00:00.010666667 (93.750 frames/fields per second for a video track) | + Audio track | + Sampling frequency: 48000 | + Channels: 6 | + Bit depth: 24 | + Track | + Track number: 3 (track ID for mkvmerge & mkvextract: 2) | + Track UID: 5588530810768662589 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Name: English SDH | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 4 (track ID for mkvmerge & mkvextract: 3) | + Track UID: 17361439311824381604 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: fre | + Name: French | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 5 (track ID for mkvmerge & mkvextract: 4) | + Track UID: 12777215532206908821 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: spa | + Name: Spanish | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 6 (track ID for mkvmerge & mkvextract: 5) | + Track UID: 4550820835936324313 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: dut | + Name: Dutch | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 7 (track ID for mkvmerge & mkvextract: 6) | + Track UID: 3867412877195664145 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: kor | + Name: Korean | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 8 (track ID for mkvmerge & mkvextract: 7) | + Track UID: 8941450293974958817 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: spa | + Name: Spanish | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 9 (track ID for mkvmerge & mkvextract: 8) | + Track UID: 14245033150005131737 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: dan | + Name: Danish | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 10 (track ID for mkvmerge & mkvextract: 9) | + Track UID: 8107723897951080734 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: fin | + Name: Finnish | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 11 (track ID for mkvmerge & mkvextract: 10) | + Track UID: 1871957355826733938 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: nor | + Name: Norwegian | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 12 (track ID for mkvmerge & mkvextract: 11) | + Track UID: 8980550873386663009 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: swe | + Name: Swedish | + Content encodings | + Content encoding | + Content compression |+ EBML void: size 4101 |+ Segment information | + Timestamp scale: 1000000 | + Multiplexing application: libebml v1.3.9 + libmatroska v1.5.2 | + Writing application: mkvmerge v40.0.0 ('Old Town Road + Pony') 64-bit | + Duration: 01:01:47.467000000 | + Title: Open Wide, O Earth | + Segment UID: 0xd3 0xec 0x44 0xd2 0x74 0x8b 0xf8 0x62 0x5f 0x99 0xcd 0x5f 0x38 0x82 0x29 0x88 |+ EBML void: size 871 |+ Chapters | + Edition entry | + Edition flag hidden: 0 | + Edition flag default: 1 | + Edition UID: 8599732971859229138 | + Chapter atom | + Chapter UID: 6783792330602429672 | + Chapter time start: 00:00:00.000000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:00:40.123000000 | + Chapter display | + Chapter string: Chapter 01 | + Chapter language: eng | + Chapter atom | + Chapter UID: 9588704279373687150 | + Chapter time start: 00:00:40.123000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:09:12.218000000 | + Chapter display | + Chapter string: Chapter 02 | + Chapter language: eng | + Chapter atom | + Chapter UID: 10050609776139427501 | + Chapter time start: 00:09:12.218000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:18:15.303000000 | + Chapter display | + Chapter string: Chapter 03 | + Chapter language: eng | + Chapter atom | + Chapter UID: 2921876960381293319 | + Chapter time start: 00:18:15.303000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:30:07.764000000 | + Chapter display | + Chapter string: Chapter 04 | + Chapter language: eng | + Chapter atom | + Chapter UID: 16087147551117505909 | + Chapter time start: 00:30:07.764000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:38:56.918000000 | + Chapter display | + Chapter string: Chapter 05 | + Chapter language: eng | + Chapter atom | + Chapter UID: 2014500393172319452 | + Chapter time start: 00:38:56.918000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:50:33.280000000 | + Chapter display | + Chapter string: Chapter 06 | + Chapter language: eng | + Chapter atom | + Chapter UID: 13650615785365163670 | + Chapter time start: 00:50:33.280000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:58:58.702000000 | + Chapter display | + Chapter string: Chapter 07 | + Chapter language: eng | + Chapter atom | + Chapter UID: 4118494873361489949 | + Chapter time start: 00:58:58.702000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 01:01:47.203000000 | + Chapter display | + Chapter string: Chapter 08 | + Chapter language: eng | + Chapter atom | + Chapter UID: 2050106467082172595 | + Chapter time start: 01:01:47.203000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 01:01:47.467000000 | + Chapter display | + Chapter string: Chapter 09 | + Chapter language: eng |+ EBML void: size 26 |+ Cluster
</details>

@robwalch
Copy link
Copy Markdown
Collaborator

robwalch commented Apr 1, 2026

The entire segment still needs to be appended whether whole or in parts. Forcing it into the queue as multiple appends does not prevent QuotaExceeded errors. I'm seeing either corruption errors or unresolved buffer full errors with these changes.

It's unclear if these changes are intended to work on top of #7749 or replace them. Please rebased against latest - #7749 has been merged, but those changes are missing from this branch.

Comment on lines +796 to +798
this.onBufferAppending(event, {
...eventData,
data: chunks[i],
Copy link
Copy Markdown
Collaborator

@robwalch robwalch Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The BUFFER_APPENDING event data's chunkMeta should specify the chunk "id" (index) to aid in tracking append progress:

const segment: BufferAppendingData = {
type: data.type,
frag,
part,
chunkMeta,
offset,
parent: frag.type,
data: buffer,
};
this.hls.trigger(Events.BUFFER_APPENDING, segment);

    const chunkMeta = new ChunkMetadata(
      frag.level,
      frag.sn,
      i, // chunk ID should match index of split data
      payload.byteLength
    );

@robwalch robwalch removed this from the 1.7.0 milestone Apr 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

Fragments keeps reloading in a loop There is a segment in the hls video that loads many times (QuotaExceededError)

2 participants