fix: split large appendBuffer to stop QuotaExceededError on high-bitrate segments#7753
fix: split large appendBuffer to stop QuotaExceededError on high-bitrate segments#7753alchemyyy wants to merge 2 commits intovideo-dev:masterfrom
Conversation
…high-bitrate segments
|
Can you provide a sample that reproduces the issue? |
Move fMP4 box-boundary splitting functions (`readMp4BoxSize`, `splitAtBoxBoundaries`, `splitAppendData`) from buffer-controller to mp4-tools where they belong. Replace hardcoded `MAX_APPEND_SIZE` constant with a new `config.maxAppendSize` option (default: `Infinity`) so the feature is opt-in and the chunk size is user-configurable. Splitting only activates when `maxAppendSize` is finite.
Howdy! Apologies for the delay in reply, wanted to be able to sit down with this. I wouldn't be able to provide the test file here directly. If you'd like to reach out via email rather than source it yourself, that may work. I can at least provide an mkvinfo dump and some general context. I came across this issue when beginning at or moving into playback around the 11 minute mark in the test file through jellyfin-web 10.11 on Brave 1.88.134. I was able to reproduce this issue consistently. Test File Info``` mkvinfo "Chernobyl (2019) - S01E03 - Open Wide O Earth [Bluray-2160p Remux][DTS-HD MA 5.1][HDR10][x265].mkv" + EBML head |+ EBML version: 1 |+ EBML read version: 1 |+ Maximum EBML ID length: 4 |+ Maximum EBML size length: 8 |+ Document type: matroska |+ Document type version: 4 |+ Document type read version: 2 + Segment: size 18310660753 |+ Seek head (subentries will be skipped) |+ EBML void: size 19 |+ Tracks | + Track | + Track number: 1 (track ID for mkvmerge & mkvextract: 0) | + Track UID: 10143929875395923631 | + Track type: video | + "Lacing" flag: 0 | + Codec ID: V_MPEGH/ISO/HEVC | + Language: und | + Video track | + Pixel width: 3840 | + Pixel height: 1920 | + Display width: 3840 | + Display height: 1920 | + Codec's private data: size 2682 (HEVC profile: Main 10 @L5.1) | + Default duration: 00:00:00.041708333 (23.976 frames/fields per second for a video track) | + Track | + Track number: 2 (track ID for mkvmerge & mkvextract: 1) | + Track UID: 4534071386780693852 | + Track type: audio | + Codec ID: A_DTS | + Default duration: 00:00:00.010666667 (93.750 frames/fields per second for a video track) | + Audio track | + Sampling frequency: 48000 | + Channels: 6 | + Bit depth: 24 | + Track | + Track number: 3 (track ID for mkvmerge & mkvextract: 2) | + Track UID: 5588530810768662589 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Name: English SDH | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 4 (track ID for mkvmerge & mkvextract: 3) | + Track UID: 17361439311824381604 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: fre | + Name: French | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 5 (track ID for mkvmerge & mkvextract: 4) | + Track UID: 12777215532206908821 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: spa | + Name: Spanish | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 6 (track ID for mkvmerge & mkvextract: 5) | + Track UID: 4550820835936324313 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: dut | + Name: Dutch | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 7 (track ID for mkvmerge & mkvextract: 6) | + Track UID: 3867412877195664145 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: kor | + Name: Korean | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 8 (track ID for mkvmerge & mkvextract: 7) | + Track UID: 8941450293974958817 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: spa | + Name: Spanish | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 9 (track ID for mkvmerge & mkvextract: 8) | + Track UID: 14245033150005131737 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: dan | + Name: Danish | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 10 (track ID for mkvmerge & mkvextract: 9) | + Track UID: 8107723897951080734 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: fin | + Name: Finnish | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 11 (track ID for mkvmerge & mkvextract: 10) | + Track UID: 1871957355826733938 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: nor | + Name: Norwegian | + Content encodings | + Content encoding | + Content compression | + Track | + Track number: 12 (track ID for mkvmerge & mkvextract: 11) | + Track UID: 8980550873386663009 | + Track type: subtitles | + "Default track" flag: 0 | + "Lacing" flag: 0 | + Codec ID: S_HDMV/PGS | + Language: swe | + Name: Swedish | + Content encodings | + Content encoding | + Content compression |+ EBML void: size 4101 |+ Segment information | + Timestamp scale: 1000000 | + Multiplexing application: libebml v1.3.9 + libmatroska v1.5.2 | + Writing application: mkvmerge v40.0.0 ('Old Town Road + Pony') 64-bit | + Duration: 01:01:47.467000000 | + Title: Open Wide, O Earth | + Segment UID: 0xd3 0xec 0x44 0xd2 0x74 0x8b 0xf8 0x62 0x5f 0x99 0xcd 0x5f 0x38 0x82 0x29 0x88 |+ EBML void: size 871 |+ Chapters | + Edition entry | + Edition flag hidden: 0 | + Edition flag default: 1 | + Edition UID: 8599732971859229138 | + Chapter atom | + Chapter UID: 6783792330602429672 | + Chapter time start: 00:00:00.000000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:00:40.123000000 | + Chapter display | + Chapter string: Chapter 01 | + Chapter language: eng | + Chapter atom | + Chapter UID: 9588704279373687150 | + Chapter time start: 00:00:40.123000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:09:12.218000000 | + Chapter display | + Chapter string: Chapter 02 | + Chapter language: eng | + Chapter atom | + Chapter UID: 10050609776139427501 | + Chapter time start: 00:09:12.218000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:18:15.303000000 | + Chapter display | + Chapter string: Chapter 03 | + Chapter language: eng | + Chapter atom | + Chapter UID: 2921876960381293319 | + Chapter time start: 00:18:15.303000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:30:07.764000000 | + Chapter display | + Chapter string: Chapter 04 | + Chapter language: eng | + Chapter atom | + Chapter UID: 16087147551117505909 | + Chapter time start: 00:30:07.764000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:38:56.918000000 | + Chapter display | + Chapter string: Chapter 05 | + Chapter language: eng | + Chapter atom | + Chapter UID: 2014500393172319452 | + Chapter time start: 00:38:56.918000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:50:33.280000000 | + Chapter display | + Chapter string: Chapter 06 | + Chapter language: eng | + Chapter atom | + Chapter UID: 13650615785365163670 | + Chapter time start: 00:50:33.280000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 00:58:58.702000000 | + Chapter display | + Chapter string: Chapter 07 | + Chapter language: eng | + Chapter atom | + Chapter UID: 4118494873361489949 | + Chapter time start: 00:58:58.702000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 01:01:47.203000000 | + Chapter display | + Chapter string: Chapter 08 | + Chapter language: eng | + Chapter atom | + Chapter UID: 2050106467082172595 | + Chapter time start: 01:01:47.203000000 | + Chapter flag hidden: 0 | + Chapter flag enabled: 1 | + Chapter time end: 01:01:47.467000000 | + Chapter display | + Chapter string: Chapter 09 | + Chapter language: eng |+ EBML void: size 26 |+ Cluster |
|
The entire segment still needs to be appended whether whole or in parts. Forcing it into the queue as multiple appends does not prevent QuotaExceeded errors. I'm seeing either corruption errors or unresolved buffer full errors with these changes. It's unclear if these changes are intended to work on top of #7749 or replace them. Please rebased against latest - #7749 has been merged, but those changes are missing from this branch. |
| this.onBufferAppending(event, { | ||
| ...eventData, | ||
| data: chunks[i], |
There was a problem hiding this comment.
The BUFFER_APPENDING event data's chunkMeta should specify the chunk "id" (index) to aid in tracking append progress:
hls.js/src/controller/base-stream-controller.ts
Lines 1242 to 1251 in 6373b33
const chunkMeta = new ChunkMetadata(
frag.level,
frag.sn,
i, // chunk ID should match index of split data
payload.byteLength
);
This PR will...
Split large HLS fMP4 segments into smaller chunks before appending to MSE SourceBuffer, preventing
QuotaExceededErrorwith high-bitrate streams.Why is this Pull Request needed?
When an HLS fMP4 segment exceeds Chromium's per-SourceBuffer quota (~150MB),
appendBuffer()throwsQuotaExceededError. With high-bitrate remuxed streams (80-150+ Mbps), a single segment can easily reach 95-100MB. At playback start there is no back buffer to evict, so hls.js enters an unrecoverable loop: download → append → QuotaExceededError → flush → retry → repeat.This PR adds
splitAppendData()tobuffer-controller.tsthat splits largeUint8Arraybuffers into ≤16MB chunks before appending. Whendata.byteLengthexceeds 16MB,onBufferAppendingsplits the data and recursively creates separate append operations for each chunk — each with full error handling including QuotaExceeded recovery.Splitting is attempted first at fMP4 top-level box boundaries. If that can't produce small-enough chunks (e.g. a single giant
mdatbox, which is the typical FFmpeg HLS output:styp + moof + mdat), it falls back to naive byte splitting. MSE SourceBuffer natively handles reassembly of partial box data across sequentialappendBuffer()calls.Are there any points in the code the reviewer needs to double check?
splitAppendData()— verify it correctly parses top-level box headers and falls back to naive splitting when boxes exceed the chunk sizeonBufferAppending— each chunk gets its own queued operation with the same frag/part/chunkMeta context and error handlingResolves issues:
Fixes #6711, #6776. Related: #5587, #6529.
Checklist