Disk-buffered recording and crash recovery - #175
Draft
ruccho wants to merge 3 commits into
Draft
Conversation
The previous storage layer kept frame payloads in two append-only files with a side index. It had two blocking defects: codec configuration was never written to disk, so a recovered session could not be muxed on Android, Windows, or Linux; and the payload files were never truncated, so disk usage grew for the whole session regardless of MaxDiskUsageBytes. Frames are now written to segment files that rotate at video key frames. Space is reserved before every write and the oldest segments are deleted to make room, so MaxDiskUsageBytes is a hard bound on the session directory at every instant rather than a target. Codec configuration goes to a separate file that is never evicted, and the reader treats its absence as normal, so recovery works both on platforms that emit it and on those that do not. Also addressed: - Writes move to a dedicated worker thread with a bounded queue, replacing synchronous I/O on the encoder thread. No allocation occurs per frame. - Each record carries a CRC-32, and scanning stops at the first record that cannot be complete, so a file truncated by a crash recovers cleanly. - The flush policy is explicit and documented: data reaches the operating system after every batch, which is what survives a process crash, and the storage device only at segment boundaries, which keeps flash wear low. - Recovery no longer deletes the session directory implicitly; Delete is separate, so a failed export can be retried. - Export failures no longer skip FinishVideoAsync and FinishAudioAsync. The live and recovery paths share EncodedFrameMuxer and EncodedFrameSelector, so both follow the same protocol and select frames identically. - The inconsistent conditional compilation is gone; the code compiles under every supported API compatibility level without directives. - The manifest records the platform, and the metadata leak on the early return paths of BoundedEncodedFrameBuffer is fixed. The storage layer has no dependency on UnityEngine and is exercised by InstantReplay.Externals/src/InstantReplay.DiskBuffer.Tests, which covers the record round trip, recovery from truncated and corrupt files, the hard bound under sustained eviction, and an end-to-end run that encodes with the real platform encoder, recovers from the files alone, and muxes a valid MP4. Design and rationale: docs/disk-buffered-recording.md
Describes enabling the buffer through RealtimeEncodingOptions.DiskBuffer, recovering a session left behind by an abnormal termination through DiskEncodedFrameBufferRecovery, and every DiskBufferOptions property. Spells out the two points that are easy to get wrong: MaxDiskUsageBytes is a bound rather than a target, so a value close to MaxSegmentBytes degrades the recording instead of the retention; and the default sync mode defends against a process crash, not against power loss. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #39.
Adds an opt-in disk buffer to
RealtimeInstantReplaySession. Encoded frames are written tosegment files instead of being held in memory, which lowers memory pressure and makes the
footage leading up to a crash recoverable for bug investigation.
Disabled by default. Enable with:
Recover after a crash:
MaxDiskUsageBytes is a hard bound
It bounds the whole session directory — manifest, codec configuration, and every segment — and
it is a bound rather than a target. Space is reserved before each write and the oldest segments
are deleted to make room; when the bound still cannot be met the record is dropped rather than
written. The directory never exceeds the limit at any instant, not even transiently between a
write and a subsequent eviction. Segments rotate at video key frames, so discarding one never
leaves a partial group of pictures behind.
Flush policy
The default,
DiskBufferSyncMode.OperatingSystem, hands data to the operating system afterevery batch and to the storage device only when a segment is closed. Frames therefore survive a
process crash — a native fault, an OOM kill, or an abort — which is what crash recovery targets,
without a device flush per frame. Power loss or a kernel panic loses at most one segment. The
manifest and codec configuration always reach the device immediately.
DiskBufferSyncMode.EveryRecordtrades markedly higher flash wear for durability against powerloss.
Crash recovery works on every platform
Codec configuration is written to a dedicated file that is never evicted, and the reader treats
its absence as normal. Recovery therefore succeeds both on platforms that emit configuration
records (Android MediaCodec, Windows Media Foundation, FFmpeg) and on those that carry the
parameter sets inside every sample and emit none (Apple VideoToolbox, WebCodecs).
Records carry a CRC-32 and are only appended, so a file truncated by a crash recovers up to the
last complete record. The live export path and the recovery path share one reader, one
selection, and one muxing helper, so every export exercises the recovery code.
Verification
The storage layer has no dependency on UnityEngine and is exercised outside the Editor:
Checks cover the record round trip, recovery from truncated and corrupt files, the hard bound
under sustained eviction, survival of codec configuration across full eviction, key-frame
alignment of segments, the manifest, and selection. The final check drives the real platform
encoder, recovers from the files alone, and muxes; on macOS the result is a valid MP4 that
ffprobe and ffmpeg accept.
Design and rationale:
docs/disk-buffered-recording.md.Not yet verified
Not run inside the Unity Editor, and crash recovery has not been exercised on a device.