Implement RFC 5053 Raptor - #61
Draft
jordijoangimenez wants to merge 18 commits into
Draft
Conversation
…r alive until Boost.Asio calls the completion lambda.
…ebuild inner IPv4/UDP headers with byte-wise network-order checksum generation so tunneled end-to-end delivery passes reliably in release builds.
…eceive-buffer overflow Squashed re-application of accumulated fixes/features previously developed on a personal fork whose history had diverged from 5G-MAG/rt-libflute's actual current development/main (unrelated histories, confirmed via git merge-base returning nothing, despite matching content up to a shared point -- likely from a prior history rewrite on one side only). Re-applied here as a single clean commit against the real upstream base rather than replaying the original, now-irreconcilable commit sequence: - Add SSM (source-specific multicast) join support to Receiver, so a session can admit packets only from a specified source address rather than any-source multicast. - Fix a use-after-free: an async_receive_from completion already queued on the io_context when a Receiver is destroyed could run after the destructor returns and touch a freed `this` -- boost::asio only guarantees a cancelled operation's handler eventually runs with operation_aborted, not that it runs before the destructor returns. Receiver now tracks its own liveness via a shared atomic flag copied into each completion handler, checked before touching `this`. - Fix Receiver's fixed 2048-byte receive buffer silently truncating larger encoding symbols: recvfrom() on a datagram socket doesn't error on a too-small buffer, it silently truncates, so any FEC-OTI configuration with larger symbols corrupted every symbol beyond 2048 bytes without any visible error until an FDT Content-MD5 check (if present) caught it. Buffer is now 65536 bytes, covering the maximum possible IPv4 UDP payload.
Transmitter::send() reused the same FileDescription/TOI correctly for carousel-style repeated objects, but two bugs meant each resend still grew the file delivery state without bound: - _files.insert() is a no-op if the TOI key is already present, so a resend's updated File object was silently discarded in favour of the stale one already in the map. - FileDeliveryTable::add() unconditionally appends a new <File> entry with no dedup by TOI, so every resend left the previous cycle's entry for the same TOI in place -- the serialised FDT grows by one entry per resend indefinitely (observed growing from ~850 bytes to several hundred KB over a couple of hours of a 10-second carousel), eventually becoming too large for a receiver to reassemble at all. Per RFC 6726 SS3.3/3.4.2, an FDT Instance describes the current state of the file delivery session; a Content-Location may be redescribed under a new TOI to signal a new version, but parameters already described for a given TOI must not change -- so on an actual resend of unchanged content, the sender should replace that TOI's single File entry, not accumulate duplicates of it. This restores that invariant: track whether this send is a resend (TOI already assigned), replace the File map entry instead of no-op'ing, and remove the TOI's existing FDT entry before adding the current one.
The previous fix (same TOI, resent unchanged content) only handled one growth vector. A second, distinct one remained: when set_content()/set_compression() detect the content genuinely changed, they zero the FileDescription's TOI so Transmitter::send() assigns a fresh one -- but nothing ever removed the FDT entry for the TOI being vacated. Confirmed live: a carousel object whose content legitimately changes each cycle (e.g. a randomly-regenerated MIME boundary) grew its FDT to 135-170KB within about a minute, well before the previous fix's growth timescale, eventually failing to parse (XML_ERROR_PARSING_ATTRIBUTE). FileDescription now remembers the TOI it's vacating (_previous_toi, set by a new _reset_toi() helper used everywhere the TOI was zeroed) so Transmitter::send() can remove that stale entry before assigning the replacement TOI, restoring the "one current entry per logical object" invariant regardless of which of the two ways an object's description changes.
- File.cpp: use fmt::format instead of manual std::to_string() concatenation for the two bounds-check exception messages, per the suggestion -- fmt::format rather than std::format since this project targets C++17 (std::format needs C++20); same result, already a dependency via spdlog. - Transmitter.h/.cpp: added FileDescription::previous_toi()/ reset_previous_toi() public accessors instead of Transmitter::send() reaching into FileDescription's private _previous_toi directly via friend access, per the suggestion. - Receiver.cpp: restored IPv6 support the SSM/specific-interface-join fix had inadvertently dropped (the original code let Boost infer v4 vs v6 from the address types passed; the fix hardcoded .to_v4() throughout). Now branches on the multicast address's actual family: IPv6 ASM join uses join_group(address_v6, interface_index) and SSM join uses MCAST_JOIN_SOURCE_GROUP/group_source_req, both interface-by-index (unlike IPv4's interface-by-address), so a new resolve_iface_index() helper resolves the existing iface address string to its owning interface's index via getifaddrs()/if_nametoindex(). IPv4 behaviour (bind/ASM/SSM) is unchanged, verified by rerunning the existing test_end_to_end.cpp unmodified.
…nflict Both branches touched Transmitter::send_next_packet()'s tunnelled-send path: this branch (PR #56) made the encapsulated packet buffer lifetime-safe by owning it in a shared_ptr<vector<char>> captured by the async_send_to completion lambda, replacing a raw new[] with no matching delete[] (a leak on every tunnelled packet). development, in parallel, added _source_address as an explicit override for the local address used to build the inner IPv4/UDP headers, falling back to _tunnel_local_address when unset. Kept both: the shared_ptr-owned buffer from this branch, with the _source_address-or-_tunnel_local_address fallback from development.
…56 branch Integration base for building the Raptor/RaptorQ FEC work on top of both outstanding PRs' assumed-merged state: PR #56's tunnel-buffer lifetime fix (reconciled with development's _source_address fallback) plus PR #60's FDT-growth fixes with the previous_toi()/reset_previous_toi() accessor pattern and full IPv4/IPv6 parity in Receiver's SSM/ASM join and bind logic. No conflicts with the prior merge; both touch disjoint enough regions of Transmitter.cpp/.h that git combined them cleanly.
…ed state This branch is built assuming both outstanding PRs land first: PR #56's tunnel-buffer lifetime fix (with development's _source_address fallback, already reconciled in an earlier commit on this branch) and PR #60's FDT-growth fixes with the previous_toi()/reset_previous_toi() accessor pattern and full IPv4/IPv6 parity in Receiver. No shared git history exists between this fork's line of development and the current upstream development branch (a previously-diagnosed history rewrite upstream, see PR #60's own 184e482), so this reconciliation was done as a content-level 3-way patch apply against this integration base, not a rebase -- 14 files had genuine overlapping hunks, resolved as follows: - flute_types.h, AlcPacket.cpp, EncodingSymbol.cpp, File.cpp, FileDeliveryTable.cpp: kept the Raptor/RaptorQ-aware superset (FecOti's scheme-specific fields, multi-scheme FEC Payload ID / EXT_FTI / FDT attribute parsing) -- these fully subsume the integration base's CompactNoCode-only logic. - Receiver.cpp: kept the integration base's version entirely -- PR #60's IPv6-aware SSM/ASM join and bind logic is strictly more complete than this branch's older IPv4-only copy of the same code, and Raptor/RaptorQ content needs no Receiver-side changes beyond what File/EncodingSymbol already provide. - Transmitter.h/.cpp: combined per-hunk -- kept the accessor pattern (previous_toi()/reset_previous_toi()) and the graceful deactivate(bool finish_file_transmissions) lifecycle from the integration base; kept this branch's content_fec_oti constructor parameter and fec_oti()/fdt() accessors; and rebuilt this branch's dual-send fix (send both a plain copy and a tunnelled copy, needed for N3mb GTP-U tunnelling per TS 23.247 while still supporting direct SSM subscribers) on top of PR #56's cleaner byte-wise checksum/header implementation (create_udp_pkt/create_ip_hdr/calculate_sum operating on uint8_t* with explicit write_uint16_be/write_uint32_be helpers, not the older uint16_t*-punned struct-overlay approach) instead of keeping two divergent checksum implementations. - CMakeLists.txt, tests/CMakeLists.txt, examples/flute-transmitter.cpp, include/File.h: trivial additive conflicts (version bump, new test targets, new #includes, an unrelated exception-slicing/format-string bug fix already on this branch). - tests/test_transmitter.cpp, tests/test_end_to_end.cpp: kept the integration base's superset (adds a UDP-tunnel e2e test and a graceful- deactivation lifecycle test this branch didn't have); this branch's own duplicate of the basic transmit/receive test added nothing Raptor- specific -- that coverage lives in test_raptor_e2e.cpp/test_raptorq_fec.cpp. Verified: full library + all 6 test binaries (25 tests total) build and pass against the reconciled tree.
PR #56 landed this morning as a squash commit, superseding the hand-built integration base this branch was previously reconciled against. Re-merged against the real development tip: - Transmitter.cpp: one conflict, same shape as before -- kept this branch's dual-send feature (plain + tunnelled copy, needed for N3mb GTP-U per TS 23.247) rebuilt on #56's shared_ptr-owned buffer, since dual-send was never part of #56 itself and #56's actual merged content confirms that. - tests/test_end_to_end.cpp: 12 conflicts, all the same underlying change -- the real #56 merge added a std::mutex protecting TunnelBridgeStats from a genuine data race between the tunnel-bridge thread and the test's main thread, which this branch's copy of the test lacked. Took development's thread-safe version throughout. - tests/tmp/e2e_payload.bin deleted, matching development: the current test generates its payload in-code and no longer reads this fixture. Verified: full library + all 6 test binaries (25 tests) build and pass.
…livery method Checked directly against TS 26.346 v18.2.0 cl.7.2.2/7.2.12: this profile mandates Compact No-Code (FEC Encoding ID 0) and Raptor/RFC 5053 (ID 1) -- 'A UE that supports MBMS User Services shall support a decoder for the Raptor FEC scheme' -- and does not define or reference RaptorQ/RFC 6330 (ID 6) anywhere for the download delivery method. RaptorQ isn't forbidden, just outside today's 3GPP-defined scheme set, and this library's primary consumers are 3GPP/5G-MAG MBMS applications. Removed RaptorQCodec, GF256LinearSystem (RaptorQ's GF(256) linear algebra, unneeded by Raptor's GF(2)), and their headers/tests, and stripped the FecScheme::RaptorQ dispatch branches from AlcPacket/File/FileDeliveryTable/ EncodingSymbol, keeping every Raptor (FecScheme::Raptor) path unchanged. The full Raptor+RaptorQ implementation is preserved on future/raptorq-support (jordijoangimenez/rt-libflute) in case RaptorQ is wanted for a non-3GPP FLUTE deployment, or 3GPP adopts it in a future release. Verified: library + all 5 remaining test binaries (17 tests) build and pass. The two RaptorQ-only test binaries (flute_raptorq_tests, and RaptorQE2ETest within flute_raptor_e2e_tests) are gone; every remaining Raptor test is unaffected.
Contributor
Author
|
Dropped RaptorQ from this PR after checking directly against TS 26.346 Full RaptorQ implementation preserved on Rebuilt and retested against the real merged #56 -- library + all 5 |
jordijoangimenez
changed the title
This was referenced Aug 11, 2026
Closed
|
@jordijoangimenez: Is this one also replaced by #68? |
Contributor
Author
|
Hello, I wanted to have incremental PRs assuming the previous would be merged but the new PRs were created on top carrying the previous code which would not create regressions. I wonder if it would be better to check that each individual PR is acceptable so I take care of rebase (should not be needed is merged as is) or reconciling the code. So I prefer individual comments per new code added to new PRs and I take care of everything |
Contributor
|
Thanks @jordijoangimenez , two minor suggestions from my side:
|
jordijoangimenez
marked this pull request as draft
August 14, 2026 10:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements RFC 5053 Raptor FEC, built from scratch (not ported from PR #7),
for FLUTE content delivery. Adds
FecScheme::Raptoralongside the existingCompactNoCode, with scheme-aware FEC OTI (RFC 5053 §3.2.3), FEC Payload ID,and FDT attribute handling throughout
AlcPacket,EncodingSymbol,File,and
FileDeliveryTable.3GPP TS 26.346 v18.2.0 cl.7.2.2 mandates this: "A UE that supports MBMS User
Services shall support a decoder for the Raptor FEC scheme." This library's
primary consumers are 3GPP/5G-MAG MBMS applications.
RaptorQ was dropped from this PR
An earlier version of this PR also implemented RFC 6330 RaptorQ. Checked
directly against TS 26.346 v18.2.0 cl.7.2.2/7.2.12: this profile defines
Compact No-Code (ID 0) and Raptor/RFC 5053 (ID 1) only, and doesn't reference
RaptorQ/RFC 6330 (ID 6) anywhere for the download delivery method. Not
forbidden, just outside today's 3GPP-defined scheme set -- and not worth
presenting as spec-mandated when this PR's actual justification (the Raptor
requirement above) doesn't extend to it.
Preserved in full on
feature/raptorq-support(jordijoangimenez/rt-libflute)in case it's wanted for a non-3GPP FLUTE deployment, or 3GPP adopts it later.
Built assuming PR #60 lands
developmenton 2026-08-07.This branch is rebased onto that.
Receiver) hasn't merged yet.The
previous_toi()/reset_previous_toi()accessor pattern,fmt::formatexception messages, and IPv4/IPv6
Receiverparity are all carried forwardhere and used consistently in the new code. Until it merges, this diff
still includes its content; rebasing onto
developmentagain once it landsshould shrink the diff to just the Raptor-specific changes.
Reconciling this branch's own prior fixes
Two fixes that predated #56/#60 needed rebuilding, not just carrying forward:
packet -- needed for N3mb tunnelling per TS 23.247 while still supporting
direct SSM subscribers) is rebuilt on top of Feature/end to end test tunneled mode #56's cleaner, portable
byte-wise checksum/header implementation (
create_udp_pkt/create_ip_hdr/calculate_sumoperating onuint8_t*), replacing this branch's olderuint16_t*-punned struct-overlay version.dropped in favour of Fix unbounded FDT growth (same-TOI resend, and content-change cases) #60's reviewed version (with the accessor pattern).
Testing
Full library + all 5 remaining GTest binaries (17 tests) build and pass
locally: unit tests, e2e (including the UDP-tunnel variant), Raptor unit + e2e,
FDT-growth regression. Local build environment has a newer Boost than CI's
ubuntu-latest; CI is the authoritative build check.