Skip to content

Relay: detect the rest of §2.4.2's malformed-track conditions - #106

Merged
floatdrop merged 1 commit into
draft-20from
fix/malformed-final-objects
Sep 26, 2026
Merged

floatdrop merged 1 commit into
draft-20from
fix/malformed-final-objects

Conversation

@floatdrop

Copy link
Copy Markdown
Owner

Summary

Until now the relay ended a track only for malformed Object Properties, or for an Object after END_OF_GROUP on the same stream. It now detects the rest of §2.4.2's list, on live subgroup and datagram Objects from any upstream, against the last 32 Groups (the same window and lock as the dedup ledger):

What counts as an end. An end is stored as its first missing Object ID:

Signal End
END_OF_GROUP status at M the Group ends at M
END_OF_TRACK status at M the Group and the Track end at M
FIN after Object N (§11.4.3) the Subgroup ends at N+1
FIN after N on a stream whose header sets END_OF_GROUP (§11.4.2) also the Group, at N+1
datagram END_OF_GROUP bit on Object N (§11.3.1) the Group ends at N+1

Other behaviour:

  • Checked either way round: each end is also checked against Objects already received, and duplicates count. A duplicate is recorded (TrackEntry.RecordDuplicate) only after it passes the §9.1 check, and re-checked under the lock.
  • Cache purge: when the track is ended this way, Objects past the end are removed from the cache (§2.4.2: they "MUST NOT be cached"). So is a first copy that fails the §9.1 duplicate check.

Interpretations (marked per CLAUDE.md; chosen with you)

  • One end, two representations: §2.4.2 calls both the status Object at M and Object N the "final Object". The two are the same end when M = N+1, since §9.1 lets a relay turn one into the other.
  • Past an end is malformed: an Object past an end ends the track rather than being dropped. That matches §2.4.2's explicit list and §11.4.3 ("a protocol violation"), unlike Relay: act on Prior Group/Object ID Gaps across Objects (§2.1, §9.1, §12.8) #104's §2.1 gap reading.
  • FIN/bit wins: a Normal Object at an end is past it only if a FIN or END_OF_GROUP bit set that end. With status Objects alone, it's §9.1's existing-to-not-existing change or §2.1's late Object, in either order. For the same reason, a status end at M and a FIN/bit end at M+1 agree, and the end is M.
  • Datagram bit counts: a datagram's END_OF_GROUP bit counts as a Group's end. §2.4.2's list is "not considered exhaustive".

Not covered (listed in STATUS.md)

  • Where it isn't checked: upstream FETCH responses (whose End of Track isn't used either), sessions that aren't a relay's, and Groups older than the window.
  • Cache race: an Object another publisher had claimed but not yet cached when a later end put it past that end stays cached. You chose to document this rather than add a lock on every Object.

Code changes

  • registry.TrackEntry.ClaimDelivered takes an ObjectInfo. The new SubgroupEnded is called on a clean inbound FIN, through sessionHandler.inboundEnded, which I extracted from runFanout to stay under the lint limits.
  • TestFanout_MultiPublisher_MergesDisjointObjects had two streams FIN the same Subgroup with different final Objects, which item 3 makes malformed. It showed up under -race. Publisher A's stream now ends with a reset.

Tests (each verified red first)

  • TestTrackEntry_FinalObjects: priority, ends, the agreement cases, and every arrival order the reviews probed.
  • Registry tests:
    • TestTrackEntry_RecordDuplicateChecksAgain covers the race.
    • TestTrackEntry_LateEndPurgesCache covers Subgroup, Group and END_OF_TRACK, including what must stay cached.
  • TestCheckDuplicatePurgesFirstCopy: the cached first copy goes when a duplicate differs.
  • Relay-level:
    • Six new TestRelay_MalformedObjectEndsTrack cases, each malformed in either stream order.
    • TestRelay_EndSignalsAgree (a status end and a FIN end agree).
    • TestRelay_DuplicateEndOfGroupEndsGroup (datagrams from two publishers).
  • Checks: lint, the full suite, -race and make bench-quick all pass, with no allocs/op increase.

The moqt-reviewer took four rounds; the last found no order dependence across about 260,000 random interleavings.

🤖 Generated with Claude Code

The relay ended a track for Object Properties and for an Object after
END_OF_GROUP on the same stream. It now also detects, on live subgroup
and datagram Objects of any upstream, against the last 32 Groups:

- a Subgroup's Publisher Priority changing (item 1);
- an Object past a Subgroup's, Group's or Track's end, or two
  different ends (items 2 to 5).

An end is kept as the first missing ID: an END_OF_GROUP or END_OF_TRACK
status at M ends the Group at M (END_OF_TRACK the Track too); a FIN
(§11.4.3; the Group's too with the END_OF_GROUP header bit, §11.4.2) or
a datagram's END_OF_GROUP bit after Object N at N+1. Each end is also
checked against the Objects already received, duplicates included
(TrackEntry.RecordDuplicate, after the §9.1 check), and the Objects past
it are removed from the cache (§2.4.2 MUST NOT be cached), as is a first
copy that fails the §9.1 duplicate check.

Interpretations, chosen with the user and marked in STATUS.md:
- §2.4.2 calls both the status Object at M and Object N the "final
  Object"; they are one end when M = N+1, as §9.1 lets a relay turn one
  into the other;
- Objects past an end make the track malformed, not dropped;
- a Normal Object at an end is past it only if a FIN or bit set it:
  with status Objects alone it is §9.1's existing-to-not-existing
  change or §2.1's late Object; a status end at M and a FIN end at M+1
  agree for the same reason;
- a datagram's END_OF_GROUP bit counts as a Group's end;
- item 7 is read per Object (§11.2.1), which the §9.1 check covers.
A claimed-but-not-yet-cached race with a later end is documented.

ClaimDelivered takes an ObjectInfo; SubgroupEnded is called on a clean
inbound FIN via the new sessionHandler.inboundEnded, extracted from
runFanout. TestFanout_MultiPublisher_MergesDisjointObjects FIN'd one
Subgroup on two streams with different finals (item 3); A now resets.

Tests, each verified red first: TestTrackEntry_FinalObjects (priority,
ends, their agreement and every order the reviews probed),
_RecordDuplicateChecksAgain, _LateEndPurgesCache,
TestCheckDuplicatePurgesFirstCopy, six TestRelay_MalformedObjectEndsTrack
cases, TestRelay_EndSignalsAgree, TestRelay_DuplicateEndOfGroupEndsGroup.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@floatdrop
floatdrop merged commit c5d3f8b into draft-20 Sep 26, 2026
11 checks passed
@floatdrop
floatdrop deleted the fix/malformed-final-objects branch September 26, 2026 10:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant