Skip to content

Multicast e2e - #9912

Open
zeeshanlakhani wants to merge 30 commits into
mainfrom
multicast-e2e
Open

Multicast e2e#9912
zeeshanlakhani wants to merge 30 commits into
mainfrom
multicast-e2e

Conversation

@zeeshanlakhani

@zeeshanlakhani zeeshanlakhani commented Feb 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

This is the integration branch for multicast support across the stack. A lot of the Nexus-side multicast work has already landed on main through earlier PRs. This consists the cross-repo dep pinning for the in-flight revisions of the other repos that multicast depends on, OMDB tooling, and test updates.

Cross-repo pins

  • dendrite ab6c1a43 (defaults the dpd multicast feature on and wires the SoftNPU backend for multicast group programming): dendrite#224
  • maghemite 0509f500 (DDM underlay multicast route origination/import): maghemite#696
  • opte f4433191 (xde multicast delivery), via tools/opte_version_override: opte#1012
  • propolis cd0e2618 (viona receive filtering): propolis#1093
  • softnpu 284c6830 / sidecar-lite 6f3311e8 (dev virtual hardware, bitmap-based replication): softnpu#183,
    sidecar-lite#152
  • thundermuffin fda704ee (UDP multicast send/recv, merged): thundermuffin#13

omdb support

  • omdb db multicast groups|members|pools|info: inspect multicast group state, membership (with instance/sled context), pool utilization, and per-group detail from the DB.
  • omdb nexus background-tasks: we can get info for the multicast_reconciler task status, including the disabled case.

Test and schema updates

  • Multicast integration tests move to the latest versioned instance types
  • allow_tables_to_appear_in_same_query! for the multicast tables, needed by the omdb joins.
  • New object_put_upsert test helper for PUT endpoints that return 201.

@zeeshanlakhani zeeshanlakhani self-assigned this Feb 25, 2026
@zeeshanlakhani
zeeshanlakhani force-pushed the multicast-e2e branch 4 times, most recently from 52ad8d8 to cc63562 Compare March 6, 2026 01:46
@zeeshanlakhani
[multicast] add omdb mcast commands for groups, members, pools (#9464)
a741697 
This PR adds omdb commands to inspect multicast state:

- `omdb db multicast groups` - list multicast groups with optional state
and pool name filters
- `omdb db multicast members` - list group members with filters for
group-id, group-name, group-ip, state, and sled-id
- `omdb db multicast info` - show detailed info for a specific group
- `omdb db multicast pools` - list multicast IP pools

We also include:
  - Background task status display for multicast_reconciler
  - Integration tests for all multicast omdb commands
@zeeshanlakhani
merge main into multicast-e2e
8d4156a
@zeeshanlakhani
merge main into multicast-e2e
4b5c6d3
@zeeshanlakhani
merge main into multicast-e2e
bfb8c7b
@zeeshanlakhani
merge main
5c29369
@zeeshanlakhani
zeeshanlakhani force-pushed the multicast-e2e branch 8 times, most recently from 4ed3d4e to 706ba96 Compare April 21, 2026 01:32
@zeeshanlakhani
zeeshanlakhani force-pushed the multicast-e2e branch 2 times, most recently from a0fcb56 to f3754a5 Compare April 22, 2026 05:19
@zeeshanlakhani
zeeshanlakhani force-pushed the multicast-e2e branch 2 times, most recently from 9367136 to 868c08a Compare May 28, 2026 12:31
@zeeshanlakhani
[deps] merge main + dep updates for multicast end-to-end across repos
8fd9ce0 
Also carries the tools script fixes from #10825 (opte override p5p
publisher and gcc-runtime version parse), so the install scripts work
on this branch before that PR lands on main.
@zeeshanlakhani
zeeshanlakhani marked this pull request as ready for review July 15, 2026 15:15
@zeeshanlakhani

zeeshanlakhani commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator Author

We're using a non-mainline opte version, hence why that CI job (purposely) fails.

@zeeshanlakhani
merge main + enable multicast flag in deployment configs
d1f36b3 
This merges main to pick up the OPTE p5p publisher fixes (#10825), and
adds [multicast] enabled = true to the single-sled and multi-sled Nexus SMF
config partials so deployments start with multicast switch programming
and the reconciler active.
@zeeshanlakhani
zeeshanlakhani force-pushed the multicast-e2e branch 3 times, most recently from 913a625 to 3fc58bf Compare August 7, 2026 06:26
@zeeshanlakhani
[merge] merge main + SSM scope tightening, dep updates
a793ae6 
Beyond the merge itself:

- Tighten IPv6 multicast admission to the RFC 7346 usable-scope
  allowlist (admin-local, site-local, organization-local, global).
  This matching validation that landed in dendrite and thundermuffin.
- Add API version 2026080400 (MULTICAST_SSM_EXAMPLE_DOCS) for the
  SSM example-address doc updates.
- Bump multicast-tracking pins: dendrite 0d48b5a3, maghemite
  e4d24324, propolis df5fd0a3, thundermuffin 486559bc,
  sidecar-lite 461cbe19.
- Drop the oxlog [patch] entry and the ls-apis crucible-pantry
  filter rule, as maghemite was updated.
@zeeshanlakhani
[nexus] fix multicast group leak on rejected instance update
ba1e54e 
An instance update whose multicast specs contain duplicates would
be rejected only after resolution had run, and resolution implicitly
creates missing groups as part of the lifecycle. The rejected request
stranded a memberless group in a "Creating" state that implicit
lifecycle reaping would never collect, since collection only triggers on
last-member-leave.

Instead, we now reject identical identifiers before resolution. For
aliases that remain past that check (a group's IP alongside its
generated name), the resolver reports whether it created the group,
and a rejected request rolls back only its own creations via
`multicast_group_mark_removal_if_empty`. Ownership comes from
the resolver itself, since a creation race can return a group another
request created.
@zeeshanlakhani
[nexus] better handling of multicast side effects in instance update+…
87a96f1 
…create

We now split the multicast portion of instance reconfiguration into
plan, apply, and compensate phases. Planning resolves and deduplicates
the requested groups and drops any implicitly created groups when
resolution rejects the request, before any observable side effect.
Updates whose multicast snapshot no longer matches current membership
are rejected with a conflict rather than overwriting concurrent joins
and leaves.

Implicit creations that remain after planning and end up memberless after
a later failure are left to the reconciler's empty-group and orphan
passes, which now share a minimum-age grace so a just-created group is
not reaped while its first join is still in flight. A memberless group
in "Creating" is never activated: the reconciler lists members first,
reaps only past the grace window, and declines if a member attached
since the listing.

Compensation now reverses only the changes a request made itself. The
datastore attach returns an `AttachMemberResult` carrying an inserted
flag that distinguishes a fresh insert from an upsert hitting an
existing active row, so a failed request no longer detaches a
membership that a concurrent request established. Detaches are
recorded the same way, from the datastore's row-affected result.
Retried saga actions recognize groups and members from their earlier
invocations and undo only what they own. That recovery spans the
orphan pass: a reaped group that never reached the dataplane is
restored with a fresh grace window, and a hard-deleted row frees the
stable ID for re-creation. Only a soft-deleted row that had entered
the dataplane blocks retries, surfaced as a retryable 503 until
cleanup hard-deletes it.

Deletion processing now skips dataplane teardown for a group whose
underlay link was never written. The link is set before the DPD saga
launches and is never cleared, so its absence proves the group owes no
switch or sled cleanup. This also closes a race in which a slow
delete pass could wipe fresh DPD state from a group resurrected and
re-activated under the same tag, since resurrection requires the link
to be absent.
@zeeshanlakhani
[nexus] stop echoing the underlay member list from group updates
8fc8591 
Tag and source updates read the underlay group and had written the read
member list straight back. `multicast_group_update_underlay` is a
full-list replace whose only concurrency control is the per-group tag,
so that echo-write could silently revert a member add or remove that
landed between the read and the write.

`update_groups` now reads the underlay group only to obtain the tag
authorizing the external update, creating the group (memberless) when
absent, and never writing the underlay member list. The member add and
remove paths remain the list's only writers. This also removes an
underlay write the planned member-writer transition would otherwise
have to delete.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FelixMcFelix FelixMcFelix Awaiting requested review from FelixMcFelix

Assignees

@zeeshanlakhani zeeshanlakhani

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zeeshanlakhani