feat: unreliable datagram support (QUIC + libp2p-datagram)

[royzah]

Description

Adds unreliable datagram support to libp2p, end to end:

• core: StreamMuxer::send_datagram / max_datagram_size and StreamMuxerEvent::Datagram (default: unsupported).
• quic: implements them over quinn datagrams, enabled by default and configurable.
• swarm: ConnectionEvent::Datagram (inbound) and ConnectionHandlerEvent::SendDatagram (outbound).
• New libp2p-datagram crate: a Behaviour + Control with send_datagram(peer, ...) and an IncomingDatagrams stream.

Motivation and prior art: libp2p/specs#626 (IP over libp2p), #225, and the MTU edge in #4647. Tunnelling IP over reliable, ordered streams is 2-10x slower (HOL blocking; TCP-over-TCP meltdown); a datagram carrier fixes it.

Notes

• Muxer trait methods default to unsupported, so other muxers are unaffected.
• quic::poll now surfaces connection close via the datagram future (was always Pending).
• Tests: quic datagram roundtrip, datagram-crate e2e over QUIC, swarm unit tests.

Open questions

• Datagrams default to on in quic::Config; happy to make it opt-in.
• Should the new enum variants be #[non_exhaustive]?

[jxs]

Hi, and thanks for starting this!
as commented on your specs post, wdyt of implementing libp2p/specs#680 and go from there? You're in need for this in production right? That would be a nice testbed for the spec

[royzah]

Nice, that's the direction we were hoping for. The quic/core/swarm bits here are the datagram primitive #680 needs anyway, so I'll keep those and rework the libp2p-datagram crate to match #680: /dg/1 control stream, app-proto-id prefix, Control StreamID varint per datagram. One extra bit I'll need is exposing the control stream's QUIC stream id through the muxer, since #680 keys on it. We'll run it on our radio mesh and report back.

[royzah]

Hey @jxs, while you're here, mind also casting an eye over #6487 (Control::open_stream_on_connection)? It's a small one and we need it in production too, alongside the datagram work.
WDYT ?

[royzah]

specs#680 is implemented here: /dg/1 control stream + app proto id, datagrams keyed by the control stream's QUIC id (via a new StreamMuxer::substream_id), inbound filtered by it. Tests green over QUIC. Radio-mesh bench next, will report numbers.

Two spec notes: one app-protocol per behaviour for now, and we drop on a bad/unknown stream id rather than raising PROTOCOL_VIOLATION (not reachable from this layer).

[jxs]

A lot of nice things in the PR already!

I agree with the datagram support at the Muxer and Transport level implementations.

But I read the spec definition of the /dg/1 protocol control stream as a protocol that each application protocol may negotiate individually, not a standalone protocol with its own Behaviour. I.e:

// listen_protocol advertises both protocols
fn listen_protocol(&self) -> SubstreamProtocol<Self::InboundProtocol, Self::InboundOpenInfo> {
    SelectUpgrade::new(
        GossipsubUpgrade,                       
        DgControlUpgrade { app_id: "/meshsub/1.0.0" },
    )
}

Where DgControlUpgrade::upgrade_inbound reads and validates [uvarint] [app_proto_id] from the stream per the spec. The handler then tracks control_stream_id and handles SendDatagram/Datagram events itself, framing/parsing the QUIC varint inline.

However, I'm torn on one point. The decentralized approach means every datagram-enabled handler reimplements the same pattern (manage /dg/1 control stream, filter by control_stream_id). An alternative would be to add a method like supports_datagrams(protocol: StreamProtocol) -> bool to the ConnectionHandler trait.
The Swarm could then aggregate which protocols each sub-handler supports datagrams for, and the connection layer could route Datagram events directly (parsing the QUIC varint once and dispatching to the correct handler by matching control_stream_id against known datagram-supporting streams). This centralizes routing and avoids every handler filtering every datagram, at the cost of more trait machinery.

What do you think?
Thanks!

[jxs]

should instead be err no?

[royzah]

done in 15062bd

[jxs]

let's remove this, it's preferable to let CI fail while we haven't published than have this dangling afterwards and not catch semver errors

[royzah]

done in 8037022

[royzah]

Thanks @jxs. We went decentralized deliberately, and measured it.

Per handler, the /dg/1 + control_stream_id filter is about a dozen lines: one QUIC varint decode and a HashMap lookup per datagram, ~100 ns. With one datagram protocol per connection (rarely two), that stays flat.

Numbers, arm64 over two mesh bearers: 100 Mbit/s at 0% loss on Wi-Fi 802.11s, 5 Mbit/s on sub-GHz. At a 1200-byte cap that's ~10k datagram/s, so the filter is ~0.1% of a core. The QUIC path and the link are the bottleneck, never routing.

Centralized supports_datagrams + Swarm aggregation trades a ConnectionHandler change for optimizing exactly that non-bottleneck. We'd keep the connection layer as is and let each handler own its filter; cost scales with handler count, which stays small by design. Happy to switch to the trait method if you prefer it as the cleaner API.

[jxs]

Thanks @jxs. We went decentralized deliberately, and measured it.

Per handler, the /dg/1 + control_stream_id filter is about a dozen lines: one QUIC varint decode and a HashMap lookup per datagram, ~100 ns. With one datagram protocol per connection (rarely two), that stays flat.

Numbers, arm64 over two mesh bearers: 100 Mbit/s at 0% loss on Wi-Fi 802.11s, 5 Mbit/s on sub-GHz. At a 1200-byte cap that's ~10k datagram/s, so the filter is ~0.1% of a core. The QUIC path and the link are the bottleneck, never routing.

Centralized supports_datagrams + Swarm aggregation trades a ConnectionHandler change for optimizing exactly that non-bottleneck. We'd keep the connection layer as is and let each handler own its filter; cost scales with handler count, which stays small by design. Happy to switch to the trait method if you prefer it as the cleaner API.

Thanks for te measurements! Yeah let's then tackle the centralized approach to see if it's clean enough the internal routing and matching the existing behaviors.

Thanks!

[royzah]

@jxs sounds good, going with the centralized route. Plan:

• Add ConnectionHandler::supports_datagrams(&self, protocol: &StreamProtocol) -> bool, default false so other handlers don't change.
• Parse the datagram varint once in the connection layer instead of in every handler.
• When a handler says it supports datagrams for a protocol, the connection saves that control stream id. Incoming datagrams get parsed once and sent to the matching handler; unknown ids are dropped. So the datagram crate just gets datagrams already routed to it.

One thing to decide: the connection only sees the combined handler, so the final hop goes through ConnectionHandlerSelect. I'd route there using the id we already parsed, matched against the ids each child saved. That keeps supports_datagrams a simple bool and still routes right when two datagram protocols share a connection. Ok with that, or would you rather the trait just expose the stream ids directly?

[royzah]

Sounds good. The shape I'm picturing:

supports_datagrams(protocol) -> bool on ConnectionHandler (default false, so nothing else moves). The connection parses the datagram varint once, remembers which control-stream id each supporting handler claimed, and routes inbound datagrams to the matching handler. Unknown ids get dropped, so the datagram crate just receives datagrams already addressed to it.

Question though, connection only sees the combined handler, so the last hop goes through ConnectionHandlerSelect. I'd route there by the id we already parsed, matched to each child's saved id. That keeps the trait a plain bool and still does the right thing when two datagram protocols share a connection. WDYT, or would you rather the trait hand back the stream ids directly?