protocol 1.7: outpoint subs

[SomberNight]

This defines a new Electrum Protocol version: 1.7.

changes compared to 1.6:

• the blockchain.scripthash.* methods are all replaced with blockchain.scriptpubkey.* methods
  • ref feature: blockchain.scriptpubkey.* endpoints #3
• new RPCs: blockchain.outpoint.subscribe RPC (and related RPCs) to subscribe to a transaction
  outpoint, and get a notification when it gets spent.
• server.ping can now also be sent as an unrequested notification, by both the client
  and the server. The request/response signature also changed.
• standardize "history too large" error code for blockchain.scriptpubkey.* methods
• new RPC: blockchain.transaction.testmempoolaccept
  • ref Add support for testmempoolaccept #11
• new RPC: mempool.recent to list a few of the latest txs just added to the mempool.
  • ref Add mempool.recent protocol method #13

---

This proposal is less ambitious than previous drafts of 1.7:

• most notably witness SPV is postponed
  • ref protocol 1.8 (WIP) #16
• the method_flavours field for the server.features response is also postponed,
  as I realised we are not clear re how to signal/negotiate optional features
  • see related Extend server.version for feature negotiation #7 idea
• blockchain.outpoint.subscribe requires spk_hint as an input param, to help server impls

---

We have a client implementation in spesmilo/electrum#10627,
and a server implementation in spesmilo/electrumx#303.

[ecdsa]

Edge case: if the history is not too long when you subscribe, but then it becomes too long.
Should the server send a notification in that case?
Error codes in jsonrpc only exist in responses, not in notifications.

[SomberNight]

Error codes in jsonrpc only exist in responses, not in notifications.

Right, I also looked this up exactly for this reason a couple days ago :D

Well the way ElectrumX has always worked in this case is that the server silently drops the subscription at that point. Not ideal but I would leave that as-is.

The protocol 2.0 history pagination and incremental recursive status would be the true fix.

[ecdsa]

We could return zeroes as the hash in the notification.
The client will request history if he receives a hash that does not match its own.

The protocol 2.0 history pagination and incremental recursive status would be the true fix.

indeed

[SomberNight]

force-pushed https://github.com/spesmilo/electrum-protocol/compare/acc7d161188c62efba6838748e26dcc17e8193b3..e8a379c369cec7ba1edb34951044243d9cfa5c79 to address feedback

[f321x]

Reviewed to e8a379c, looks reasonable to me

[SomberNight]

@cculianu @craigraw @Davidson-Souza @ecdsa @evoskuil @f321x @Overtorment @romanz @shesek @t-bast @tnull

If you are interested, please see if what is proposed here looks reasonable.

We want to require some of the changes here soon in the Electrum Wallet, and would want to also bump the minimum protocol version the client requires to 1.7 (from the [1.4, 1.6] range to [1.7, 1.7]). So unlike with the protocol 1.6 release, now users might complain they cannot connect to some servers.
(but servers will likely serve old clients for a long time)

[craigraw]

Reviewed. I think everything looks reasonable barring one major concern: The new Traffic analysis section bundles two different things under one SHOULD:

• Padding: reasonably cheap and backwards-compatible, needs no handshake. Fine as a SHOULD in 1.7.
• Buffering / timing delay: costs interactive latency and the client can't refuse it. Should be COULD, or gated behind feature negotiation.

As I noted previously, my measurements between ElectrumX 1.16 and ElectrumX 1.18 (the latter having implementation of timing delays) indicated that the time spent loading a small wallet doubled with this change. With ElectrumX 1.18 each call took a little over a second to complete, unless it was batched. This performance hit occurs regardless of whether it is necessary, for clients connecting to servers over a local LAN or using Tor, and regardless of whether they have other mitigations in place.

My recommendation would be to hold the timing delay recommendation, or downgrade it to COULD, until #7 lands the response-side options field and a delays feature definition. Protocol version 1.6 already specifies servers must ignore extraneous parameters in server.version, so the way has been prepared for this change if #7 was to be included in the 1.7 protocol update.

Either way, I believe the recommendation of timing delays must be balanced with the ability to control them from the client side.

[SomberNight]

@craigraw

one major concern: The new Traffic analysis section bundles two different things under one SHOULD:

• Padding: reasonably cheap and backwards-compatible, needs no handshake. Fine as a SHOULD in 1.7.
• Buffering / timing delay: costs interactive latency and the client can't refuse it. Should be COULD, or gated behind feature negotiation.

[...]
My recommendation would be to hold the timing delay recommendation, or downgrade it to COULD, until #7 lands the response-side options field and a delays feature definition.
[...]
Either way, I believe the recommendation of timing delays must be balanced with the ability to control them from the client side.

Changed in 1a58f6c to explicitly say SHOULD for padding and COULD for timing delays.

I agree that it would be desirable for the client to be able to influence the server's behaviour here via an opt-out, e.g. by negotiating a max timing delay. Still, I would postpone that for later.

[evoskuil]

We had recently added everything that was listed here and didn't encounter any problems. It appears that two mempool methods (mempool.recent and blockchain.transaction.testmempoolaccept) have been added since, and these also look fine.

https://github.com/libbitcoin/libbitcoin-server/tree/master/src/protocols/electrum

[romanz]

LGTM, thanks!

[t-bast]

It looks good from my point of view as well. I'll let @sstone take another look to make sure this won't create issues for Phoenix.

[tnull]

Thanks, mostly LGTM

Just a few comments regarding the padding. IMO it could make sense to just do it at the TCP level rather than for each individual message. And in general it might be good to prescribe one way to do it rather than leaving too many degrees of freedom, as all of them increase confusion and the risk of implementers end up doing something that makes them leak information to an observer.

[tnull]

Very much appreciate the push to preserve more privacy!

Note that padding on the transport layer could be more efficient. I.e. rather than padding each individual message, implementations could make sure they send out TCP frames of uniform lengths. Then, the only way missing would simply be a new variable-length padding message types (e.g. ping with a variable length payload) that allows to fill up remaining space whenever a message is to be sent that doesn't exactly fit into a multiple of the designated length value. We are now implementing this approach in Lightning land.

[SomberNight]

rather than padding each individual message, implementations could make sure they send out TCP frames of uniform lengths

The protocol supports multiple underlying transports: TCP, SSL, WS, WSS.
(The "TCP" and the "WS" transports are not encrypted so out-of-scope here.)
The most popular transport is probably SSL.

How much control does a userspace application have over the TCP frame when using e.g. the OpenSSL library?
At least when interacting with OpenSSL from the python stdlib, in our case, we don't have a lot of control.
What we have control over is what we feed into the SSL stream.

In the websocket case, it's probably even less control.

We are now implementing this approach in Lightning land.

We currently use SSL instead of BOLT-08 :(
With BOLT-08, indeed you have ~direct control of the TCP stream, but the situation seems to be more complicated here.

[tnull]

Tbh, this is a bit odd. If we want to make it optional going forward, maybe padding shouldn't be part of the spec until everything is ready?

[SomberNight]

Tbh, this is a bit odd. If we want to make it optional going forward, maybe padding shouldn't be part of the spec until everything is ready?

Fair enough, I see your point.

However, in this version, we want to extend "server.ping" so that it can be used to mimic traffic patterns in a generic way. That's the only thing that is new in 1.7.

All the rest of the traffic analysis text is basically independent of protocol versions and is just describing what Electrum Wallet and ElectrumX have already been doing for the past year or so. It describes one specific way of doing things and gives some recommendations. It is bundled as part of this PR because I felt that rationale was needed for why "server.ping" is being extended.

The opt-in/opt-out would be a new thing that's not there yet. It is mentioned as it would be useful and is somewhat important, however I would not hold up the rest of the changes for it. We need to do smallish incremental changes as otherwise - experience suggests - all progress is stalled.

[tnull]

Right, but why not make this the default way of doing padding? The more ambiguity the higher the chances will be that implementations will mess up or specific clients will stand out, i.e., still end up leaking data to observers.

[SomberNight]

Right, but why not make this the default way of doing padding?

Do you mean that instead of injecting whitespaces, why not send json-rpc "Batches" with one of the requests in the batch being a "server.ping" notification with a custom data length? Yes that works too. (note that the "server.ping" notification is a new addition in protocol 1.7. In older protocol versions there is ~nothing the server could send.)

It is trivial to implement padding by injecting extra whitespaces. It also has high granularity: you can add any number of bytes of padding, even just one byte.

Whitespace-injection can be done at any protocol version as json parsers just ignore it. It can even be done at a lower level, abstracted away from the electrum session logic. I like this approach the most.

Conceptually even "batching" could be done in two different ways: using the JSON-RPC "Batch" concept, or just using a buffer and sending out multiple JSON-RPC messages at the same time. The main difference is that using the JSON-RPC "Batch" concept forces the responder to similarly batch the responses.

So, yes you are right there is a high degree of freedom.

In any approach one might want to do timing delays because it allows adding more padding by amortising the cost of the padding bytes over more payload data. So batching (in whatever form) needs to be mentioned here.

The more ambiguity the higher the chances will be that implementations will mess up or specific clients will stand out, i.e., still end up leaking data to observers.

I think it's probably not realistic to make sessions indistinguishable across implementations. Already the choice of server a client communicates with typically gives a good clue regarding which client implementation is being used. The choice of server can be inferred either based on the IP address or from the hostname in the TLS Client Hello. If the sniffing is done close to the client, the number of simultaneous server connections is another fingerprinting vector: if it's more than one, it's probably Electrum Wallet.

[tnull]

It might be useful if this (as well as similar APIs like get_balance, etc) would include a field with the current tip hash. When we make multiple subsequent requests it's otherwise always hard to tell whether the chain moved while we're at it.

[ecdsa]

LGTM

[Davidson-Souza]

Concept ACK; thank you for adding the spk_hints to blockchain.outpoint.subscribe as well.