feat: let the sync operator set an outage traffic allowance by timwu20 · Pull Request #71 · canton-network/splice-multi-sync

timwu20 · 2026-10-05T20:28:38Z

Implements the FR-3 outage handling agreed on 2026-10-05 (design: ChainSafe/canton-extending-mainnet#142), in place of #61 and #66. Refs ChainSafe/canton-extending-mainnet#129.

During a global-synchronizer outage, the operator sets outage-traffic-allowance (a positive number of bytes) in the sync operator app's config and restarts the app, following the runbook. Once the global synchronizer is back, the operator removes it and restarts again. Setting or removing it only takes effect on a restart, and the app needs the configured Scan to start. Nothing detects the outage, and there is no Daml change.

Reconciling traffic by polling. On a dedicated synchronizer, the event-driven reconcile trigger is replaced by a polling one.
- Each poll reads the traffic state of every member on the operator's sequencer, and every purchased total in one store query.
- It sets each participant's limit to its purchased total, or 0 if it has none, plus the allowance while it's set, wherever the two differ.
- A participant that joins later, including during an outage, gets its limit on the next poll.
- Mediators keep the unlimited traffic MediatorUnlimitedTrafficTrigger gives them, and sequencers aren't charged traffic.
- Each grant reads the member's purchased total again just before it's sent, so a purchase granted in the meantime isn't taken back.
- Purchases are now granted on the next poll rather than as soon as they're ingested. Reading every member on each poll is fine for a dedicated synchronizer, but the global synchronizer keeps the event-driven path.
- A grant that isn't applied in time fails and is retried, as on the global synchronizer.
While the allowance is set, every participant has the full allowance on top of what it has bought, including ones that join while it's set, so onboarding should stay restricted meanwhile. Once it's removed, each limit goes back to the purchased total, and a participant that used the allowance is below zero until its purchases cover what it consumed. The validator top-up that buys that back is in feat: buy a member's shortfall with its top-up after an outage allowance is removed #80, stacked on this PR.
Store: listTotalPurchasedMemberTraffic sums the purchased totals per member on the operator's synchronizer in one query. Ingestion warns, once per contract, when it drops a purchase whose member id doesn't parse.

On a synchronizer whose sequencers several operators run, the operators set and remove the allowance together. Whenever their targets differ, as while only some have it set or while one operator's store hasn't ingested a purchase yet, limits only stay stable with a sequencer threshold above half the active sequencers. See the comment below.

Tests.

SyncOperatorTrafficIntegrationTest runs the whole flow:
- The operator's participant is disconnected from the global synchronizer, and the app restarts with a 50 KB allowance. Each limit rises by exactly the allowance, and a second restart sends no grant.
- Bob transacts past what was bought for him and is refused once less than one ping's worth of the allowance is left. Alice transacts on what was bought for her without touching the allowance.
- The operator's participant joins splitwell during the outage with no purchase on record, and gets the allowance alone.
- After reconnecting, a purchase for bob raises his limit to his new purchased total plus the allowance.
- With the allowance removed, the limits go back to the purchased totals: zero for the participant that joined. Alice keeps transacting, and bob, who used more than was bought for him, is refused.
- To change the config mid-test, a second instance of the app with the allowance set shares the first one's store and is started by hand. It's added in the test rather than in sync-operator-topology.conf, which other tests load.
SyncOperatorLsuIntegrationTest passes locally with the polling trigger.
Unit tests for the polling trigger's targets, completeTask and isStaleTask, the store query, and the ingestion warning.

The four-node BFT test runs this trigger with several operators, without the allowance. Testing an allowance rolled out across several operators is a follow-up.

moritzkiefer-da · 2026-10-12T14:15:32Z

+) extends PollingParallelTaskExecutionTrigger[Task] {
+
+  // Set once a poll finds every member at its target, after which polls do nothing.
+  private val settled = new AtomicBoolean(false)


I think you're missing an important case both by pausing here in settled but also in the case where total is not empty:

We need to support adding new members during an outage. Afaict you don't really handle this at all.

I'm wondering if for dsyncs we may actually be better off turning the reconcilaition trigger into a single polling trigger that first gets all members and then for each queries the daml state (defaulting to 0 if it's not set) and set that to daml state + outage param. I know I raised scalability concerns about that in the past but I think short term we can afford it for dsyncs (I'd like to avoid it for the gsync) as we're relatively unlikely to get thousands of members on the first few dsyncs.

Good catch. A member that joins during an outage gets nothing today, and since the trigger stops once it has settled, it wouldn't until the next restart.

I'll switch dedicated synchronizers to a single polling trigger as you describe. Each poll lists the sequencer's members and reads the purchased totals from the operator's store in one query, then sets each member's limit to its total (0 if it has none) plus the allowance, wherever the two differ. Mediators stay excluded, so it doesn't fight the unlimited-traffic trigger. It replaces both the event-driven reconcile trigger for dedicated synchronizers and the startup pass.

One consequence I want to make sure you're fine with: while the allowance is set, purchases no longer pay down the credit. Every member keeps the full allowance on top of what it has bought, and the shortfall settles once the allowance is removed, which the top-up change in this PR already handles.

One consequence I want to make sure you're fine with: while the allowance is set, purchases no longer pay down the credit. Every member keeps the full allowance on top of what it has bought, and the shortfall settles once the allowance is removed, which the top-up change in this PR already handles.

that seems like expected behavior to me.

Follows the review of canton-network/splice-multi-sync#71: the allowance applies to every participant, including ones that join while it is set, purchases while it is set raise the limit to the new total plus the allowance, and the app logs no warnings yet. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>

timwu20 · 2026-10-13T14:00:26Z

One thing the polling design depends on: with exact targets, the trigger lowers limits as well as raising them, and whether that's safe on a synchronizer whose sequencers several operators run depends on the sequencer group threshold in its topology (SequencerSynchronizerState), since that's the threshold SetTrafficPurchased grants aggregate on.

With one operator nothing changes. With several, honest operators send different limits for the same member whenever their configs differ, for example while they set or remove the allowance at different times. Say there are 4 sequencers with a threshold of 2 (f+1), and operators A and B have removed the allowance while C and D haven't:

A and B's grants lower the member's limit to its purchased total;
on their next poll, C and D's grants raise it back to the purchased total plus the allowance;
this repeats on every poll until all four configs agree, and a member that has used more than it bought is refused about half the time.

At a threshold of 1, which is what bootstrap.synchronizer sets and what dedicated synchronizers are bootstrapped with today, a single operator is enough to cause this. So is a single operator whose store hasn't ingested a purchase yet: it lowers that member below what it paid until its store catches up. The global synchronizer doesn't run into this because SV grants only raise and are computed from the same ledger data, so honest SVs never send conflicting values.

A sequencer threshold above half the sequencers (3 of 4) rules it out: two groups that disagree can't both reach it, so a limit only moves once a majority of operators have made the same change. I'd like to accept this for this PR, document the requirement for operators, and have the multi-node bootstrap set a majority sequencer threshold. Would bootstrapping a multi-operator dedicated synchronizer with a sequencer threshold of 3 of 4, instead of f+1, cause problems elsewhere? The same threshold is also used to verify sequencer group signatures.

The alternative would be to go back to raise-only, with a one-time lowering by the allowance when it's removed. That works at any threshold, but the app would have to persist the last allowance it applied.

Separately, 2aeaf75 has the fixes from another review pass, and the PR description is updated to match.

During a global-synchronizer outage, the operator sets outage-traffic-allowance in the sync operator app's config and restarts the app. On start, the app sets each member's limit to its purchased total plus the allowance, or back to the purchased total once it is removed. It warns while the allowance is set, when a purchase lands meanwhile, and for any limit that does not reach its target. The validator's top-up buys a member's shortfall together with its normal top-up in one purchase. Closes ChainSafe/canton-extending-mainnet#129 Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>

- Read a member's purchased total again before its grant, after the traffic state, so a purchase granted since the sweep retrieved its tasks is in the target instead of taken back until the next poll. - When the sweep settles with the allowance set, warn once about members with a purchase on record but no traffic state on the sequencer, which it leaves out. - Word the purchase warning for the DSO's merges as well, and log it once the grant succeeds, so a retried grant warns once. - Warn about a limit that could not be set at most once per warning interval for each member, with the same text at info in between. - Require a positive allowance and warning interval: 0 counted as set, and a 0 interval ran the periodic warning in a tight loop. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>

… shortfall The top-up bought the configured amount plus the whole shortfall or nothing, so a wallet that covered the top-up alone left a member below zero for good, where before it paid the shortfall down one interval at a time. A member below zero now gets the top-up alone when the wallet does not cover both, with a warning that it stays below zero for now. The insufficient-funds warning fires only when even the top-up alone does not fit. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>

…helper [ci] Alice now pings during the outage before the check that she ran on her own traffic, so it covers a member transacting rather than only her background traffic. The helper that drops the splitwell apps takes the validators to disconnect from splitwell, and this test uses it instead of repeating it with bob left connected. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>

- When even the top-up alone does not fit, the insufficient-funds warning also names the top-up and the shortfall together for a member below zero, so the wallet is funded in one round. - Both purchases are priced from one read of the rules and the open round, and minWalletBalanceForTopup is back to its signature on main. - A task whose target a grant has already met returns TaskNoop. - The warnings for a limit that could not be set say that operators may also not all have seen the same purchases yet. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>

…e custom warnings [ci] Per review: a member that joined during an outage got nothing, since the startup pass only covered members with a purchase on record and stopped once it had settled. The event-driven reconcile trigger and the startup pass are replaced by one polling trigger. Each poll reads the traffic state of every member on the sequencer and every purchased total, and sets each participant's limit to its purchased total, or 0, plus the allowance while it is set. Mediators and sequencers are left alone. The periodic allowance warning, the purchase warning and the other operator-side logs this PR added are removed, along with the warning-interval setting. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>

…up [ci] - Where a grant needs more than one sequencer, one that times out while the operators disagree is logged at info and sent again on the next poll, rather than retried until it logs an error. With a threshold of one, a timeout still fails as before. - The top-up budget is allocated in two passes: first purchases that leave their member able to submit, then fallbacks with what is left, so a fallback that leaves its member blocked cannot starve a later target. A task carries the extra traffic it buys, so what is priced and what is bought derive from one value. - The fallback warning only says the member cannot submit when the top-up leaves it at or below zero. - The integration test checks bob is refused only once he has less than one ping's worth of the allowance left, that one top-up buys exactly his shortfall with it, and that a restart sends no grant. - isCovered is private, and the base class's doc says only the SV trigger extends it now. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>

Per review, the handling for removing the outage traffic allowance moves to a separate PR, so this one no longer touches the validator's top-up trigger or TopupUtil: buying a member's shortfall with its top-up, the plain top-up fallback and the two-pass budget all go there, with an integration test for that scenario. The integration test here ends once removing the allowance sets the limits back and bob is refused. listTotalPurchasedMemberTraffic now warns when it skips a member id that does not parse. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>

- State the multi-operator requirement in the trigger and config docs: operators set and remove the allowance together, and a partial rollout only leaves limits stable when the sequencer threshold is above half the active sequencers. The config also notes the restart, the Scan dependency at startup, and that participants joining while the allowance is set get it too. - Only a grant that setSequencerTrafficControlState reports as not applied in time is skipped until the next poll, where it needs more than one sequencer; a failed call to the sequencer goes through the normal failure handling. A new attemptSetSequencerTrafficControlState returns that outcome as a status, and setSequencerTrafficControlState behaves as before. - Warn at ingestion, once per contract, when a purchase is dropped because its member id does not parse; the total query no longer has a warning that could not fire. - Unit tests for completeTask and isStaleTask, and a store test for the dropped purchase. - Scope the outage test's log suppression to the splitwell validator's expected warnings and bob's refused pings. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>

moritzkiefer-da · 2026-10-14T15:51:36Z

+      current <- currentTask(task)
+      limit = trafficState.extraTrafficLimit.value
+      outcome <-
+        if (current.isDone(limit)) Future.successful(TaskNoop)


shouldn't this get handled by the staleness check? why are we setting this here

Removed. It only saved a redundant grant when the member reached its target between the poll and the grant. The staleness check only runs after a failure, but a redundant grant is harmless.

moritzkiefer-da · 2026-10-14T15:53:41Z

+                s"Set the traffic limit of ${task.member} from $limit to ${current.target}"
+              )
+            )
+            // Where a grant needs more than one sequencer, it is not applied until enough of the


this makes no sense to me, why do we need this here? this seems like the exact same setup as for the gsyncs where we do not have this.

Fair, removed. A grant that doesn't land now fails and is retried, as on the global synchronizer.

moritzkiefer-da · 2026-10-14T15:54:48Z

  ): Future[SequencerAdminConnection] =
-    synchronizerNodeService.sequencerAdminConnection()
+    for {
+      connection <- synchronizerNodeService.sequencerAdminConnection()


why are we rechecking the synchronizer id in this trigger? if anything I would expect we check that the dsync synchronizer id matches once on startup, checking that in some random places later just seems fragile.

Removed. The store only holds purchases for this synchronizer, so the trigger doesn't need it.

moritzkiefer-da · 2026-10-14T15:55:07Z

-  ): Future[Either[String, Long]] =
-    // No prior consumption to carry.
-    Future.successful(Right(0L))
+  /** Whether this trigger sets the member's limit. Only participants: a mediator has no purchase


shouldn't this be shared between gsync and dsync?

Do you mean the rule for which members get skipped? The SV trigger already skips the members it grants unlimited traffic, its SV nodes, through trafficLimitOffset. The equivalent here would be the synchronizer's mediators, which MediatorUnlimitedTrafficTrigger grants unlimited traffic. I can share one hook for "members granted unlimited traffic" between the two triggers and skip those, instead of filtering by member type. Should I do that in this PR, or did you have something broader in mind, like sharing the reconcile step itself?

…stale re-read [ci] The trigger and config docs now say limits only stay stable with a sequencer threshold above half the active sequencers whenever the operators' targets differ, which includes one operator's store not having ingested a purchase yet, not just a partial allowance rollout. Above half, a split where neither side reaches the threshold leaves the limits as they are until enough operators agree. Also: a test that isStaleTask re-reads the total, clearer test names, and the ingestion warning's comment says the row encoder gets no trace context. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>

…er [ci] Per review: completeTask always sends the grant for the target computed from the member's current total, without an early return when it is already there; a grant that is not applied in time fails and is retried, as on the global synchronizer, so SequencerAdminConnection is back to main; and the trigger no longer checks on every call which synchronizer the sequencer serves, as the store only holds purchases for this one. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>

…nce (#142) * docs: design for sequencing a dedicated synchronizer through a global-synchronizer outage (FR-3) Signed-off-by: Timothy Wu <tim.wu@chainsafe.io> * docs: record the FR-3 decision, an operator-set outage traffic allowance Signed-off-by: Timothy Wu <tim.wu@chainsafe.io> * docs: one allowance for every member, set exactly on each restart Signed-off-by: Timothy Wu <tim.wu@chainsafe.io> * docs: update the FR-3 decision for the polling implementation Follows the review of canton-network/splice-multi-sync#71: the allowance applies to every participant, including ones that join while it is set, purchases while it is set raise the limit to the new total plus the allowance, and the app logs no warnings yet. Signed-off-by: Timothy Wu <tim.wu@chainsafe.io> --------- Signed-off-by: Timothy Wu <tim.wu@chainsafe.io>

timwu20 marked this pull request as ready for review October 7, 2026 01:26

timwu20 requested a review from moritzkiefer-da October 7, 2026 01:27

This was referenced Oct 7, 2026

docs: design for sequencing a dedicated synchronizer through a global-synchronizer outage (FR-3) ChainSafe/canton-extending-mainnet#141

Closed

docs: record the FR-3 decision, an operator-set outage traffic allowance ChainSafe/canton-extending-mainnet#142

Merged

moritzkiefer-da reviewed Oct 12, 2026

View reviewed changes

timwu20 requested a review from moritzkiefer-da October 13, 2026 14:04

moritzkiefer-da reviewed Oct 13, 2026

View reviewed changes

timwu20 mentioned this pull request Oct 14, 2026

feat: buy a member's shortfall with its top-up after an outage allowance is removed #80

Draft

timwu20 added 8 commits October 14, 2026 09:49

timwu20 force-pushed the feat/outage-traffic-allowance branch from 88e43a6 to 6f76e28 Compare October 14, 2026 13:54

moritzkiefer-da reviewed Oct 14, 2026

View reviewed changes

timwu20 added 2 commits October 14, 2026 12:18

timwu20 requested a review from moritzkiefer-da October 14, 2026 18:17

Conversation

timwu20 commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timwu20 commented Oct 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timwu20 Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

timwu20 commented Oct 5, 2026 •

edited

Loading

timwu20 Oct 14, 2026 •

edited

Loading