Skip to content

Transport changes cancel all IO loops, retire affected transports cooperatively instead #8513

Open

Description

@hpk42

Any transport change (synced add or removal, delete_transport(), ...) currently restarts the whole IO scheduler. Scheduler::stop()
races every loop future against a cancellation token, so the restart drops IMAP loops at arbitrary await points and reconnects every transport even when only one changed.

#8499 moved the restart trigger to a safer point: the fetch loop that processed the transport sync message defers the restart until its iteration's persistence is committed. That fixed losing the sync message's own remaining processing, but it is not a complete solution: The restart still cancels the loops of the other transports at arbitrary points because only the triggering loop's iteration is protected.

On hard-cancellation all connections are torn down and rebuilt for every transport change and almost all await points in the IO loops must reckon with getting arbitrarily cancelled when transport config changes, producing hard to replicate and hard to debug problems, apart from potentially user visible degradations.

I suggest to reconcile transports instead of stop-and-restarting-all transports:

  • just spawn loops for added transports,

  • retire loops of removed or reconfigured transports (via a token that each loop checks at iteration boundaries).

  • IMAP fetch loops finish (or fail) on processing their current message and then gracefully exit when the retire-token is present.

  • When we need to reconfigure or restart a transport, we can probably retire the current and concurrently start a new IMAP fetch loop already.

  • nothing is cancelled at all, unaffected transports keep their connections, and the deferral flag becomes unnecessary.

Note that the SMTP loop singleton needs no real teardown: on a retire-signal it can rebind to the current primary.

Hard cancellation remains only for stop_io() and pause(), where tearing everything down is the actual intent. Although we could probably also try to stop cooperatively for a few seconds before hard-cancel.

Activity

  1. added
    bugSomething is not working
    on Aug 1, 2026
  2. Hocuri commented on Aug 1, 2026

    @Hocuri
    Collaborator

    This is actually two steps, which we could also implement one by one rather than both at once:

    • Stop loops cooperatively
    • Add machinery to start and stop loops independently of each other, rather than restarting everything on a transport change

    We often need to stop loops, but the transports don't change that often, so the first step seems more important than the second one.

    FTR, an alternative would be to make sure that receive_imf() runs to the end if the future is cancelled:

    • Make SQL calls non-async by using non-async Mutexes and Rwlocks structs (i.e. the ones from std rather than tokio). We're already using block_in_place() to wrap the calls to sqlite, we would just need to expand its scope to also wrap any calls to lock(), read(), write(). I'm not sure if that would actually work, but in this way, the whole of receive_imf()` would become non-async. Then, it couldn't be cancelled in the middle but would just run to completion when the future is dropped.
      • this would make it possible to put the whole of receive_imf() into one SQL transaction by using a thread-local transaction variable, and using it in Sql::call() if it's set. Today we can't do that because we can't hold a transaction over an await point. I'm not sure whether this actually works and is a good idea, though.

    Another alternative would be to somehow mark the message as not fully processed when writing it into the database, and then marking it as fully processed only at the very end of receive_imf(). And then download not-fully-processed messages again when the imap loop is started again.

    I find it hard to judge which of these three options is best.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is not working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions