Leave the consumer group explicitly on shutdown

[delthas]

Stacked on #2818 (BB-835). Review that one first — this branch contains its commit, and the base will move to development/9.5 once it merges.

close() unsubscribed and then waited for the rebalance callback to un-assign before disconnecting. librdkafka delivers no such callback when the consumer holds no assignment, and postpones the unsubscribe outright while a rebalance is in progress — so close() never returned and the pod was SIGKILLed with the member still registered at the broker.

sequenceDiagram
    participant C as BackbeatConsumer
    participant K as Kafka
    Note over C: SIGTERM during a rebalance
    C->>K: unsubscribe()
    Note right of K: postponed — a rebalance<br/>is already in progress
    C->>C: wait for 'unassign' … forever
    Note over C: SIGKILL at the grace period,<br/>no LeaveGroup ever sent
    Note over K: member still registered,<br/>may be elected leader of the next<br/>generation and never SyncGroup

Loading

The group then holds zero partitions until session.timeout.ms (45 s) evicts the member. During a rolling update a rebalance is in progress essentially by construction, since the new pod joins before the old one is told to stop.

Changes

Release the partitions and drop the subscription before closing, so the close path has nothing to hand back to us and the LeaveGroup goes out whatever state the group is in:

before   unsubscribe → wait for a revoke callback → [ drain → commit → unassign ] → disconnect → wait 'disconnected'
after    drain → commit → unsubscribe → unassign → disconnect → wait 'disconnected'

The bracketed steps only ran if librdkafka delivered a revoke callback. It delivers none when the consumer holds no assignment, and postpones the unsubscribe outright while a rebalance is in progress — in both cases the wait never ends. The same steps now run unconditionally, in close() itself.

The order of the last two matters. rd_kafka_cgrp_unassign() does not clear rkcg_group_assignment, so un-assigning first is inert as far as the group state machine is concerned; unsubscribe() then fires a revoke at us and parks in WAIT_UNASSIGN_CALL, leaving the LeaveGroup gated on a callback round trip that disconnect() is simultaneously blocking on. Unsubscribing first puts us in the one join-state where unassign() is meaningful, so our own call completes it and sends the LeaveGroup before disconnect() is reached.

• A revoke arriving once the shutdown has started is left for close() to answer. Releasing the partitions there would cut short the drain close() is waiting on and strand the offsets that drain exists to commit (BB-758).
• A deferred un-assign from before the shutdown is treated as superseded, and a partition grant arriving mid-close no longer tears down the drain close() installed. Both previously left close() waiting on a callback that could never fire.
• close() no longer waits on an in-flight backlog publish, which could otherwise keep it rescheduling itself every second indefinitely.
• The final disconnect is bounded: once the group has been left, a client that will not finish tearing down must not hold the process.

In-flight work is still drained before the partitions are released, so offsets are committed exactly as before, and that wait keeps the bound it already had through the revoke path (max.poll.interval.ms - 1000) — a wedged task delays the departure no longer than it does today. Draining is skipped once the client is disconnected, since there is then nothing to commit and no partitions to give back.

That bound is inherited, not chosen: at the default max.poll.interval.ms it is ~299 s, far longer than a pod's grace period, so a wedged task is still killed rather than departing cleanly. Replacing it with a deadline derived from the grace period is the budget work, deliberately left out here — BB-854 shortens the drain first.

Verification

Unit tests for the call ordering, completion with no assignment held, the drain wait, the offset-publish skip, watchdog cleanup, and a revoke arriving mid-drain. Each was checked against the previous implementation to confirm it fails there.

Beyond that, a pod-level harness against a real broker (Kafka 3.9.0), terminating consumers in the two states that matter — waiting to rejoin after a revoke, and mid-drain with work in flight:

state	before	after
waiting to rejoin	hung 4/4	0/6
mid-drain	hung 1/6	1/6

LeaveGroup reached the coordinator in 11/12 iterations. One R3 leave took 38 s — not this path: an orphaned member id (BB-843) held the group in PreparingRebalance first, which is the other half of the same stall.

The remaining mid-drain hang is the drain wait itself: the harness sees five tasks whose callbacks never fire, so close() waits out its bound. That bound is ~299 s at the default max.poll.interval.ms, far longer than a pod's grace period, so such a shutdown is still killed as a ghost. Whether those tasks are genuinely stuck or an artefact of the harness is unresolved — but either way it is the drain budget, not the departure, that is exposed, and this is the strongest argument yet for doing that work next.

Issue: BB-833

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.56%. Comparing base (fa04a32) to head (882ec92).

Additional details and impacted files

Files with missing lines	Coverage Δ
lib/BackbeatConsumer.js	94.63% <ø> (-0.22%)	⬇️
lib/constants.js	100.00% <ø> (ø)

Components	Coverage Δ
Bucket Notification	80.27% <ø> (ø)
Core Library	81.44% <ø> (-0.06%)	⬇️
Ingestion	70.09% <ø> (ø)
Lifecycle	80.46% <ø> (ø)
Oplog Populator	85.83% <ø> (ø)
Replication	62.04% <ø> (ø)
Bucket Scanner	85.76% <ø> (ø)

@@                          Coverage Diff                           @@
##           improvement/BB-835/rebalance-guard    #2819      +/-   ##
======================================================================
- Coverage                               75.59%   75.56%   -0.03%     
======================================================================
  Files                                     200      200              
  Lines                                   13937    13920      -17     
======================================================================
- Hits                                    10536    10519      -17     
  Misses                                   3391     3391              
  Partials                                   10       10              

Flag	Coverage Δ
bucket-scanner	85.76% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.