Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 0 additions & 31 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,6 @@
ARG UBI_VERSION=10.3
ARG FDB_VERSION=7.3.69
ARG UV_VERSION=0.13.6
# Toolchain of the two-node HA event collector (tools/event-collector). The Go
# toolchain links its own stdlib statically into the built binary, so this is
# also what the image's scanned Go stdlib version is (grype flags it as
# `stdlib`, package type go-module) — bump it whenever a scan flags a stdlib
# CVE with no fix in 1.24.x, same as the CACHE_KEY-driven OS-package refresh
# does for dnf packages. go.mod's `go 1.24` line is an unrelated minimum
# language-version requirement, not a toolchain pin, so any newer toolchain
# here keeps building it fine.
ARG GO_VERSION=1.27.4

# The application interpreter, free-threaded ("t" suffix). Managed by uv rather
# than dnf: UBI carries no free-threaded build, and tying the runtime to the
Expand Down Expand Up @@ -49,7 +40,7 @@

ARG TARGETARCH
ARG FDB_VERSION
ARG CACHE_KEY

Check warning on line 43 in docker/Dockerfile

View workflow job for this annotation

GitHub Actions / build

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "CACHE_KEY") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

# CACHE_KEY is referenced here only to tie it into this layer's cache key; the
# layers below inherit the invalidation from this one.
Expand Down Expand Up @@ -201,33 +192,11 @@
RUN rm -rf /opt/venv/lib/python*/site-packages/kmip/demos


# The two-node HA event collector (tools/event-collector, docs/design/two-node-
# arbitration.md section 4): a Go binary linked against the FoundationDB client
# library the base stage installs, so cgo and not static. Its own stage keeps
# the compiler and the Go toolchain out of the runtime image.
FROM base AS event-collector

ARG TARGETARCH
ARG GO_VERSION

RUN set -eux; \
dnf install -y gcc && dnf clean all; \
curl -fsSL "https://go.dev/dl/go${GO_VERSION}.linux-${TARGETARCH}.tar.gz" | tar -C /usr/local -xz

WORKDIR /src
COPY tools/event-collector/go.mod tools/event-collector/go.sum ./
RUN /usr/local/go/bin/go mod download
COPY tools/event-collector/ ./
RUN CGO_ENABLED=1 /usr/local/go/bin/go build -tags fdb -trimpath -ldflags="-s -w" \
-o /out/sb-event-collector .


FROM base AS runtime

COPY --from=builder /opt/python /opt/python
COPY --from=builder /opt/venv /opt/venv
COPY --chown=simplyblock:simplyblock . /app
COPY --from=event-collector /out/sb-event-collector /usr/local/bin/sb-event-collector

# The services log through the container's gelf log driver, and a buffered
# stdout loses the tail of the log when a process dies.
Expand Down
12 changes: 6 additions & 6 deletions docs/design/two-node-arbitration.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ campaign c1 (no preferred node; bookkeeping, explicit-sync, fence-port, SPDK-dow
rules). Design rationale, diagrams and failure analysis:
[`two-node-ha-design.pdf`](two-node-ha-design.pdf) (revision 2; where it describes a preferred
node, this document wins). This document is the binding contract between the work packages:
ultra (JC, distrib), the SPDK fork (lvol, blobstore), sbcli (arbiter, Go event collector, API)
ultra (JC, distrib), the SPDK fork (lvol, blobstore), sbcli (arbiter, event collector, API)
and the simplyblock operator.

Scope: clusters whose storage-node membership is exactly two (`apply_jc_dual_node` in
Expand Down Expand Up @@ -204,11 +204,11 @@ Capability probe: the CP calls `rpc_get_methods`; the protocol is used only if i
`jc_unfence`. Older nodes keep today's behaviour and are read with `distr_status_events_get`
every 1 s.

## 4. Collector (sbcli, Go)
## 4. Collector (sbcli)

- Runs in the CP image as its own process, one goroutine per storage node of each cluster with
`two_node_arbitration` on.
- Each goroutine loops on `jc_wait_events`; on transport errors it reconnects with backoff
- Runs in the CP image as its own process (`simplyblock_core/services/two_node_event_collector.py`),
one thread per storage node of each cluster with `two_node_arbitration` on.
- Each thread loops on `jc_wait_events`; on transport errors it reconnects with backoff
(100 ms, doubling, max 5 s) and re-sends the last `instance` and `after_seq`.
- On `instance` change or `overflow: true`: call `jc_ha_status`, emit a synthetic
`ha_resync` event carrying the full status to the arbiter, continue from the returned `seq`.
Expand Down Expand Up @@ -757,7 +757,7 @@ Kubernetes hosts run the kernel defaults instead of these values until the drive
| ultra | Hold trigger in dual-node mode; lease and epoch; `jc_grant_solo`, `jc_fence`, `jc_unfence`, `jc_lease_renew`, `jc_ha_status`, `jc_wait_events`; event fields and new statuses; resume gating after JCERR |
| spdk fork | Hold on JC's signal through the frozen-IO queue (`blobstore.c:blob_execute_queued_io`); path error instead of EIO on fence; no rule removal while fenced (`lvol.c:spdk_lvs_remove_rules_poller`) |
| sbcli | Arbiter service, `ClusterArbitration` model, API, verdict RPC client methods, healing flow, feature flag; RPC client methods for the new RPCs |
| sbcli | Go collector (section 4), packaged in the CP image |
| sbcli | Collector (section 4), packaged in the CP image |
| operator | Taint handling, remediation status, CR fields |
| test | Unit (arbiter table, epochs, lease arithmetic, long-poll sequencing); two-SPDK tests with tc/iptables; bed scenarios: storage link cut, uplink cut, both, power-off, SIGSTOP, CP restart mid-hold, latency injection; OpenShift two-node with arbiter and with fencing |

Expand Down
4 changes: 2 additions & 2 deletions docs/design/two-node-kubernetes-fencing.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ survivor then provably runs alone. The storage never decides; it waits for that
### 4.2 Control plane

- The arbiter does not act on the cluster at all: no leases, epochs, grants, fences, unfences,
heals or leadership restoration. The Go collector does not poll its nodes (the node agent owns
heals or leadership restoration. The collector does not poll its nodes (the node agent owns
the single `jc_wait_events` waiter), legacy remote-JM events are not queued for the arbiter, and
no planned-loss markers are written.
- `jc_set_dual_node` still carries the full arbitration config on every call (node add, restart
Expand Down Expand Up @@ -322,7 +322,7 @@ Read-only; the agent never asks Pacemaker to fence. In order of preference:
| sbcli | settings, `client_host_nqns`, `host_fences`, API (section 3.5) | both |
| sbcli | host fence lifecycle in the arbiter (section 3.2) | witness |
| sbcli | winner policy for partitions (section 3.3) | witness |
| sbcli | `pacemaker` gates: arbiter, Go collector, event queue, planned loss (section 4.2) | pacemaker |
| sbcli | `pacemaker` gates: arbiter, collector, event queue, planned loss (section 4.2) | pacemaker |
| operator | fence agent DaemonSet, `StorageFenced`, `StorageFenceRemediation(Template)` CRDs + controller, NHC sample, release protocol | both |
| operator | node agent duties of section 4.3, Pacemaker readers of section 4.5 | pacemaker |
| ultra + SPDK fork | grant lifts a self-fence (section 4.4) | pacemaker |
Expand Down
2 changes: 1 addition & 1 deletion simplyblock_core/arbitration/arbiter.py
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,7 @@ def consume_events(self, cluster_id: str) -> int:
legacy = item.instance == ev.LEGACY_INSTANCE
if legacy and self.capable.get(item.node_id):
# JC pushes a remote-JM event into both queues; on a node with
# jc_wait_events the Go collector already delivered it.
# jc_wait_events the event collector already delivered it.
item.remove(self.db.kv_store)
continue
sig = self.signals.setdefault(item.node_id, ev.NodeSignals())
Expand Down
4 changes: 2 additions & 2 deletions simplyblock_core/arbitration/events.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,8 +128,8 @@ def forward_legacy_jm_event(db, cluster, node_id: str, event_dict: dict) -> bool
the Python distr collector to the arbiter.

The distr event queue is consumed destructively by that collector
(``distr_status_events_discard_then_get``), so the Go collector must not
read it as well; this is the fallback path instead. The events carry no
(``distr_status_events_discard_then_get``), so the two-node event collector
must not read it as well; this is the fallback path instead. The events carry no
sequence, so the receive time in ns orders and identifies them.
Returns True when the event was a remote-JM event (handled here).
"""
Expand Down
7 changes: 6 additions & 1 deletion simplyblock_core/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -1467,11 +1467,16 @@ def get_config_var(name, default=None):
TWO_NODE_LEASE_MARGIN_MS = 500
TWO_NODE_STABLE_FOR_S = 30
TWO_NODE_WAIT_TIMEOUT_MS = 20000
#: jc_wait_events max events per answer (contract section 3.7).
TWO_NODE_COLLECTOR_MAX_EVENTS = 256
#: How often the collector re-reads cluster/node topology to start or stop
#: per-node collector threads.
TWO_NODE_COLLECTOR_RECONCILE_S = 5
TWO_NODE_VERDICT_HISTORY = 50
#: Arbiter decision loop period (events are folded and decided every tick).
TWO_NODE_DECISION_TICK_MS = 100
#: With no cluster that has ``two_node_arbitration`` on, the arbiter (and the
#: Go event collector) re-scan the cluster list only this often instead of every
#: event collector) re-scan the cluster list only this often instead of every
#: tick, so deployments without two-node clusters carry almost no FDB load.
TWO_NODE_IDLE_RESCAN_S = 30
#: How often the arbiter re-reads each node's ``jc_ha_status`` and replaces its
Expand Down
2 changes: 1 addition & 1 deletion simplyblock_core/models/arbitration.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ def lease_expires_at(self, node_id: str) -> int:


class ArbitrationEvent(BaseModel):
"""One HA event from a node, queued by the Go collector for the arbiter.
"""One HA event from a node, queued by the event collector for the arbiter.

Key ``{cluster_id}/{node_id}/{instance}/{seq:020d}``: a prefix read per
cluster returns them in order per node, and a redelivered event (at-least-
Expand Down
2 changes: 1 addition & 1 deletion simplyblock_core/scripts/docker-compose-swarm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ services:
TwoNodeEventCollector:
<<: *service-base
image: $SIMPLYBLOCK_DOCKER_IMAGE
command: "/usr/local/bin/sb-event-collector"
command: "python3 simplyblock_core/services/two_node_event_collector.py"
networks:
- hostnet
deploy:
Expand Down
2 changes: 1 addition & 1 deletion simplyblock_core/services/two_node_arbiter.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
- the lease loop renews every arbitrated node's lease every
``TWO_NODE_LEASE_RENEW_MS`` (its own thread, so a slow decision never delays
a renewal);
- the decision loop folds the events the Go collector queued in FDB, decides
- the decision loop folds the events the event collector queued in FDB, decides
and applies verdicts every ``TWO_NODE_DECISION_TICK_MS``.

Only clusters with ``two_node_arbitration`` on and exactly two members are
Expand Down
184 changes: 184 additions & 0 deletions simplyblock_core/services/two_node_event_collector.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
"""Two-node HA event collector (docs/design/two-node-arbitration.md, section 4).

One thread per storage node of every cluster with ``two_node_arbitration`` on,
long-polling ``jc_wait_events`` and queuing each event for the arbiter
(``arbitration.events.queue_event``) as an ``ArbitrationEvent`` in FDB. The
connection is always opened by the control plane; nodes never call back.
"""

import threading
import time

from simplyblock_core import constants, db_controller, utils
from simplyblock_core.arbitration import events as arbitration_events
from simplyblock_core.arbitration.arbiter import witness_authority
from simplyblock_core.models.storage_node import StorageNode
from simplyblock_core.rpc_client import RPCException

utils.init_sentry_sdk()
logger = utils.get_logger(__name__)

db = db_controller.DBController()

#: Backoff on a transport error: starts here, doubles, caps at BACKOFF_MAX_S.
BACKOFF_MIN_S = 0.1
BACKOFF_MAX_S = 5.0

#: Synthetic event carrying a full jc_ha_status, queued on resync.
STATUS_RESYNC = "ha_resync"


def _resync(client, cluster_id: str, node_id: str, reported_instance: str, reason: str) -> tuple[str, int]:
"""Node restart (instance change) or a dropped event (overflow): fetch the
full ``jc_ha_status`` and queue it as one ``ha_resync`` event, so the
arbiter's view is replaced rather than built back up one event at a time.
Returns the ``(instance, seq)`` to continue the long-poll from.
"""
status = client.jc_ha_status() or {}
instance = str(status.get("instance") or reported_instance)
seq = int(status.get("seq", 0))
payload = {"status": STATUS_RESYNC, "instance": instance, "seq": seq,
"reason": reason, "ha_status": status}
arbitration_events.queue_event(
db, cluster_id, node_id, instance, seq, payload, int(time.time() * 1000))
logger.info(f"Node {node_id} resynced ({reason}): instance={instance} seq={seq}")
return instance, seq


def _poll_once(client, cluster_id: str, node_id: str, instance: str, after_seq: int) -> tuple[str, int]:
"""One jc_wait_events call; queues what it returns. Raises on a transport
error so the caller can back off and retry with the same (instance,
after_seq) -- at-least-once delivery, contract section 3.7."""
res = client.jc_wait_events(
instance, after_seq,
timeout_ms=constants.TWO_NODE_WAIT_TIMEOUT_MS,
max_events=constants.TWO_NODE_COLLECTOR_MAX_EVENTS) or {}

if res.get("superseded"):
# Another waiter took over (e.g. this collector restarted elsewhere).
return instance, after_seq

res_instance = str(res.get("instance", ""))
overflow = bool(res.get("overflow"))
if res_instance != instance or overflow:
reason = "overflow" if overflow and res_instance == instance else "instance"
return _resync(client, cluster_id, node_id, res_instance, reason)

events = res.get("events") or []
received_at = int(time.time() * 1000)
for event in events:
seq = int(event.get("seq", 0))
if seq <= after_seq:
continue
event.setdefault("instance", instance)
arbitration_events.queue_event(db, cluster_id, node_id, instance, seq, event, received_at)
after_seq = seq

reported_seq = int(res.get("seq", 0))
if reported_seq > after_seq and not events:
# An empty answer reports the node's current seq: nothing newer is pending.
after_seq = reported_seq
return instance, after_seq


def _collect(node_id: str, cluster_id: str, stop: threading.Event) -> None:
logger.info(f"Starting two-node event collector on node: {node_id}")
instance, after_seq = "", 0
backoff = BACKOFF_MIN_S
while not stop.is_set():
try:
snode = db.get_storage_node_by_id(node_id)
except KeyError:
logger.info(f"Node {node_id} deleted; stopping two-node event collector")
return
if snode.status == StorageNode.STATUS_REMOVED:
logger.info(f"Node {node_id} removed; stopping two-node event collector")
return

try:
client = snode.rpc_client(retry=1)
instance, after_seq = _poll_once(client, cluster_id, node_id, instance, after_seq)
backoff = BACKOFF_MIN_S
except RPCException as e:
logger.warning(f"jc_wait_events failed on {node_id}: {e}; retry in {backoff:.1f}s")
stop.wait(backoff)
backoff = min(backoff * 2, BACKOFF_MAX_S)
except Exception as e:
# A store write (arbitration_events.queue_event) can fail for reasons
# that have nothing to do with the RPC transport (FDB unavailable,
# a conflict). after_seq is only advanced once the write that
# covers it succeeds (see _poll_once), so retrying here with the
# same (instance, after_seq) keeps at-least-once delivery instead
# of letting the thread die -- the collector would otherwise sit
# dead until the next _reconcile tick restarts it from scratch.
logger.warning(f"Collector error on {node_id}: {e}; retry in {backoff:.1f}s")
stop.wait(backoff)
backoff = min(backoff * 2, BACKOFF_MAX_S)

logger.info(f"Stopping two-node event collector on node: {node_id}")


#: node_id -> (thread, stop event) for every running collector.
_workers: dict[str, tuple[threading.Thread, threading.Event]] = {}


def _reconcile() -> None:
"""Start a collector thread per node of every arbitrated cluster; stop any
whose node is gone, removed, or no longer in an arbitrated cluster."""
want: dict[str, str] = {}
try:
# Mode pacemaker (docs/design/two-node-kubernetes-fencing.md, section
# 4.2): the node agent owns the single jc_wait_events waiter and this
# collector leaves the cluster alone.
clusters = [c for c in db.get_clusters() if c.two_node_arbitration and witness_authority(c)]
except Exception as e:
logger.error(f"Failed to get clusters: {e}")
return
for cluster in clusters:
cluster_id = cluster.get_id()
try:
nodes = db.get_storage_nodes_by_cluster_id(cluster_id)
except Exception as e:
logger.error(f"Failed to get nodes for cluster {cluster_id}: {e}")
continue
for snode in nodes:
if snode.status == StorageNode.STATUS_REMOVED or not snode.mgmt_ip or not snode.rpc_port:
continue
want[snode.get_id()] = cluster_id

for node_id in [n for n in _workers if n not in want]:
thread, stop = _workers.pop(node_id)
stop.set()
thread.join(timeout=constants.TWO_NODE_WAIT_TIMEOUT_MS / 1000.0 + 15)
logger.info(f"Stopped two-node event collector for node: {node_id}")

for node_id, cluster_id in want.items():
existing = _workers.get(node_id)
if existing is not None and existing[0].is_alive():
continue
stop = threading.Event()
thread = threading.Thread(
target=_collect, args=(node_id, cluster_id, stop), daemon=True,
name=f"two-node-collector-{node_id}")
thread.start()
_workers[node_id] = (thread, stop)


def _next_wait_s() -> float:
"""``TWO_NODE_COLLECTOR_RECONCILE_S`` while any collector thread runs,
``TWO_NODE_IDLE_RESCAN_S`` while none do -- deployments without two-node
clusters are scanned rarely instead of on every tick."""
if _workers:
return constants.TWO_NODE_COLLECTOR_RECONCILE_S
return constants.TWO_NODE_IDLE_RESCAN_S


def main() -> None:
logger.info("Two-node event collector started")
while True:
_reconcile()
time.sleep(_next_wait_s())


if __name__ == "__main__":
main()
Loading
Loading