fix(reputation,qos,gateway): health-check bench loop, circuit-breaker gate, batch responses, plus cardinality/solana/archival follow-ups

[oten91]

Follow-up to #527. The branch grew past its original scope: it now carries the F5/F6
cardinality follow-up plus four other lines of work that were deployed and validated
together over the last two weeks.

Every commit here is live on canary and mainnet as sha-040ae42-rc and has been
validated in production (earlier cuts of the branch ran as sha-064bc62-rc and
sha-45e00a2-rc; sections 6–8 below cover what landed since). That is the main argument for reviewing it as one unit —
these commits were never exercised separately, so splitting them would put untested
combinations into main.

---

1. Metric cardinality (the branch's original purpose)

e3f5c7e2 — bound histogram labels, remove supplier from aggregate metrics.

Two rounds of a production cardinality incident converged on one rule: only a label's
value set bounds it. A sanitizer bounds a value's shape, never the set. A cardinality
guard bounds the live registry, never the number of distinct series Prometheus retains —
path_supplier_signal_total sat at 26% of its cap while being one of the two largest
series sources in the job (6,523 tuples live in 10 minutes against 60,674 distinct over one
pod's 7.7h life).

A label on a histogram costs roughly 12× what it costs on the counter beside it.
path_relay_latency_seconds_bucket was 31.6% of all gateway series because it carried
status_code × reputation_signal, a 20× pair no dashboard ever queried from the
histogram. Outcome taxonomy now lives on the counter; the histogram keeps topology labels.

Per-supplier questions are served by GET /ready/<service>?detailed=true — a point lookup
instead of ~74K retained timeseries. Three metrics keep supplier deliberately, where the
address is the actionable payload rather than a way of naming an operator.
Test_SupplierLabelIsGone enforces the rest.

2. Solana sync allowance and health checks

2304bc9b 2d49d8e5 ad729c3f 5c1c8d60 0e856cc4 068ff990

Solana's ValidateEndpoint had no sync allowance at all — sync_allowance: 750 was
configured but never implemented. During the 2026-08-18 surge (200 → 2500 rps, ~2.5
blocks/s) only the freshest endpoint stayed valid, producing near-total lock-in on one
operator while others sat at score 100 with no traffic. Health checks cannot correct this;
they run at the same per-endpoint rate on every operator.

Also here: kava CometBFT checks routed to comet_bft, slower xrplevm websocket probes, a
missing solana health observation no longer scored as a fault, and relays that never
received an HTTP status no longer counted as successes.

3. Go, dependencies, CI

d3e0273c e6e5530e fa146acd c10e5be6 — poktroll v0.1.35, Go directive to 1.26.6 for
the stdlib security fixes, drop the libsecp CGO variant, build release platforms
concurrently.

The Go bump takes govulncheck from 12 reachable findings to 6, all of which are
Fixed in: N/A (three withdrawn lib/pq advisories, openpgp, two cosmos x/crisis init-only).
The net/http CVE was genuinely reachable via router.go ListenAndServe. The Dockerfile
Go patch is deliberately left floating so future patches are picked up; the directive sets
the floor.

4. Heuristic and reputation detectors

dd710d38 4931c6f4 6ef6ca1a 22e9dff1

An endpoint returning zero-length payloads at ~0.2% of its traffic held a reputation score
of 100 all day, and neither existing mechanism could reach it:

• Additive scoring is outvoted by volume. At one violation per 1000 requests the endpoint
  earns +998 and loses −25, so the score returns to its ceiling however long the behaviour
  continues. Raising the per-event penalty to FATAL (−50) does not change the sign.
• The critical-rate detector is tuned for "unambiguously broken" (30%), and more
  fundamentally CriticalRateEWMAAlpha's ~20-request memory cannot represent a sub-1%
  rate at all — the EWMA can only be 0 or ~0.05 there. No threshold change to that
  detector could have worked.

So 22e9dff1 adds a second detector rather than retuning the first, with a much longer
window (alpha 0.001, ≈1000 requests) and a threshold three orders of magnitude lower.

Two classifier bugs fixed alongside it: a "(method=X)" suffix made every exact-match case
in classifyHeuristicErrorAsSignal unreachable, so an empty payload degraded to
unknown_payload_error (MINOR) — only the HasPrefix("error_indicator_") case survived,
which is why it hid. And getProgramAccounts returning an empty array is a valid success,
not a fault.

5. Archival promotion and health-check contamination (today)

54659fb6 — geth PBSS pruned state

Geth's path-based state scheme reports metadata is not found, <block>. Every archival
pattern in PATH used hash-based-scheme wording (missing trie node, state has been pruned), so a PBSS node's honest "I do not retain that state" matched nothing, at four
separate sites.

The deeper defect: IsArchival returned true for any successful eth_getBalance /
eth_call / eth_getCode / eth_getStorageAt / eth_getTransactionCount without
reading the block parameter. Those are also the ordinary way to read current state, and a
pruned node answers them perfectly — so the archival pool was polluted by construction and
marked archival for 8h. targetsHistoricalBlock now gates the promotion.

Known residual: the DataExtractor interface carries no perceived chain tip, so a numeric
block a few blocks back still reads as archival. Closing that needs an interface change
across all four extractors.

d8f4c3c1 — health-check probes were feeding both rate detectors

Both volume-independent rate detectors are wrapped in if !signal.IsHealthCheck, so a
probe cannot bench an endpoint on its own — a strict or flaky check must not cool an
endpoint that serves user reads perfectly. That guard was intact. The stamp was not.

IsHealthCheck was set at only three call sites, all in the health-check executor. One
probe also reaches reputation through the protocol layer twice more — the relay itself via
requestContext, and Apply{HTTP,WebSocket}Observations on that same relay's observations
— and neither stamped it. The field doc on requestContext.isHealthCheck stated outright
that it "does not affect reputation signals or observations", which is why the omission
read as deliberate.

The result is a self-sustaining loop rather than a one-off penalty: a benched endpoint
receives no user traffic, so probes become its only signal, so its rate EWMAs are entirely
probe-derived, so it re-benches itself on the next probe failure.

This predates the new detector. The critical-rate detector has been contaminated since it
shipped; 22e9dff1 only made it visible by tripping at a much lower threshold.

Apply{HTTP,WebSocket}Observations now take isHealthCheck as a required parameter rather
than defaulting it — the receiving protocol layer cannot distinguish synthetic observations
from real ones, so each caller states it at compile time.

Scope is narrow on purpose: only the rate detectors exclude probes. A probe still moves the
additive score — that is how a benched endpoint recovers when it receives no user traffic —
and still increments the counters, so no rate's denominator changes shape.
TestHealthCheckSignals_StillMoveTheAdditiveScore guards that.

064bc628 — each rate cooldown escalates against its own history

Score.InvalidRateCooldownCount is documented as kept separate from RateCooldownCount so
the two detectors escalate independently. The counters were separate; the timestamp they
escalated against was not — both compared against the shared Score.CooldownUntil, which
the strike system also writes.

Note the sign: time.Since() on a cooldown still in force is negative, hence always below
DefaultMaxCooldown. Any bench in force, from any mechanism, made the next trip of either
detector read as consecutive.

Each detector now records the end of the cooldown it set and escalates against that.
CooldownUntil is unchanged and remains the only field selection reads.

b395d183 — the "historical state" pruned-state wordings

Found by probing rather than by reading. Sending a block 27M deep to endpoints PATH had
marked archival returned two wordings that missed every pattern in
archivalErrorIndicators by a single word:

gnosis: "historical state is not available"   -- "state not available" misses on "state IS not"
poly:   "historical state <hash>"             -- "historical data" misses on "historical STATE"

Both fell through to the "some other error" branch, which returns an error rather than
false, so an endpoint that had just failed an archival query was never demoted out of the
archival pool. The bare "historical state" prefix covers both, and is already present in
qos/heuristic/indicators.go — the two catalogues had drifted, so this realigns them.

31617122 — an unverified archival mark no longer outlives a verified one

The two sources of archival status had drifted 16× apart, in the damaging direction:

health-check mark   30m
user-traffic mark    8h

The health-check path pins an exact expected historical value in the rules file, so a node
that ignores the block parameter and answers from current state fails it. The user-traffic
path cannot pin a value — the query is whatever a client sent — so it grants archival status
on any successful archival-method call, and 54659fb6 notwithstanding it still trusts a
successful response, which is what a fabricating node always produces.

Measured in production: four endpoints on one operator were marked archival while returning
current state for every block asked, including one 256× past the chain tip. The archival
health-check rule for the service they served had been deleted for failing every endpoint —
which was the rule working correctly, that service has no archival nodes — leaving only the
unverified 8h path to promote them.

Both paths now share gateway.ArchivalStatusTTL. The old comment on the 8h constant claimed
it "matches health check archival TTL"; it did not, and the false comment is probably why
the drift went unnoticed. The test reads the stored expiry back through
UpdateFromExtractedData rather than comparing constants, so re-hardcoding a duration at
the call site fails it.

Bootstrapping is unaffected: promotion still happens via requests naming a numeric block
within the archival-required threshold, which route freely rather than being filtered to
already-archival endpoints.

---

Production validation

Deployed to canary at 07:18 UTC and mainnet at 08:57 UTC on 2026-08-20. Because the two
environments flipped at different times, the same metric collapsing twice — each time
following the build — rules out pod age and traffic composition.

	before	after
mainnet rate_cooldown_total	0.178/s	0.0006/s (297×)
canary rate_cooldown_total	0.005/s	0.0012/s
pool_collapse_guard{solana} on canary	143,090 / 14h	~35 / 85m
invalid-rate trips	10 services	solana only, both envs

Guardrails flat throughout, judged against 6h ranges rather than point readings: fleet
success 95.7% mean (min 92.5, max 97.3), solana pool size unchanged, no cooldown spike.

Redis confirms the escalation fix directly. Mainnet DB2 went from 0 of 5,901 keys
carrying rate_cooldown_until to 292 of 297 sampled within five minutes of the flip, and
two solana endpoints that tripped post-deploy each recorded invalid_rate_cooldown_count = 1 with their own timestamp — a first offence, unescalated. For contrast, the pre-flip
distribution had 84 keys above zero with 35 at ≥ 6, i.e. benched the full hour every
time, topping out at 98.

Archival counts from /ready/<svc>?detailed=true moved as intended. The old build pinned
five services at exactly 100% — every endpoint of every operator archival, which is not a
plausible ground truth. The fixed build discriminates: on gnosis it keeps one operator at
12/12 while dropping two other operators to 0/17 and 4/17, where the old build had all
three at 100%.

Testing

go build, go vet and golangci-lint clean. Unit tests pass; reputation/storage needs
Docker for testcontainers-Redis and fails without it.

Every call site in the two reputation fixes was revert-checked individually — seven reverts,
seven confirmed test failures. Two traps found while writing those tests, both recorded in
comments:

• The first escalation test passed against the revert. A fresh key plus a foreign bench
  in force does not discriminate, because incrementing a zero counter yields 1 — exactly
  what a correct reset yields. The discriminating case needs stale non-zero history and a
  foreign bench.
• runAtRate(4000, 100) trips the detector five times, not once: a trip resets the EWMA and
  the loop continues. Any first-offence assertion has to drive one signal at a time and stop
  at the first trip. The pre-existing test never noticed because it only asserted
  IsInCooldown().

Tests assert on the signal reputation receives, from the production caller, rather than on
the flag the caller set — the flag was already true and proved nothing.

CI note: the xrplevm HTTP E2E check has been failing on this repo independently of this
branch. Please confirm it also fails on main before treating it as a blocker here.

Known open items

1. moonbeam genuinely has no archival nodes. Answered by direct probing:
   moonbeam_archival was deleted from the rules file on 2026-08-07 for failing 100% of
   runs — which was the rule working correctly — and the single endpoint still marked
   archival there is one that ignores the block parameter. Impact is small: moonbeam draws
   1.7–3.5 archival_required rejections/s against poly's 761/s.
2. Endpoints that ignore the block parameter cannot be detected by any success-based
   check, including the targetsHistoricalBlock gate added in 54659fb6, which verifies
   the request named a historical block and then trusts a successful response. Measured on
   one operator across two services: identical balances at block 1, block 5,000,000 and
   block 4,294,967,295. A design for detecting this is written up in
   DESIGN_NEGATIVE_HEALTH_CHECK.md and deliberately parked — the two commits above
   cover most of the exposure without a new check type. The cheapest future detector is
   eth_getBlockByNumber(H) asserting result.number == H, which is self-verifying and
   needs no external reference.
3. Only 22 of 69 services have an archival health-check rule. On the rest, archival
   status comes exclusively from the unverified user-traffic path.
4. poly_archival and xrplevm_archival assert expected_response_contains: "0x0", a
   substring matching a large share of hex values. Weak, and in the external rules file
   rather than this repo.
5. InvalidRateThreshold = 0.005 was sized from one hour of data with a contaminated
   denominator. Now that the denominator is honest it fires ~31/hour on solana, correctly
   confined but not near-silent. Worth re-deriving from the real per-key rate distribution.
6. User relays record a reputation signal twice — once in context.go and once via the
   observation path. Pre-existing double-count, not addressed here.

---

6. Circuit-breaker failure-rate gate

db817520 aef31dd6 1a9ca5cc a7063b59 d82b1e1d

Hysteresis. The gate used one threshold to break a domain and nothing but TTL expiry
to restore it, so a host whose true failure rate sat just above the line was removed every
time it was let back in, with escalation holding it out longer each cycle. Measured on six
relay-miner hosts behind one operator: the four marginal ones (within 1.7 points of the
80% line) spent 69–92% of a six-hour window removed from the pool, identically in both
environments, while answering 40 consecutive probes with zero errors at the same latency as
the host carrying the service. A domain that broke recently must now be clearly worse to
break again (threshold + 0.15).

The denominator was blind to hedge-race successes. Failures reach the gate from every
path, but a success only counts where the returning path calls RecordSuccess, and the
hedge-race branch never did. With a hedge delay configured every first attempt returns
through it — including the overwhelming majority where the hedge never fires — so the gate
saw 5% of a high-volume operator's successes and 26% fleet-wide, and read a low-volume host
at 30–66% failure where the relay counters read ~21%. No threshold, hysteresis included, can
hold against a rate inflated past it by construction. Tested through the real retry loop.

path_circuit_breaker_outcome_total{service_id, domain, outcome} exposes both sides of
the fraction the gate actually computes, per hostname — the gate keys on hostname while
path_relays_total keys on eTLD+1, and the blended figure cost four wrong hypotheses in one
investigation. Two labels by design, registered with the cardinality guard.

7. Traffic shape and classifier fixes

d7e4d81a 7ace2a41 45e00a24

GET /admin/request-sample. Every quality signal rewards whoever answers fastest, and
an endpoint fronted by a cache answers a repeated request in sub-millisecond time without
touching a node. Whether a fast operator is fast or merely cached has to be read from the
traffic, and nothing recorded its shape. One request in N is fingerprinted on method +
compacted params (ids excluded) and counted per service in fixed windows; the endpoint
reports uniqueness, top-1 share, and per-method uniqueness — block-height calls are
legitimately repetitive, account lookups are not. Bounded table, per-pod, two gauges keyed
on service_id only.

Solana's account-index exclusion is a capability limit, not a fault: -32010 "<key> excluded from account secondary indexes" is node configuration, and another operator serves
the identical call from its index. It matched nothing in the catalogue, so a dapp polling
three getProgramAccounts queries continuously charged one operator a breaker failure and a
reputation penalty on every poll. Added as an allowlisted phrase, deliberately not an
archival pattern.

path_observation_pipeline_total mislabelled every no-fault error as major_error —
it re-derived reputation_signal from the observation's error type instead of the signal
actually recorded. path_relays_total, labelled from the real signal, disagreed about the
same relays; two observables disagreeing about one state, and the pipeline label was the
wrong one.

8. Batch responses and client cancellations (2026-08-24/25)

9137cdbb da13f402 796ffc7d 51ffb44a 43abf62e 040ae428

Started from an operator asking why they were "broken" for batch_transport on one
service. The label misleads: PATH never sends a multi-request relay — batches are split and
each item is its own relay. batch_transport means one item of a client batch failed
before any HTTP response existed.

Every one of those breaks was context canceled — the client hung up mid-batch, and the
batch item loop stamped the abort of our request on whichever supplier was holding the
item. It hit every operator serving the service at once. The single-request loop already
returned on a done context before reaching MarkBroken; the batch loop now does the same
(9137cdbb). Measured: breaks 116/h → 0 on canary with the control unchanged.

Worth recording honestly: the user-facing effect over 40 minutes was neutral. The
cancels are not uniform across operators — they track p99 tail latency (0.23s on the best
operator, 1.0–2.3s on the worst), because context canceled is the client's timeout
firing on the supplier's tail — so the old behaviour was accidentally benching the one slow
operator for ~20 minutes at a time. Reputation does not bench it because these errors are
no-fault. The signal belongs in reputation's latency path, not the breaker; that is the
follow-up, and this A/B is its evidence.

A failed batch item threw away the whole batch. An item that failed every attempt has
no body; the assembler dropped it, the length check saw N−1 for N, and the client got a
single id:null -32603 "batch response length mismatch" in place of the N−1 answers already
relayed and paid for. Every retained log line of the failure was that shape. Per JSON-RPC
2.0 each request object gets a response object, so the item nothing answered now gets an
error object carrying its own typed id, next to the successes (da13f402, EVM and Cosmos
via the shared validator; 796ffc7d for NoOp, which had the quiet form — the item simply
vanished from the array with HTTP 200).

Three more shapes surfaced on canary and are fixed in the same helper:

• A one-element batch that retries runs down the single-request path, which records
  every response it saw; the batch assembler collected all of them, saw 2 for 1, and
  replaced a request that had succeeded with a 500. Keep the latest per id (51ffb44a).
• An id-less item is not a notification once it has been through PATH — it is relayed
  alone with "id":null written out and the node answers it. da13f402 had excluded such
  items from the expected count and rejected every batch carrying one with "expected N, got
  N+1"; caught on canary within the hour, reproduced with a two-item probe (500 on canary,
  200 on the control), fixed in 43abf62e. A test written from the spec instead of from
  what suppliers actually return; the replacement test uses the production shape.
• A response that is not a well-formed Response — an error given as a bare string, an
  id type the parser rejects — is still an answer to some item. Attribution now reads only
  the id; an unreadable id is a wildcard like a null id (040ae428). The mismatch error now
  lists the response ids, bounded, so any survivor names its shape in the log.

Validation

Canary ran each cut against mainnet as an untouched control, then mainnet followed after a
12-hour soak. Marshal-failure log lines in the retained tail: 242 on the control, 0 on
canary over 12h; 0 on both since mainnet rolled. Fleet relay success 0.926 vs 0.927,
5xx and rps equal, 0 restarts, heap on the known ~30 MiB/h curve. Probed end-to-end,
tallied by the serving environment: an id-less two-item batch 8/8 → 200 with both answers;
a one-element batch forced to retry (an archival call most suppliers refuse) → 3 of 5 500s
on the control, 200 with the supplier's own error on the fix.

Two shapes are not forceable from outside — an item with no body, and an unreadable id —
and rest on the log evidence plus the new ids diagnostic.

Every fix has a test that fails with the fix reverted; each was checked.

Also worth knowing

• path_requests_total{status_code="error"} is derived from the endpoint observation
  (backend status 0 with an error set) — per-relay transport failure, not what the client
  received. No metric records the client-facing batch outcome; the Error-level log line is
  the only evidence, and a counter for synthesized batch errors (service_id only) would be
  a reasonable addition.
• A batch where every item fails still returns the empty-batch response (nothing, 200) —
  the emptiness check runs before the fill. Rare; separate item.
• Solana's batch context already answered per id; it turns numeric-looking string ids into
  ints and says "malformed response" for a missing body. Cosmetic, unchanged.