webhook: runtime settle-violation detector + idempotent delivery outcome#203
Merged
Conversation
…outcome Two P3 limitations left documented by the post-11.07 review, closed together (they share the webhook delivery path); the P3 DuckDB DST-fold delay is left documented as an accepted narrow limit. P3(a) — the settle watermark's operator invariant (AGENTFLOW_WEBHOOK_SETTLE_SECONDS > writer stamp-to-visibility lag + clock skew) was enforced only implicitly: a violation silently dropped the late-visible row behind the keyset frontier. Add a cheap sampled runtime detector that probes the bounded band immediately behind the frontier (a newest_first window under a new symmetric max_processed_at bound on fetch_pipeline_events, once per interval — no per-pass cost, no journal-wide scan) for rows the scan never marked seen, and raises the new agentflow_webhook_settle_violations_total counter with a webhook_settle_invariant_violation warning. Reads only fetch_pipeline_events and the in-memory seen-set, so it runs on both DuckDB and ClickHouse serving stores; silent under the settle=0 opt-out. P3(b) — record_webhook_delivery_outcome did a non-idempotent read-modify-write of attempts, so the PostgreSQL adapter's transient-error retry could count one failure twice (attempts+2 -> premature dead-letter) when an UPDATE committed but its commit-ack was lost. Thread the delivery round's delivery_id through and record it on the queue row (last_outcome_id); a repeat outcome write no-ops. Embedded and PostgreSQL adapters both stamp/guard on the id; PostgreSQL adds migration 2 for the column. Store-level regression test is red on the pre-fix path, green on the fix; detector tests cover fire / settle=0 silence / no false positive on already-delivered rows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ixes serving-bridge.md: the settle-invariant violation is no longer silent (runtime detector + agentflow_webhook_settle_violations_total), and delivery outcomes are idempotent per round (no attempts+2). The DuckDB autumn DST-fold delivery-delay paragraph is left unchanged (still an accepted limit). CHANGELOG: unreleased P3 follow-ups entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DORA Metrics
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the two known webhook-dispatch limitations documented in serving-bridge.md:
agentflow_webhook_settle_violations_totalwhen a row becomes visible behind the settle watermark (i.e. the operator-setAGENTFLOW_WEBHOOK_SETTLE_SECONDSis smaller than real visibility lag + clock skew). Silent atsettle=0(explicit opt-out). Both stores.record_webhook_delivery_outcomeis idempotent perdelivery_id(last_outcome_idguard), so a retry after a dropped commit-ack can no longer inflateattemptsby 2 and trigger premature dead-lettering. Both adapters, schema migration 2.Verified locally: ruff check + format clean, 99 passed (2 new test files). Live PG/CH suites run in CI.
🤖 Generated with Claude Code