Skip to content

4xx alerts against a baseline, the failing request and response in every alert - #2

Merged
necco-c merged 4 commits into
mainfrom
alerts-bodies-4xx
Oct 8, 2026
Merged

necco-c merged 4 commits into
mainfrom
alerts-bodies-4xx

Conversation

@necco-c

@necco-c necco-c commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

What this changes

  • 4xx alerts against each API's own baseline (--client-errors baseline, the default). Each API learns its normal share of 4xx answers over its first five minutes (and at least 50 requests), then alerts when the last minute is at least 3x that share and 10 points higher, with at least 5 4xx answers. The baseline is frozen while the alert is firing. --client-errors all|off, --client-codes 401,403,429, --min-client-errors 5.
  • The failing request and response in every 5xx and 4xx alert (src/lib/bodies.js). Bodies are decompressed through yeet:compression (gzip, deflate, brotli) and cut to about 1000 characters each; binary bodies are shown as a size. --bodies redacted (default) blanks values under secret-looking keys in JSON, form and query strings (password, token, api_key, authorization, card, cvv, ssn and similar) plus bearer tokens, JWTs and Luhn-valid card numbers anywhere. raw and off are the alternatives. Headers other than the content type are never shown. Email addresses and other personal data are not redacted (decided).
  • Status line every minute in --watch, so anyone reading the log sees one within a minute.
  • Batched messages carry each API's sample; fixed the footer slot.
  • README covers all of the above, plus the /log route (yeet service unit add apiwatch/web -W http://127.0.0.1:9470 and yeet service mount apiwatch/web -L /log -t watch -p console) and the privacy note that alert bodies travel through yeet's servers to Slack.
  • package.json (ES modules) and test/: 16 tests for redaction and the 4xx baseline, npm test.

How it was tested

  • npm test: 16 of 16 pass.
  • Live dry-runs from this branch via yeet run github:yeet-src/apiwatch@alerts-bodies-4xx:
    • Debian 13, kernel 6.12 (nginx and two Python services): a payments outage produced one batched message with each failing request and response. A POST body's card_number and password and an access_token query parameter came out [redacted].
    • Debian 12, kernel 6.1 (Node API, kprobes): learned a 15% normal 404 share, fired once at 52% with the request (its api_key blanked), reported back to normal at 13%.
  • Cold Claude Code agents ran the API-debugging prompt end to end on both boxes in a test mode that skips sign-in (dry-run instead of Slack). Both installed the service with the /log route, which returned 403 as expected while signed out. A real outage under the agent-installed service produced the batched alert.

Not yet tested: the live Slack post

Every Slack-side behavior so far is a dry-run. To close it, on a Linux box with yeet installed:

  1. yeet login and approve the link, then connect Slack at https://yeet.cx/settings. Invite the yeet app to the channel if it is private.
  2. yeet run -y github:yeet-src/apiwatch@alerts-bodies-4xx -- --test-alert --slack "#your-channel" --name test-box: one message should arrive.
  3. Run --watch --slack "#your-channel" against something you can break (stop an upstream behind a proxy) and check that the batched message renders, request and response included, with nothing redacted that shouldn't be.
  4. Open question for the yeet platform: the sender shows as the yeet Slack app. Whether yeet.alert can set Slack's username / icon_url (to say "apiwatch") depends on the platform forwarding them and the Slack app having chat:write.customize.

@necco-c
necco-c merged commit 6faee28 into main Oct 8, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant