Skip to content

feat(check-links-online): list timeouts and accepted 429 and 5xx links in the summary - #844

Merged
kyle-sexton merged 2 commits into
mainfrom
fix/lychee-online-transient-report
Oct 21, 2026
Merged

kyle-sexton merged 2 commits into
mainfrom
fix/lychee-online-transient-report

Conversation

@kyle-sexton

@kyle-sexton kyle-sexton commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Refs: melodic-software/claude-code-plugins#8730
Refs: #739

Summary

standards#834 makes the managed lychee.toml accept 429 and 5xx and turn on accept_timeouts. The owner wants those transient links kept out of the failure count but still listed in a read-only report, so a host that stays down is eventually noticed. The job summary the reusable posts today lists timeouts, but not accepted 429 or 5xx responses.

What lychee 0.24.2 outputs for each transient case:

Case compact or markdown report markdown with -v --format json --format json -v stderr at -vv
Timeout (accept_timeouts) listed ("Timeouts per input") listed timeout_map timeout_map listed
Accepted 429 or 5xx counted in Successful only counted only not listed success_map, with code, file and span listed among all successes

The sources at lychee-v0.24.2:

  • Status::new maps an accepted code to Status::Ok, per status.rs#L90-L97.
  • ResponseStats files a success into success_map only when detailed_stats is set, per response.rs#L109-L126.
  • detailed_stats is on at -v (Info) and above, per check.rs#L41-L45.
  • The markdown formatter writes Errors, Timeouts, Redirects and Suggestions, and never success_map, per markdown.rs#L106-L130.
  • A local probe of 0.24.2 against a status server agreed with each cell above.

Fix

.github/workflows/maintenance-check-links-online.yml:

  • The lychee-action step now sets format: json, output: ${{ runner.temp }}/lychee/report.json and --verbose. It also sets jobSummary: false, fail: false and failIfEmpty: false. Per entrypoint.sh at v2.9.0:
    • the action writes the job summary only when its format input is markdown;
    • its failIfEmpty check greps the markdown table, so it cannot match JSON;
    • --format and --output go through inputs, not args.
  • A new Write the report step (bash and jq) writes the job summary: a counts table, then "Broken links (fail the run)", "Timeouts", and "Accepted 429 and 5xx responses (do not fail the run)". The last section is success_map filtered to code 429 or 500 to 599.
    • The rows are sorted by input and URL. A null column is omitted, input names that contain backticks are fenced, and newlines in details are folded.
    • Each section is capped at 200 rows, followed by a "+N more" line. This keeps the summary under the 1 MiB per-step job summary limit.
    • The step fails the job when lychee's exit code is non-zero.
      • A run failed only by timeouts says so: the config does not set accept_timeouts, so check.rs#L113-L121 exits 2.
      • lychee-action creates the report file before lychee runs, so an empty report fails with a pointer to the lychee step log. A report that is not JSON fails with its own error, and a report with zero links fails too.
  • There are no workflow_call input, secret or permission changes, so the runner-policy contract in standards needs no edit. The claude-code-plugins caller picks this up when its pin moves.

The test and README are updated to match.

Verification

  • node --test .github/scripts/*.test.cjs: 556 pass, 0 fail. The 15 cases in maintenance-check-links-online.test.cjs run the step's own run script against JSON recorded from lychee 0.24.2. In that recording only the input path is renamed and the durations are dropped. The cases cover:
    • broken links;
    • transient-only runs;
    • a run failed only by timeouts;
    • an empty report, a non-JSON report and a report with zero links;
    • an entry with no code and a null column;
    • sorting and the row cap.
  • Mutation checks: I removed, one at a time, the empty-report guard, the column guard, the input sort, the row cap, the newline fold and the timeout message. Each removal failed at least one test.
  • actionlint, zizmor --offline (no findings), biome ci .github/scripts, markdownlint-cli2 README.md and typos pass.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01Tje1j8KEYaYSxYe8RgJ3Lw

…s in the summary

The standards lychee.toml now accepts 429 and 5xx and sets
accept_timeouts, so those links no longer fail the online run. lychee's
markdown report lists timeouts but never lists accepted responses; only
its JSON report at --verbose carries them, in success_map. The reusable
now asks lychee-action for that JSON and a report step writes the job
summary: broken links (which fail the job), timeouts, and accepted 429
and 5xx responses (which do not). The step also keeps the no-links and
unreadable-report failures.

Refs: melodic-software/claude-code-plugins#8730
Refs: #739

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tje1j8KEYaYSxYe8RgJ3Lw
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T18:12:12.344358Z 5468b60 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@claude

claude Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 29s —— View job


Security review complete

  • Gather context (PR diff, changed files)
  • Review workflow YAML changes (maintenance-check-links-online.yml)
  • Review the new "Write the report" bash/jq step for injection and trust-boundary issues
  • Review test file changes
  • Report findings

No security issues found.

Scope reviewed: .github/workflows/maintenance-check-links-online.yml, .github/scripts/maintenance-check-links-online.test.cjs, and README.md.

Notes from the review (none rise to a finding):

  • The new "Write the report" step (lines 151–185) passes $REPORT and $LYCHEE_EXIT through env: rather than interpolating them into the run: script text, so there's no expression-injection surface there.
  • The jq filter only reshapes report.json (produced by lychee from the caller's own Markdown) into the job summary; it doesn't introduce a new trust boundary versus the markdown summary lychee-action wrote before this change — URL/status text already flowed into the summary previously via lychee-action's own formatter.
  • permissions: {} at workflow level and contents: read at job level are unchanged by this PR, and the PR body correctly notes there are no workflow_call input/secret/permission changes.
  • GitHub Actions hardening (pinning, trigger safety, permissions:) is zizmor's lane and is out of scope here; zizmor is reported as passing in the PR description.

claude-lane-summary: lane=claude-security-review sha=5468b60ebc87827075041c29439e2d489b9a4189 run=39650393250

@claude

claude Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 5m 59s —— View job


Review in progress

  • Gather context (diff, changed files, rules)
  • Check path-scoped rules (pr-body-contract.md) and PR body contract compliance — body uses Refs: (no closing issue), has all four required sections
  • Review workflow YAML changes
  • Review test file changes
  • Review README changes
  • Adversarial validation of candidate findings (hunter agent running)
  • Post findings / summary

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

claude-security-review has reviewed this pull request through c7842ef; a later push is reviewed from there.

@claude

claude Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 7m 19s —— View job


Claude is reviewing this PR

  • Gather context (diff, REVIEW.md, CLAUDE.md, path-scoped rules)
  • Review workflow YAML changes
  • Review test changes
  • Review README changes
  • Verify two candidate findings adversarially
  • Post findings

…, cap the summary

- Timeouts fail the run when the caller's config does not set
  accept_timeouts, so the heading no longer says they do not, and a run
  failed only by timeouts says so.
- lychee-action creates the report file before lychee runs, so an empty
  report now fails with a pointer to the lychee step log instead of
  "No links were found"; a non-JSON report gets its own annotation.
- Rows omit a null column, fence input names that hold backticks, fold
  newlines in details, and sort by input and URL.
- Each section is capped at 200 rows with a "+N more" line, under the
  1 MiB per-step job summary limit.

Refs: melodic-software/claude-code-plugins#8730
Refs: #739

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude

claude Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 5m 18s —— View job


Claude finished reviewing this PR

  • Gather context (diff, REVIEW.md, AGENTS.md, path-scoped rules)
  • Check path-scoped rules (pr-body-contract.md) and PR body contract compliance
  • Review workflow YAML changes (maintenance-check-links-online.yml)
  • Review test file changes (maintenance-check-links-online.test.cjs)
  • Review README changes
  • Trace the jq report-generation logic against the recorded lychee fixture and every new test case
  • Post findings

No findings.

Notes (not findings)

  • PR body contract (pr-body-contract.md): compliant. The body opens with Refs: lines (not "does not close," which would still close), and includes ## Summary, ## Fix, ## Verification, ## Related.
  • Security scope: this repo runs a dedicated pr-review-security lane (already reviewed this head and found nothing), so security-adjacent observations are left to that lane per REVIEW.md.
  • I traced the new jq pipeline in maintenance-check-links-online.yml#L172-L212 (sorting, code-span escaping, column/details fallbacks, per-section capping, the errors == 0 and timeouts > 0 branch used to name the failure cause) by hand against the recorded lychee 0.24.2 fixture added in the test file, and every branch I checked matches the corresponding assertion in maintenance-check-links-online.test.cjs. The !cancelled() condition on the new "Write the report" step is deliberate (not a mistake): omitting a status-check function would implicitly AND the step's if with success(), which would skip the step whenever the lychee step itself fails — but the step's own job is to turn an empty/unparseable report from a crashed lychee step into a clear error, which the "empty report" test exercises.
  • I could not execute node --test, jq-based reproduction, or network fetches (e.g. to confirm lychee-action's exit_code output or directory auto-creation for a custom --output path) in this sandbox, so I'm relying on static tracing plus the PR's own evidence: the linked dispatch probe (claude-code-plugins run 39648476685) exercised this exact code end-to-end against a real config and reported a mix of an accepted 429, timeouts, and failing 421s classified correctly, which is strong corroboration — though I'm treating it as author-supplied evidence, not something I independently reran.
  • Test coverage is notably thorough: it covers sorting, capping with a remainder line, odd input names requiring a longer backtick fence, missing span.column, and the empty/non-JSON/zero-link failure paths, and the author's described mutation check (429 → 4290) is a good sanity signal for the filter.

claude-lane-summary: lane=claude-review sha=c7842ef523a87bd0d60d09c1d40a71fd6298be39 run=39662407043

@claude

claude Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 3s —— View job


Security review complete (incremental: 5468b60 → c7842ef)

  • Read .claude-lane/incremental.diff (scope: changes since 5468b60)
  • Review workflow YAML changes (maintenance-check-links-online.yml: jq filter rewrite, new empty/non-JSON report checks, timeout-failure message)
  • Review test changes (maintenance-check-links-online.test.cjs)
  • Review README changes
  • Finalize findings

No security issues found in the incremental diff.

Notes (none rise to a finding):

  • maintenance-check-links-online.yml#L151-L197: the jq program text is static; $REPORT, $LYCHEE_EXIT, and the new $MAX_ROWS all flow through env:/--argjson/file args rather than being interpolated into the run: script, so the new logic preserves the no-expression-injection pattern already noted in the prior review of this step at 5468b60.
  • The new code jq function picks a backtick fence longer than any run found in the input filename before wrapping it, and gsub("[\r\n]+"; " ") strips newlines from status.details. Both guard the job-summary Markdown against being broken by filenames/messages containing backticks or embedded newlines. This is a correctness/robustness fix for the summary's own rendering, not a trust-boundary change: the filenames and status text still come from the same repo's own content/lychee's own messages, as before.
  • The new empty-report and non-JSON-report checks ([ ! -s "$REPORT" ], jq -e 'type == "object"') only read a file at a fixed, workflow-controlled path and emit a log line; no new input is trusted that wasn't already.
  • MAX_ROWS: 200 is a literal in the workflow's env:, not user input, so the --argjson max "$MAX_ROWS" pass-through has no injection surface.
  • No workflow_call input/secret/permission changes in this increment (consistent with the PR body), and no changes to AGENTS.md/CLAUDE.md/rules files, so the instruction-surface-deletion lens does not apply here.

claude-lane-summary: lane=claude-security-review sha=c7842ef523a87bd0d60d09c1d40a71fd6298be39 run=39662407043
· branch fix/lychee-online-transient-report

@github-actions

Copy link
Copy Markdown

claude-review has reviewed this pull request through c7842ef; a later push is reviewed from there.

@kyle-sexton
kyle-sexton added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit 4aff993 Oct 21, 2026
81 of 82 checks passed
@kyle-sexton
kyle-sexton deleted the fix/lychee-online-transient-report branch October 21, 2026 19:46
kyle-sexton added a commit to melodic-software/standards that referenced this pull request Oct 21, 2026
)

Refs: melodic-software/claude-code-plugins#8730
Refs: melodic-software/ci-workflows#739

## Summary

The online link check failed on transient responses: a 429, a 5xx such
as 502, 503 or 522, or a timeout says nothing about whether a link is
dead. The owner decided on 2026-10-21 that these responses must not fail
the run, and must still appear in a read-only report. Dead links (404,
410 and every other 4xx) keep failing. lychee stays on v0.24.2 and
lychee-action stays on v2.9.0, the latest stable releases.

## Fix

- `lychee.toml`:
- Sets `accept = ["100..=103", "200..=299", "429", "500..=599"]` and
`accept_timeouts = true`.
- Replaces the comment that explained why 429 was not accepted. The new
comment gives the reason for the change and its cost: an accepted 429 or
5xx is not retried, and its fragment is not checked. It names
[lycheeverse/lychee#2324](lycheeverse/lychee#2324)
as the trigger to revisit 5xx.
- `components/lychee/lychee.test.sh`: the status-code block now asserts
the owner's rule.
  - 200, 429, 500, 503 and 599 pass the run, each counted as `1 OK`.
  - An accepted 429 is requested once.
  - 404 and 410 fail the run with `Rejected status code`.
- A timeout passes the run and is still reported as `1 Timeouts`, while
the same request with `--accept-timeouts=false` fails.
- `components/lychee-online/README.md`: the prose now matches the new
behavior.

Basis for each setting, all at lychee v0.24.2:

- **`accept` replaces the default, so `100..=103` and `200..=299` are
restated.** The `--accept` default is in the
[README](https://github.com/lycheeverse/lychee/blob/lychee-v0.24.2/README.md).
`Status::new` in
[status.rs#L90-L97](https://github.com/lycheeverse/lychee/blob/lychee-v0.24.2/lychee-lib/src/types/status.rs#L90-L97)
returns `Ok` only for a code in the accepted set. A local probe with
`--accept 429` alone rejected a 200.
- **A rejected 5xx is never retried, so accepting 5xx is the only way to
tolerate it.** The basis is
[retry.rs#L97-L100](https://github.com/lycheeverse/lychee/blob/lychee-v0.24.2/lychee-lib/src/retry.rs#L97-L100).
In the probe, 502 and 503 each got one request with `--max-retries 2`.
- **An accepted 429 is a success whose fragment is not checked.** The
basis is the comment and the `response.status.is_success()` guard in
[website.rs#L137-L142](https://github.com/lycheeverse/lychee/blob/lychee-v0.24.2/lychee-lib/src/checker/website.rs#L137-L142).
In the probe, an accepted 429 got one request.
- **`accept_timeouts` exists from v0.24.0.** Basis: the [lychee-v0.24.0
release notes
(#2063)](https://github.com/lycheeverse/lychee/releases/tag/lychee-v0.24.0).
[check.rs#L113-L117](https://github.com/lycheeverse/lychee/blob/lychee-v0.24.2/lychee-bin/src/commands/check.rs#L113-L117)
uses `is_success_ignoring_timeouts` when it is set.

- **Repin, in the same merge.** The claude-code-plugins caller
(`components/lychee-online/targets/claude-code-plugins.yml`) moves from
v0.55.0 (`3a2c748`) to
[v0.59.0](https://github.com/melodic-software/ci-workflows/releases/tag/v0.59.0)
(`4aff99320ef484a7beac35ee3a7b7b854ec1e33f`). That release carries
[ci-workflows#844](melodic-software/ci-workflows#844),
the report step. The README "Pin" procedure moves the pin and the
`policy.json` contract key together in one PR, so the config and the
reusable that reports it land in one merge.
- `components/runner-policy/policy.json`: the contract at `4aff993` is a
verbatim copy of the v0.55.0 entry.
- `components/runner-policy/README.md`: a v0.59.0 rollout line, signed
off by hand on 2026-10-21. The owner approved the change on that date.
- `components/runner-policy/runner-policy.test.mjs`: a test that the
`4aff993` entry equals the v0.55.0 entry and is the only key at that
SHA.
- No other standards component pins
`maintenance-check-links-online.yml`. The other `3a2c748` pins are
`pr-review`, `pr-review-security`, `maintenance-sync-standards` and
`check-managed-files`, which the repin cascade owns.

### Contract review basis

- The workflow blob changed: `b2caebc3` at `3a2c748` and `ed87540b` at
`4aff993`. The diff is 83 lines added and 6 removed.
- I parsed both versions with ci-workflows' `workflow-yaml.cjs`. The
contract surface hashes identically at both SHAs (sha256 `9d258d23...`).
That surface covers the `on` block (the `workflow_call` inputs, secrets
and outputs), the top-level `permissions: {}`, and every job's
`permissions` (`contents: read`), `runs-on` (`${{ inputs.runner }}`) and
`uses`.
- The changes are all in steps:
- The lychee-action step, which holds `token: ${{ github.token }}` as
before, gains `id: lychee` and moves to `format: json`, an `output`
path, `--verbose`, and `fail`, `failIfEmpty` and `jobSummary` set to
`false`.
- A new `Write the report` step uses only `env` and `jq` and holds no
credential.
- Because the token-bearing step changed, the gate declines
auto-approval: "credentialReferences changed since the previously
reviewed ...@3a2c748".

**Consumers.** Every target in `distribution/sync-manifest.yml` takes
the `lychee` component as managed; none holds it locally owned. The file
is read by these lanes:

- **Offline lanes.** These are the ci-workflows `lychee-offline` action
and this repo's `lychee-fixtures` lane. Both run `--offline`, so they
make no HTTP requests and neither key changes them. The offline run over
this repo's Markdown gives the same result under the old and the new
config: `959 Total, 402 OK, 0 Errors, 557 Excluded`. The bad fixture
fails with 2 errors under both.
- **The online check.** This is the ci-workflows reusable
`maintenance-check-links-online.yml`, which the claude-code-plugins
caller runs. Under this config, lychee's markdown job summary lists
timeouts but not accepted 429 or 5xx responses.
[ci-workflows#844](melodic-software/ci-workflows#844)
adds that list.

The api.github.com 403s are out of scope here. They are handled by
exact-URL ignores in
[claude-code-plugins#8875](melodic-software/claude-code-plugins#8875)
([standards#831](#831)).

## Verification

- `npm run test:runner-policy`: 325 pass, 0 fail, including `the online
link-check contract at the v0.59.0 tag copies its v0.55.0 entry forward
verbatim`. `npm run lint:runner-policy`: `Runner policy passed.`
- End to end: I copied the repinned caller and claude-code-plugins'
`.github/runner-policy.json` into a temp tree and ran `runner-policy.mjs
--repository-visibility public`.
  - With this branch's policy: `Runner policy passed.`
- With main's policy: one `runner-target-contract` failure for the
caller (auto-approval declined, `credentialReferences` changed), so the
contract is needed.
- `bash harness/shell/run-tests.sh components/lychee/lychee.test.sh`
still passes after merging main.
- `bash harness/shell/run-tests.sh components/lychee/lychee.test.sh`:
pass. I also mutated `lychee.toml` (added 404 to `accept`, set
`accept_timeouts = false`), and the test failed on the 404 and timeout
cases as expected. Then I restored the file.
- `bash harness/shell/run-tests.sh
components/lychee/private-repo-inventory.test.sh`: pass.
- `bash components/lychee/private-repo-inventory.sh check --config
lychee.toml`: OK.
- `shellcheck`, `typos`, `markdownlint-cli2`, `editorconfig` and
`gitleaks` pass through the lefthook pre-commit.

## Related

-
[ci-workflows#844](melodic-software/ci-workflows#844):
the reusable lists timeouts and accepted 429 and 5xx links in the job
summary.
-
[lycheeverse/lychee#2324](lycheeverse/lychee#2324):
retrying 5xx. When it ships, revisit accepting 5xx.
- Merge order: done. ci-workflows#844 merged, v0.59.0 was released, and
this PR now carries the caller repin and its contract, so one merge
lands everything.
- Dispatch probe: [claude-code-plugins run
39648476685](https://github.com/melodic-software/claude-code-plugins/actions/runs/39648476685)
ran the #834 config through the #844 reusable on a throwaway branch. It
found 18676 links: 1 accepted 429 (listed), 0 5xx, 4 timeouts (listed),
and 2 failures, both a 421 Misdirected Request from
nap.nationalacademies.org. The 421s are an other 4xx, so the run failed,
as the rule requires.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Tje1j8KEYaYSxYe8RgJ3Lw

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant