Skip to content

Escalate stale spec drift issues, and open a pull request for each new one - #11

Merged
liady merged 6 commits into
mainfrom
claude/ora-ax-guide-spec-adherence-vbf8pf
Oct 20, 2026
Merged

Escalate stale spec drift issues, and open a pull request for each new one#11
liady merged 6 commits into
mainfrom
claude/ora-ax-guide-spec-adherence-vbf8pf

Conversation

@liady

@liady liady commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

What changed

Spec drift issues now lead to action instead of waiting for someone to notice:

  • Escalation: an issue still open after 14 days gets the spec-drift-overdue label and one comment that mentions @liady. To change who is mentioned, edit --escalate-mention in spec-drift-watch.yml.
  • Failed runs: a watch run that fails opens a spec-drift-watch issue, which the next good run closes. Cancelled runs don't count.
  • Fix workflow: each new drift issue starts the new spec-drift-fix.yml. It opens a pull request that moves the pins and carries Claude's review of the upstream change: what changed, which claims on the page still hold, and suggested edits with sources. A maintainer makes those edits on the branch and merges.

How the fix workflow stays safe. Spec text is written by whoever controls the upstream sites, so nothing a model writes is ever committed, and the review is the only thing the agent produces. The workflow has five jobs:

  • prepare (read-only, no secrets): checks the issue was opened by the watch, then records what changed upstream: diffs, both IETF revisions, page text.
  • draft (only the Claude key; no GitHub token, no .git): Claude Code 2.1.290 runs with --restricted and file tools only, with no shell, web, subagents, hooks or MCP. I tested this locally with the pinned version and a hostile prompt. Shell, web fetch, writes outside the checkout, reads of /proc/self/environ, subagents, a project hook and a project MCP server were all refused.
  • verify (no token): spec-drift.mjs apply-pins moves the pins from values the watch already shape-checked. vet-staged then asks git whether only pin values changed, after which the checks run. A failing check turns the pull request into a draft; it is not hidden.
  • propose: uses the write token only to push and to open the pull request, after cleaning the review with pr-body.
  • notify: comments on the issue if any job failed.

A newer dated revision (ACP's spec folders) is never adopted automatically. The review goes on the issue instead.

It needs an ANTHROPIC_API_KEY (preferably from a workspace with a spend limit) or CLAUDE_CODE_OAUTH_TOKEN repository secret, and skips without one. Each run is capped at 40 turns and USD 5 of estimated spend. Pull requests opened with the workflow token don't start CI, so close and reopen one to run it.

scripts/spec-drift.mjs gains context, apply-pins, vet-staged, pr-body, health, escalation, --opened-out and a json-field source kind; poppy.ax uses the last one. spec-pins.json gains pin_file. The script stays byte-identical in all three repos. The dispatch step runs after the status publish and is allowed to fail.

Sources

No page content changed. The fix workflow follows the anthropics/claude-code-action and Claude Code CLI documentation (--restricted, --tools, --permission-mode dontAsk, --permission-prompts none, disableAllHooks, --max-turns, --max-budget-usd). I used the CLI directly, not the action, so the agent's job never holds a GitHub token.

Checklist

  • Every new or changed claim links a primary source. (No claims changed.)
  • If I changed a status, the evidence meets the rules in content/contribute.md. (No status changed.)
  • I only changed last_verified on pages where I rechecked every claim. (None changed.)
  • Plain words, no em dashes.

Reviews. @ora-security-reviewer ran four rounds. The first three blocked: the agent had a shell and a write token; then the patch-text gate could be bypassed; then MDX and pin-shape gates could be beaten. The fourth approved the design in which no model output is committed, and its hardening points are fixed in the last commit. A correctness review's findings are also fixed.

Verification

  • npm run typecheck, npm run lint and npm test (17 tests) pass.
  • Live: context and apply-pins ran on the real Poppy drift (open issue Poppy: the spec changed since ax.guide last checked it #10). They moved exactly the three pins in content/standards/poppy.md and spec-pins.json, vet-staged passed, the tests passed, and the watcher then read Poppy as clean.
  • Stand-in gh: escalation fires once, health issues open and close, and opened issues are recorded. Every workflow run: block passes bash -n.
  • Not yet run: the workflow itself on GitHub Actions, which also needs a Claude key secret.

Merge order: merge this first. The Ora and poppy.ax PRs compare their script copies with this repo's main.

Generated by Claude Code

claude added 6 commits October 20, 2026 07:15
@claude
Escalate stale spec drift issues and ask Claude to fix new ones
f67b36a 
An issue still open after 14 days gets the spec-drift-overdue label and a
comment for a maintainer. Each new drift issue starts spec-drift-fix.yml,
where Claude rechecks the page, moves the pins and opens a pull request for
review (it needs a Claude key secret and skips without one). A failed watch
run opens its own issue, closed by the next good run. The shared script
gains these commands and a json-field source kind for poppy.ax.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VeACDNmxsBifFG4eSqRsUi
@claude
Run the fix agent with no shell, web access or write token
0d2bd5a 
Rework spec-drift-fix.yml into three jobs after a security review: a
read-only job gathers the issue and what changed upstream (spec-drift.mjs
context), Claude drafts the fix with file tools only and no GitHub token,
and a job with no model refuses any patch outside content/standards
(spec-drift.mjs vet-patch) before opening the pull request. The dispatch now
runs after the status publish and may fail without costing it, cancelled
runs no longer count as failures, and escalation tolerates runs that start a
little early and posts its comment before the label.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VeACDNmxsBifFG4eSqRsUi
@claude
Let vet-patch refuse markup and code in MDX
71a0ecc 
A new --plain option refuses added lines with <, { or } or import/export lines in matching files. poppy.ax uses it for its .md and .mdx content, which a site build (Vercel previews included) would run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VeACDNmxsBifFG4eSqRsUi
@claude
Vet the fix agent's change with git, and run checks without the token
1cd25e2 
Review found that vet-patch read the patch as text, so a path with a space or a line starting with ++ slipped past it, and the job that ran the checks held the write token. vet-staged now applies the patch and asks git what changed (only added or modified regular files, pin lines only in pin files, only string pin values in manifest JSON, no markup or code in MDX). A new verify job with no token vets and runs the checks; propose runs only git with hooks off and gh. The agent's description is cleaned (pr-body), a failed run comments on its issue, branch names carry the run id, and the dispatch step runs unless the run was cancelled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VeACDNmxsBifFG4eSqRsUi
@claude
Commit no model output: move pins by code, keep Claude's review in th…
0c51627 
…e pull request

A third security review showed that gates judging agent-written files keep leaking (removed MDX lines, any constant with a pin-shaped line). So nothing a model writes is committed any more. Claude writes only a review; code moves the pins (spec-drift.mjs apply-pins, from values the watch already shape-checked), git confirms only manifest-named pin values changed (vet-staged, now pairing pins by name), and the pull request carries the review for a person to act on, as a draft when a check failed. The propose job keeps no stored credentials and uses the token only for the push and gh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VeACDNmxsBifFG4eSqRsUi
@claude
Harden the fix workflow after its fourth review
8cf1f4f 
From the fourth security review: the draft job deletes .git before the agent runs and later jobs check the pack's hash; poppy.ax's prepared catalog must match the profile the watch saw and use https links only; cleanPrBody escapes raw HTML, drops link reference lines and invisible characters, and catches more closing-keyword forms; apply-pins searches a manifest pin only in its own entry, reads every pin back afterwards, and leaves a newer dated revision for a person to adopt (the run then posts the review on the issue).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VeACDNmxsBifFG4eSqRsUi
@vercel

vercel Bot commented Oct 20, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
standards Ready Ready Preview Oct 20, 2026 8:14am UTC

Request Review

@liady liady mentioned this pull request Oct 20, 2026
Merged
4 of 5 tasks
@liady
liady merged commit da4d651 into main Oct 20, 2026
3 checks passed
@liady
liady deleted the claude/ora-ax-guide-spec-adherence-vbf8pf branch October 20, 2026 08:16

This branch was successfully deployed

1 active deployment
Preview — 8cf1f4f3 Deployed Oct 20, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@liady @claude