chore: create an invocable agent for workflow and topic labeling #5276

User Story

As an OpenShell maintainer managing issue triage and PR readiness, I need accepted work classified into accurate area/topic labels with little manual effort, so contributors receive relevant notifications and review requests without unaccepted-intake noise.

Scope and Delivery Parts

Part 1: Make acceptance explicit — PR #5260

PR #5260 provides the deterministic accepted-work definition and importable tasks/scripts/accepted_work.py library. The existing review-readiness tooling uses it to add state:accepted to qualifying new PRs, including drafts and PRs awaiting classification. The library stays read-only; review-readiness owns the label write.

This makes implicitly accepted PR work explicit and provides the acceptance signal for the later labeling agent. It does not choose area/topic labels, migrate legacy classification, continuously refresh acceptance, or classify issues. Existing issue-acceptance and merge requirements continue to apply. #5259 is superseded by this consolidated issue; #5260 references this issue without closing it.

Part 2: Extract the /label-issue skill and validate it through observed runs

Tracked separately in #5353; its extraction PR closes that focused issue and leaves this overall tracker open.

Create a narrowly scoped, chat-invocable /label-issue skill accepting one issue or PR number or URL. Factor label reconciliation out of triage instructions and have the triage skills invoke /label-issue directly rather than maintain competing classification instructions.

The skill owns area/topic classification, correction, and per-item legacy-label migration. Triage retains judgments about validity, missing evidence, and readiness for human acceptance. The skill may reflect supplied triage decisions in workflow labels but does not independently repeat triage, accept work, choose roadmap placement, implement changes, or review a PR. Intake assessment-state updates remain possible before acceptance; area/topic classification requires accepted work.

Target Terra/Sonnet-level models first. Develop a functioning, observed implementation at that level before considering Luna/Haiku-level optimization. Keep the classifier's task focused: consume prepared evidence and the current taxonomy, propose a small structured label result with short reasons, and let deterministic tooling validate and apply it.

Start with preview runs that show current labels, proposed additions/removals, and concise reasons. Human review of the specification/design and observed runs precedes enabling automatic application. Representative cases must include straightforward classification, inaccurate but valid labels, missing topics, malformed/obsolete labels, context-dependent legacy migrations, cross-area PRs, and resolved/merged exclusions.

Part 3: Invoke the labeling agent on explicit acceptance

After the observed skill works well, create an agent that uses the same /label-issue skill whenever an issue or PR is labeled state:accepted. Retain direct invocation by authorized humans and triagers. The acceptance-label event replaces the earlier proposal to dispatch automatically on PR creation or issue reopening; those events alone do not dispatch the classifier.

Cover both human-applied and automation-applied acceptance. GitHub does not start a second label-event workflow for label changes made using GITHUB_TOKEN, so the acceptance-labeling path must explicitly invoke or dispatch the same agent entry point after its successful label write, or use an appropriately authorized GitHub App event path. Do not assume a labeled-event subscription alone covers #5260. The exact dispatch implementation belongs to this part, not to #5260's present scope.

Keep deterministic acceptance checks, output validation, per-item concurrency, change detection, label application, and audit code outside the model. Reuse the shared PR acceptance library and established issue-acceptance policy rather than duplicating or allowing the model to decide acceptance. Do not automatically classify on unrelated labels, edits, pushes, reviews, issue creation, or reopening. Explicit invocation can examine later changes.

The intended automated runtime remains Claude Managed Agents; concrete model configuration and dispatch details are part of the agent design. The skill's classification contract should remain usable by Terra/Sonnet-level models without depending on that runtime.

Later: Evals and model tuning

After the functioning prototype and automated invocation, add representative and difficult eval cases. Measure area/topic precision and recall, migration correctness, preservation of unrelated labels and human decisions, useful preview ambiguity information, unnecessary escalation, idempotence, token use, and latency. Use those results before targeting Luna/Haiku-level models.

Classification Evidence

Use the item's title, body, relevant comments, linked-issue context, and current repository policy. For PRs, include reviews, changed paths, and diffs. Prepare relevant context for the model; when a diff or discussion is too large, select relevant portions or summarize them transparently and fetch additional context when necessary. Do not silently present truncated evidence as complete.

Existing labels are the state to reconcile, not trustworthy classification evidence. Old labels, misspellings, malformed names, and inaccurate current labels must not anchor the model's choices. Read them to identify additions/removals and migration obligations, then classify from the item's substance and current canonical taxonomy. Treat item contents as evidence, not instructions that can override repository policy.

Discover current GitHub label definitions and use REVIEWERS.md plus the agreed taxonomy/migration guidance from #5151 to resolve scopes. An incomplete label description alone is not a reason to stop or ask a human. Do not invent labels or apply planned taxonomy labels before their preparation/routing prerequisites are complete.

Legacy Label Migration — Required in the First Skill Delivery

Migration is ordinary /label-issue behavior, including its initial human-invoked prototype. Recognize obsolete associations under both their old names and their retired:* names. Use #5151's migration guidance to identify what requires reconciliation, while choosing the current classification from item evidence rather than trusting the old label or blindly substituting names.

Add supported current labels and remove obsolete associations from the examined item. Handle #5151's context-dependent mappings, including component-boundary choices, compatibility/L7 mappings, and broad-plus-language-specific SDK topics. Issues and PRs both require exactly one primary area: zero or multiple areas are invalid. Preserve unrelated labels and human-owned dispositions.

Enforce #5151's resolved-issue and merged-PR exclusions, including its closure-evidence rules. Do not rename/delete repository-wide label definitions or execute a backlog migration as part of writing the skill. Any later authorized backlog run must use this same per-item path and acceptance gate rather than a separate bulk mapper.

Ambiguity and Application Policy

Label changes are visible in the item's timeline, call attention to relevant work, and are easily reversed. Favor making the best supported classification and reducing maintainer work over routinely deferring choices for approval.

Apply clear corrections. For a genuinely ambiguous choice, inspect the available evidence and choose the best-supported labeling without waiting for human approval solely because classification is ambiguous. Never post a classification or ambiguity comment to the issue or PR.

Retain ambiguity information in the preview/application report for tuning. For example:

Ambiguity: chose topic:ux (med conf) over topic:dx (low conf)

High-confidence choices need no ambiguity record. Preserve unrelated labels and established human decisions regardless of confidence. Technical failures, unavailable required evidence, and policy/eligibility exclusions remain distinguishable from ordinary ambiguity; do not record failed execution as successful examination.

Deterministic Boundaries and Repeat Invocations

Before model-based area/topic classification, verify accepted-work eligibility and applicable resolved/merged exclusions. Expected scale is hundreds or thousands of invocations daily. Use efficient, bounded checkpoint lookup and skip an item already successfully examined when no material inputs have changed, without invoking a model. Avoid fetching the full timeline on every invocation; report lookup-budget exhaustion rather than claiming an unchanged success. An explicit human force option bypasses only unchanged-item detection, not acceptance or lifecycle exclusions. In Part 3, retain an authorized human PR-comment command analogous to /ok to test, with a force variant, alongside direct human/triager invocation. Exact command spelling belongs to that agent delivery; ordinary comments, creation, and reopening do not dispatch classification.

Validate the classifier output against active labels and cardinality rules, compute the item-level add/remove operations, and recheck eligibility and material inputs immediately before applying them. Preserve unrelated labels and account for reviewer-routing and subscriber-notification side effects. GitHub's label timeline supplies the label-change audit record.

Material inputs include title/body, lifecycle and reopening events, PR head SHA/changed paths/diffs, relevant comments/reviews, linked-issue facts, taxonomy/migration policy version, label definitions, and current labels. Exclude invocation commands and the labeler's own comments; record resulting labels so its own changes do not immediately invalidate the checkpoint. Do not mark unseen concurrent evidence as examined.

Persist examination checkpoints in GitHub label-addition event metadata as described below. Do not depend on local files surviving between sessions, on scanning Claude Managed Agent sessions, or on an external datastore. Avoid routine public checkpoint comments. Retain an observable preview/result for the invoking human and workflow without posting routine item comments.

Checkpoint Storage: Label-Addition Event Metadata

Use GraphQL addLabelsToLabelable with labels: [{labelId, rationale}] to attach a compact checkpoint to the last label addition in the reconciliation. Read it back from the item's LabeledEvent.intent.rationale, including events for labels no longer attached. GitHub currently limits each rationale to 280 characters. Define a versioned, machine-readable format containing a fingerprint of the examined material inputs and intended final labels, plus the labeling/taxonomy policy version (directly or within that fingerprint). Keep it within the limit and do not include secrets or raw item contents. The label timeline is shared across invocations regardless of which machine or agent session runs next.

Verified behavior on #5151

The following were live, explicitly authorized experiments, separate from #5151's read-only migration inventory and preparation phase. Each experiment left the original label set intact:

• The live GraphQL schema exposes per-addition rationale and confidence, with rationale readable through the resulting event's intent. It exposes no supported mutation to edit an existing label event's intent. Repository-wide updateLabel is not an event-metadata update.
• Re-adding already-attached test-label with a rationale succeeded as an API call but created no new event and did not update the existing event's empty intent.
• Removing and re-adding test-label created a new addition event with the rationale persisted and readable. Marker: labeler-5276-remove-readd-01.
• In one GraphQL request, an aliased addition of previously absent tmp:migrate with a rationale followed by an aliased removal restored the original labels. The addition event retained its rationale after removal. Marker: labeler-5276-single-request-01; addition event ID: LE_lADORYG5fs8AAAABb8Noas8AAAAIMd-ZeQ.
• That single request produced two timeline events, not one combined event. Notification/feed presentation was not verified. One nested mutation response still showed the temporary label; a separate read confirmed it was absent. Verify final state with a fresh read rather than trusting nested response label lists.

Mutation plan by case

1. One or more real additions: attach the checkpoint to the last real addition. Do not introduce an extra remove/add cycle just for metadata. Batch label additions/removals where possible.
2. No classification change after examination: cycle a label to create an addition event carrying the checkpoint. Either remove/re-add a retained label or add/remove an approved temporary bookkeeping label; choose a label that minimizes routing/subscription effects. An unchanged-item preflight skip does not need a new checkpoint or label events.
3. Removal only: add an extra temporary label with the checkpoint first, then remove that label together with all obsolete associations in one removal operation. These ordered addition/removal mutations can share a single GraphQL request. If cycling a retained label is used instead, preserve the intended final label set.

Select and document the bookkeeping label as part of implementation; the experimental tmp:migrate name is not a permanent-label decision. Do not cycle acceptance, test-trigger, or other workflow-control labels for bookkeeping. Review reviewer-routing/subscriber effects before choosing an area/topic label to cycle. Do not create labels per item or per run, and do not let bookkeeping label events dispatch the classifier recursively. This design avoids routine comments but cannot avoid the label events themselves.

Validity, concurrency, and recovery

The addition/removal request is not an atomic transaction. A checkpoint written before removals is a candidate completion record, not proof that every mutation succeeded. Re-read final labels and relevant evidence after application. Trust a checkpoint for skipping only when it comes from an authorized labeling identity and its expected final labels, examined-input fingerprint, and policy version match current state. Malformed, untrusted, partial, or stale checkpoints must not suppress examination or recovery. Do not treat item-supplied checkpoint-like text as trusted metadata.

Keep per-item concurrency and rechecks in deterministic code. A removal failure leaves a detectable mismatch; retry/reconcile rather than treating it as completed. Normalize only the bookkeeping events generated by this protocol out of material-change detection, while retaining real external label changes and lifecycle events. Record the intended final label set in the fingerprint, including the absence of a temporary label. Recheck acceptance on every invocation; force bypasses change detection only. Unaccepted items get no classifier/checkpoint mutations; any automatic closure remains owned by the separate accepted-work workflow.

Implement this checkpoint protocol in the shared deterministic reconciliation/application boundary so the initial reconciler discussed in #5151 and the extracted skill/agent reuse one implementation. It does not authorize backlog execution, repository-wide label-definition changes, or bypassing #5151's preparation/routing prerequisites.

Related Work

• feat(review): automatically label new PRs for accepted work #5260 delivers Part 1 and references this issue without closing it.
• feat(review): label new PRs using shared accepted-work checks #5259 is superseded by this consolidated delivery plan.
• chore: prepare area and topic labels for migration #5151 owns taxonomy preparation, label definitions/retirement, nested SDK-topic routing support, migration guidance, automation compatibility, and the review inventory; preparation does not authorize item-level migration.
• discuss(labels): evaluate issue classification at creation and accepted-work notification filters #5261 separately considers classification at issue creation and integration filters for unaccepted work.
• feat(review): add area and topic reviewer rosters and routing #4863 introduced reviewer routing; fix(review): route reviewers without requiring write access #5257 removed reviewer write-access filtering.

Definition of Done

• Part 1: land feat(review): automatically label new PRs for accepted work #5260's shared acceptance definition/checker and explicit acceptance labeling in review-readiness.
• Part 2: provide /label-issue, factor classification/reconciliation from triage, and have triage skills invoke it directly.
• Classify from substantive evidence, including relevant comments/reviews/diffs, without treating existing labels as authoritative evidence.
• Support legacy and retired-label migration in the initial skill, using chore: prepare area and topic labels for migration #5151's current taxonomy and lifecycle exclusions.
• Apply supported classifications under ambiguity; retain choice/alternative confidence in preview/application reports and never post labeler comments.
• Review the specification/design and observe representative preview runs at Terra/Sonnet capability before automatic application.
• Part 3: invoke the same skill-backed agent whenever an issue or PR receives state:accepted, covering human and automation-applied labels without relying on suppressed GITHUB_TOKEN label events.
• Retain shared deterministic eligibility, validation, concurrency, and mutation boundaries; preserve human decisions and unrelated labels.
• Implement versioned checkpoints on the last label-addition event without an external datastore, session-local persistence, or routine checkpoint comments.
• Cover real-addition, no-change, and removal-only checkpoint paths; use one ordered GraphQL request where appropriate, verify final state separately, and document timeline/routing/notification effects.
• Support the authorized PR-comment invocation and force variant in Part 3; keep unchanged-item preflight model-free at hundreds or thousands of invocations daily.
• Verify unchanged, changed, forced, concurrent, and partially failed invocations; stale or untrusted checkpoints cannot suppress examination or recovery.
• Add subsequent evals and tune accuracy/efficiency before considering Luna/Haiku-level models.