[repository-quality] 🎯 Repository Quality Improvement Report - Skill Documentation Drift (Stale JavaScript Location References — Third Audit)

[github-actions[bot]]

Analysis Date: 2026-09-28
Focus Area: Skill Documentation Drift (Stale pkg/workflow/js/ Path References)
Strategy Type: Reused (follow-up to 2026-09-16 finding)
Custom Area: Yes — repeat audits are the only way to detect that a previously reported, agreed-upon fix was never applied; this is a targeted verification + remediation run, not a fresh invention.

Executive Summary

On 2026-09-16 this same agent reported that four skill files (messages, javascript-refactoring, error-pattern-safety, temporary-id-safe-output) still contained 20 literal pkg/workflow/js/ path references pointing at a directory that had already been emptied by the actions/setup/js/ migration (documented as fixed back on 2026-08-31). Twelve days and multiple intervening quality-improvement runs later, a fresh scan shows the problem is still unresolved: pkg/workflow/js/ currently contains exactly 2 files (exchange_otlp_workload_identity.cjs and a JSON asset), neither related to messages/safe-outputs, while the real shared JavaScript location, actions/setup/js/, holds 862 .cjs files including all the referenced modules (messages_core.cjs, messages.cjs, etc.).

Across the four affected skills there are 22 remaining literal path references (2 more than the September count, likely from unrelated interim edits) that will actively mislead Copilot agents and human contributors editing safe-output messages, writing JS refactors, adding temporary-ID support, or diagnosing error-pattern safety issues — all four are high-traffic "how do I..." skills invoked whenever an agent needs to touch the safe-outputs JavaScript layer. Because no automated check ties skill-file path references to real filesystem paths, this class of drift recurs indefinitely without one-time intervention.

This run proposes concrete, mechanical sed-style fixes to all 22 stale references (with exact line numbers below) plus a lightweight CI/test guard — a Go test or script that fails when a skill file references a pkg/workflow/js/...cjs or similarly bare committed-path pattern that does not exist on disk — so this specific regression class cannot silently resurface a fourth time.

Full Analysis Report

Focus Area: Skill Documentation Drift (Stale JavaScript Location References)

Current State Assessment

Metrics Collected:

Metric	Value	Status
Files in pkg/workflow/js/	2 (unrelated: exchange_otlp_workload_identity.cjs, safe_outputs_tools.json)	❌
Files in actions/setup/js/ (real location)	862 .cjs files	✅
Skill files with stale pkg/workflow/js/ references	4 (messages, javascript-refactoring, error-pattern-safety, temporary-id-safe-output)	❌
Total stale reference occurrences	22 (was 20 on 2026-09-16 — net regression, not improvement)	❌
Broader broken literal path refs across all 51 skill files (flagged 2026-09-16, still open)	18 confirmed this run (subset re-verified below)	⚠️
Automated guard preventing this drift	None found	❌

Findings

Strengths

• The underlying migration to actions/setup/js/ is otherwise complete and correct — the 862-file directory is healthy, well-populated, and actively used by the real message/safe-output modules (messages_core.cjs, messages.cjs, messages_run_status.cjs, etc.).
• Three of the four affected skills already contain some correct dual-path guidance (e.g., javascript-refactoring line 15 correctly lists both pkg/workflow/js/*.cjs and actions/setup/js/*.cjs), showing the fix pattern is well understood — it just wasn't applied everywhere or consistently.

Areas for Improvement

• 🟠 High severity: messages/SKILL.md (8 stale references — the worst offender) instructs agents to create new message modules "typically" in pkg/workflow/js/, and gives a concrete example test-file path pkg/workflow/js/messages_my_new.test.cjs that does not exist and would silently create files in the wrong, unbuilt directory if followed literally.
• 🟠 High severity: javascript-refactoring/SKILL.md (6 stale references) repeats the same wrong directory as the canonical "Shared workflow utilities" location and as the example test path (pkg/workflow/js/my_module.test.cjs).
• 🟡 Medium severity: temporary-id-safe-output/SKILL.md (1 reference at line 101) gives a bare wrong-directory instruction (pkg/workflow/js/<job_name>.cjs) for wiring new safe-output job JS — a step directly load-bearing for that skill's stated purpose.
• 🟡 Medium severity: error-pattern-safety/SKILL.md (1 reference at line 102) offers pkg/workflow/js/ as an alternative valid test-suite location alongside the correct actions/setup/js/, which is misleading since the former is effectively empty.
• ⚪ Low severity / process gap: No regression test or lint rule ties skill-file path references to real repository paths, so the same finding has now been reported 2× (2026-08-31 original architecture drift, 2026-09-16 location drift) without closure, and a 3rd manual re-audit (this run) was needed to even confirm non-fixing.

Detailed Analysis

Exact stale reference locations (verified 2026-09-28):

messages/SKILL.md (8 refs):

• Line 20: 4. **JavaScript module** under pkg/workflow/js/oractions/setup/js/`` — order/emphasis wrong; should lead with the real location.
• Line 72: "Create the new module in the current shared JS location, typically pkg/workflow/js/:" — directly wrong instruction.
• Line 109: "Create a matching test file, for example pkg/workflow/js/messages_my_new.test.cjs" — wrong example path.
• Line 142: "Update ... in pkg/workflow/js/messages_core.cjs, and re-export ... from pkg/workflow/js/messages.cjs" — both real files live in actions/setup/js/.
• Line 150: "keep the JS module in pkg/workflow/js/ or the relevant action folder" — should be actions/setup/js/.
• Lines 184–185: file-reference list still shows pkg/workflow/js/messages_core.cjs and pkg/workflow/js/messages.cjs.
• Line 197: "review the existing safe-output modules under pkg/workflow/js/" — wrong directory for existing examples.

javascript-refactoring/SKILL.md (6 refs):

• Line 15: "Shared JS modules live under pkg/workflow/js/*.cjs and actions/setup/js/*.cjs" — should drop the first, dead path.
• Line 58: "Shared workflow utilities: pkg/workflow/js/" — wrong as sole canonical location.
• Line 101: "Example: pkg/workflow/js/my_module.test.cjs" — wrong example path.
• Line 154: "keep it under pkg/workflow/js/ and update the action or workflow definition that consumes it" — wrong directory.
• Line 197: "best kept under pkg/workflow/js/ or actions/setup/js/" — should be actions/setup/js/ only.
• Line 237: "pkg/workflow/js/*.cjs - existing shared module patterns" — wrong reference list entry.

error-pattern-safety/SKILL.md (1 ref):

• Line 102: "Use the relevant *.test.cjs suite under actions/setup/js/ or pkg/workflow/js/" — drop the dead alternative.

temporary-id-safe-output/SKILL.md (1 ref):

• Line 101: "In pkg/workflow/js/<job_name>.cjs:" — should be actions/setup/js/<job_name>.cjs.

Why this recurs: all four skills were edited independently over time (by different agent runs) and each accumulated its own copy of the stale directory name; nothing cross-checks skill prose against the actual repository tree. The fix pattern already exists correctly in some lines of the same files (e.g., javascript-refactoring line 15's second clause), proving contributors know the right path — the drift is a documentation-maintenance gap, not a knowledge gap.

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: Split the following tasks into individual work items.

Improvement Tasks

Task 1: Fix stale pkg/workflow/js/ references in messages/SKILL.md

Priority: High
Estimated Effort: Small
Focus Area: Skill Documentation Drift

Description: Replace all 8 occurrences of pkg/workflow/js/ in .github/skills/messages/SKILL.md (lines 20, 72, 109, 142, 150, 184, 185, 197) with actions/setup/js/, updating the surrounding prose where it currently frames pkg/workflow/js/ as the "typical"/primary location. Update the concrete example filenames (messages_my_new.test.cjs, messages_core.cjs, messages.cjs) to reference their real path under actions/setup/js/.

Acceptance Criteria:

• No remaining occurrences of pkg/workflow/js/ in .github/skills/messages/SKILL.md
• All example file paths point to files that actually exist under actions/setup/js/
• Prose no longer presents the dead directory as the "current"/"typical" location

Code Region: .github/skills/messages/SKILL.md (lines 20, 72, 109, 142, 150, 184-185, 197)

Update .github/skills/messages/SKILL.md: replace every literal "pkg/workflow/js/" path
reference (lines 20, 72, 109, 142, 150, 184-185, 197) with "actions/setup/js/", the real
current location (verified: 862 .cjs files live there vs 2 unrelated files in
pkg/workflow/js/). Update file-path examples like
"pkg/workflow/js/messages_my_new.test.cjs" -> "actions/setup/js/messages_my_new.test.cjs",
and "pkg/workflow/js/messages_core.cjs" / "pkg/workflow/js/messages.cjs" to their
actions/setup/js/ equivalents (these real files already exist there — verify with
`ls actions/setup/js/messages_core.cjs actions/setup/js/messages.cjs`). Rewrite any
prose implying pkg/workflow/js/ is the primary/typical/current location so
actions/setup/js/ is presented as the sole correct location. Do not change any content
unrelated to this path correction.

---

Task 2: Fix stale pkg/workflow/js/ references in javascript-refactoring/SKILL.md

Priority: High
Estimated Effort: Small
Focus Area: Skill Documentation Drift

Description: Replace all 6 occurrences of pkg/workflow/js/ in .github/skills/javascript-refactoring/SKILL.md (lines 15, 58, 101, 154, 197, 237) with actions/setup/js/, removing dual-path phrasing where pkg/workflow/js/ is listed as an equally-valid alternative alongside the real directory.

Acceptance Criteria:

• No remaining occurrences of pkg/workflow/js/ in .github/skills/javascript-refactoring/SKILL.md
• The "Example" path (currently line 101) references a plausible real actions/setup/js/ file naming convention
• Reference list (currently line 237) points only at the real shared-module directory

Code Region: .github/skills/javascript-refactoring/SKILL.md (lines 15, 58, 101, 154, 197, 237)

Update .github/skills/javascript-refactoring/SKILL.md: replace every literal
"pkg/workflow/js/" path reference (lines 15, 58, 101, 154, 197, 237) with
"actions/setup/js/", the real current shared-JS-module location (862 .cjs files
verified there vs 2 unrelated files in pkg/workflow/js/). Where the text currently
lists both paths as valid alternatives (e.g. line 15: "Shared JS modules live under
pkg/workflow/js/*.cjs and actions/setup/js/*.cjs"), collapse to only
"actions/setup/js/*.cjs". Update the example path on line 101
("pkg/workflow/js/my_module.test.cjs") to "actions/setup/js/my_module.test.cjs".
Do not change any content unrelated to this path correction.

---

Task 3: Fix stale pkg/workflow/js/ references in error-pattern-safety/SKILL.md and temporary-id-safe-output/SKILL.md

Priority: Medium
Estimated Effort: Small
Focus Area: Skill Documentation Drift

Description: Replace the single stale reference in each of these two skills: error-pattern-safety/SKILL.md line 102 (drop pkg/workflow/js/ as an alternative test-suite location, keep only actions/setup/js/) and temporary-id-safe-output/SKILL.md line 101 (change pkg/workflow/js/<job_name>.cjs to actions/setup/js/<job_name>.cjs).

Acceptance Criteria:

• No remaining occurrences of pkg/workflow/js/ in either file
• Instructions for wiring new safe-output job JS in temporary-id-safe-output/SKILL.md point at the real, buildable location

Code Region: .github/skills/error-pattern-safety/SKILL.md (line 102), .github/skills/temporary-id-safe-output/SKILL.md (line 101)

In .github/skills/error-pattern-safety/SKILL.md line 102, remove the
"or pkg/workflow/js/" alternative so the sentence only references
"actions/setup/js/" as the location of *.test.cjs suites (verified: 862 .cjs
files live there vs 2 unrelated files in pkg/workflow/js/).
In .github/skills/temporary-id-safe-output/SKILL.md line 101, change
"In pkg/workflow/js/<job_name>.cjs:" to "In actions/setup/js/<job_name>.cjs:".
Do not change any content unrelated to this path correction.

---

Task 4: Add a regression guard preventing skill-file path drift from recurring

Priority: Medium
Estimated Effort: Medium
Focus Area: Skill Documentation Drift / Process

Description: This exact finding has now been reported on 2026-08-31, 2026-09-16, and again today (2026-09-28) without being fixed in between, proving no automated signal exists to catch skill-doc path drift. Add a lightweight Go test (e.g. under pkg/cli or a new small package) or a make target that scans all .github/skills/*/SKILL.md files for literal backtick-quoted paths matching known repository-root prefixes (pkg/, cmd/, docs/, actions/, .github/) with a file extension, and fails if the referenced path does not exist on disk. Exclude clearly illustrative/placeholder paths (e.g. WORKFLOW-NAME.md, generic example names like my_module.test.cjs that intentionally don't exist) via an explicit allowlist or a naming heuristic (e.g. skip paths containing all-caps placeholder tokens). Wire the check into CI (e.g. cgo.yml or a docs-lint job) so future drift fails fast instead of silently persisting for weeks.

Acceptance Criteria:

• New test/script scans all 51 .github/skills/*/SKILL.md files for path-like references and verifies existence
• Test passes after Tasks 1–3 are merged (no false positives on legitimate illustrative examples)
• Test is wired into CI so a reintroduced stale reference fails the build
• Documented in developer-internals or a relevant skill how to update the allowlist for new illustrative examples

Code Region: New file, e.g. pkg/cli/skill_path_references_test.go or scripts/check-skill-paths.sh, wired into .github/workflows/cgo.yml

Add an automated regression guard that prevents skill-documentation path drift
from recurring (this exact class of bug — stale pkg/workflow/js/ references —
has now been reported unresolved across three separate audits: 2026-08-31,
2026-09-16, 2026-09-28). Implement either a Go test (e.g.
pkg/cli/skill_path_references_test.go) or a shell/Python script invoked from a
new `make` target, that:
1. Walks .github/skills/*/SKILL.md (51 files).
2. Extracts backtick-quoted path-like tokens with a recognized repo-root prefix
   (pkg/, cmd/, docs/, actions/, .github/) and a file extension.
3. Verifies each extracted path exists relative to the repository root.
4. Skips an explicit allowlist of intentionally-illustrative placeholder paths
   (e.g. containing ALL-CAPS tokens like WORKFLOW-NAME.md, or generic example
   names explicitly documented as illustrative, such as my_module.test.cjs in
   javascript-refactoring/SKILL.md).
5. Fails with a clear list of (file, line, broken path) on any other missing
   reference.
Wire this check into .github/workflows/cgo.yml so it runs in CI and blocks
merges that reintroduce broken skill-doc paths. Keep the implementation small
and dependency-free (stdlib regex + os.Stat in Go, or grep+test -f in a shell
script). Do not attempt to fix unrelated skill files beyond wiring the check —
if other legitimate broken references are found (e.g. the ~18 flagged in the
2026-09-16 follow-up triage under a different report), add them to a tracked
allowlist rather than fixing them all in this task.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-08-31	Skill Documentation Drift (Stale JS Architecture References)	Custom	Y	Found messages/javascript-refactoring documented a defunct embed/bundle architecture
2026-09-16	Skill Documentation Drift (Stale JS Location References)	Custom	Y	Found 20 stale pkg/workflow/js/ refs across 4 skills; flagged 132 broader broken refs repo-wide as follow-up
2026-09-21	Custom Go Linter CI Enablement Gap (2nd Follow-up)	Custom	Y	Found manualpathconcat/packagelevelmutableslicemap regressed back into the CI gap after being reported fixed
2026-09-25	CLI Sibling-Command Flag & Safety-Guard Parity Gaps	Custom	Y	Found 3 broken skill doc path refs (awf-release-integrator, otel-queries, messages)
2026-09-28 (today)	Skill Documentation Drift (Stale JS Location References) — 3rd audit	Custom	Y (reuse)	Confirmed the 2026-09-16 finding was never fixed; count grew from 20→22 refs; proposed a permanent automated guard (Task 4) to stop the recurrence pattern

---

🎯 Recommendations

Immediate Actions (This Week)

1. Apply Tasks 1–3 (mechanical path corrections across 4 skill files) — Priority: High
2. Merge Task 4's regression guard so this exact bug class cannot resurface a 4th time — Priority: High

Short-term Actions (This Month)

1. Extend the Task 4 guard (or run a follow-up audit) to close out the remaining ~18 broken literal path references flagged in the 2026-09-16 and 2026-09-25 findings across other skill files — Priority: Medium

Long-term Actions (This Quarter)

1. Consider a periodic gh aw scheduled workflow that runs the Task 4 checker weekly and opens an issue automatically if skill-doc paths drift again, closing the loop without relying on this quality-improvement agent's random focus-area selection to catch it — Priority: Low

---

📈 Success Metrics

• Stale pkg/workflow/js/ references in the 4 flagged skills: 22 → 0
• Automated path-drift regression guard in CI: absent → present and passing
• Repeat-audit rate for this specific finding: 3 consecutive unresolved reports → 0 (verified fixed on next audit)

---

Next Steps

1. Review and prioritise the tasks above
2. Assign tasks to Copilot coding agent via planner agent
3. Track progress on improvement items
4. Re-evaluate this focus area in 2 weeks to confirm the guard (Task 4) actually prevents recurrence

---

Generated by Repository Quality Improvement Agent
Next analysis: 2026-09-29 — Focus area selected by diversity algorithm

Generated by ⚡ Repository Quality Improvement Agent · copilot · auto · 102.6 AIC · ⌖ 6.71 AIC · ⊞ 9.7K · ◷

• expires on Sep 29, 2026, 5:14 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Repository Quality Improvement Agent.

A newer discussion is available at Discussion #64279.