diff --git a/docs/repo-assist.md b/docs/repo-assist.md index 37c0d69..d8b16cb 100644 --- a/docs/repo-assist.md +++ b/docs/repo-assist.md @@ -62,6 +62,24 @@ Each assessment examines a concrete source/test/docs area or existing check; lis Explicit command mode and scheduled maintenance-only mode receive no generic fallback plan. At the scheduled PR cap, Repo Assist stays with Tasks 6, 1, and 2 and their investigation/triage-only fallbacks. +### Verified outcomes, not just green runs + +Before publishing normal-mode outputs, Repo Assist checks the actual output batch and recorded agent tool calls: + +- A justified `noop` must contain a structured assessment of every selected and generic fallback task. Each entry links a concrete finding to an exact successful command and inspected path/API endpoint. Proactive tasks require source, test, documentation, or check evidence; issue lists and filename listings alone do not qualify. +- Missing tools/access, failed memory updates, insufficient time, and skipped required checks use `report_incomplete`, not a successful noop. Explicit command-mode stop requests are exempt; maintenance-only noops need only the restricted task plan. +- Labels and label removals are substantive work too: normal-mode public changes require the Monthly Activity update and cannot be accompanied by `noop`. + +The gate verifies recorded assessment coverage, not the semantic quality of every conclusion or a requirement to create changes. It fails explicitly if required artifacts or evidence are missing. Regression tests exercise the actual gate with complete, skipped, fabricated, failed-command, maintenance-only, stop-command, and label-only scenarios. + +GitHub reads use the pre-authenticated, read-only `gh` proxy rather than a broad GitHub MCP tool inventory. This avoids unauthenticated CLI fallbacks and MCP PR-enrichment lookups against contributor forks. Writes still go exclusively through safe outputs; no additional write permissions are granted. + +### Reliable memory and publication + +The agent prompt includes the exact memory entry keys, enums, examples, limits, and uniqueness rules enforced by the validator. All collection fields, including `checks`, are arrays. Updates preserve unrelated records, keep backups outside the memory directory, and sequence edit, parse, validation, and push. Failed validation is repaired, not bypassed by clearing blockers or resetting cursors; a zero-byte patch is not proof of an intended edit. + +Safe-output calls queue requests; they do not prove GitHub publication. Final responses and memory distinguish pending publication from live-confirmed results, and subsequent runs reconcile pending attempts before retrying. PR maintenance inspects the full delta against the current PR head and preflights the bundle against the 4096 KiB limit. Oversized updates and workflow-file changes unsupported by the configured credential are reported as blockers, not retried unchanged or hidden by relaxed protections. Protected-file updates request review. + ### Task 1: Issue Labelling Default weighting: dominates when the label backlog is large. @@ -192,6 +210,8 @@ When triggered this way, Repo Assist starts a new coding agent session immediate - `/repo-assist add documentation for the new API endpoints` - `/repo-assist review this PR and suggest improvements` +Information-only requests receive a reply on the triggering issue/PR, or a run summary for manual dispatches. A requested local report file does not implicitly authorize a pull request; publication must be requested. Explicit stop instructions produce no public changes or memory edits. + All the same guidelines apply (AI disclosure, running formatters/linters/tests, being polite and constructive). > NOTE: There are a few glitches with "/repo-assist" direct commands (which are meant to start a contextualised agent coding run directly) diff --git a/tests/test_repo_assist_generic_fallback.py b/tests/test_repo_assist_generic_fallback.py index 2a47168..bc3a255 100644 --- a/tests/test_repo_assist_generic_fallback.py +++ b/tests/test_repo_assist_generic_fallback.py @@ -81,6 +81,16 @@ def test_issue_backlog_does_not_suppress_fallback_plan(self): result = self.select([1, 2, 3], issues=[{"number": 1, "labels": []}]) self.assertEqual(result["generic_fallback_tasks"], GENERIC_TASKS) + def test_activity_trackers_do_not_bias_issue_weights(self): + issues = [ + {"number": 1, "title": "[repo-assist] Monthly Activity 2026-11", "labels": []}, + {"number": 2, "title": "Reproducible bug", "labels": [{"name": "bug"}]}, + ] + result = self.select([1, 2, 3], issues=issues) + self.assertEqual(result["open_issues"], 1) + self.assertEqual(result["unlabelled_issues"], 0) + self.assertEqual(result["weights"]["2"], 4) + def test_explicit_commands_have_no_generic_fallback_plan(self): for command in ["Investigate this issue", "Run Task 9", "\nFix the regression\n"]: with self.subTest(command=command): diff --git a/tests/test_repo_assist_maintenance.py b/tests/test_repo_assist_maintenance.py index 7a2c3d1..dad562d 100644 --- a/tests/test_repo_assist_maintenance.py +++ b/tests/test_repo_assist_maintenance.py @@ -18,7 +18,10 @@ def script_after(marker, indent): SELECTION = script_after(" python3 - << 'EOF'\n", 6) -CAPACITY = script_after(" node <<'EOF'\n", 10) +CAPACITY = textwrap.dedent( + WORKFLOW.split(" - name: Enforce scheduled PR capacity\n", 1)[1] + .split(" node <<'EOF'\n", 1)[1].split("\n EOF", 1)[0] +) class MaintenanceModeTests(unittest.TestCase): @@ -71,7 +74,7 @@ def test_manual_and_command_events_remain_exempt(self): def test_cap_no_longer_gates_activation(self): self.assertNotIn("needs.pre_activation.outputs.check_result", WORKFLOW) - self.assertEqual(WORKFLOW.count("if: github.event_name == 'schedule'"), 2) + self.assertEqual(WORKFLOW.count("if: github.event_name == 'schedule'"), 1) self.assertIn('--paginate "repos/$GITHUB_REPOSITORY/pulls', WORKFLOW) def check_capacity(self, count, requests, other_prs=0, api_failure=False): diff --git a/tests/test_repo_assist_outcomes.py b/tests/test_repo_assist_outcomes.py new file mode 100644 index 0000000..08b3fb0 --- /dev/null +++ b/tests/test_repo_assist_outcomes.py @@ -0,0 +1,322 @@ +import copy +from datetime import datetime, timezone +import json +import os +from pathlib import Path +import subprocess +import tempfile +import textwrap +import unittest + + +WORKFLOW = (Path(__file__).parents[1] / "workflows/repo-assist.md").read_text() +OUTCOME = textwrap.dedent( + WORKFLOW.split(" - name: Validate Repo Assist outcome\n", 1)[1] + .split(" node <<'EOF'\n", 1)[1].split("\n EOF", 1)[0] +) +MEMORY = textwrap.dedent( + WORKFLOW.split(" script: |\n", 1)[1].split("\nsafe-outputs:", 1)[0] +) + + +def assessment(task, command="sed -n '1,160p' tests/Tests.fs", path="tests/Tests.fs", + kind="test"): + return { + "task": task, + "result": "Inspected concrete assertions; no justified change identified", + "evidence": [{"command": command, "path": path, "kind": kind}], + } + + +def trace(command, content="", success=True): + return [ + {"type": "tool.execution_start", "data": { + "toolCallId": "read-tests", "toolName": "bash", + "arguments": {"command": command}, + }}, + {"type": "tool.execution_complete", "data": { + "toolCallId": "read-tests", "success": success, + "result": {"content": content}, + }}, + ] + + +class OutcomeTests(unittest.TestCase): + def check(self, items=None, assessments=None, selection=None, events=None, + command="", omit=None): + entries = assessments if assessments is not None else [ + assessment(task) for task in range(1, 11) + ] + if items is None: + items = [{"type": "noop", "message": json.dumps({ + "outcome": "justified_noop", "assessments": entries, + })}] + state = selection or { + "selected_tasks": [1, 2, 3], + "generic_fallback_tasks": [9, 8, 10, 7, 4, 5], + "maintenance_only": False, + } + calls = trace("sed -n '1,160p' tests/Tests.fs") if events is None else events + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + files = { + "agent_output.json": json.dumps({"items": items}), + "task_selection.json": json.dumps(state), + "agent-session.jsonl": "\n".join(map(json.dumps, calls)), + } + for name, value in files.items(): + if name != omit: + (root / name).write_text(value) + return subprocess.run( + ["node", "-e", OUTCOME], + env=dict(os.environ, OUTPUT_DIR=directory, REPO_ASSIST_COMMAND=command), + capture_output=True, text=True, check=False, + ) + + def report(self): + return { + "type": "update_issue", "issue_number": 10, + "body": "## Suggested Actions for Maintainer\n\n## Run History\n", + } + + def test_complete_recorded_noop_passes(self): + result = self.check() + self.assertEqual(result.returncode, 0, result.stderr) + + def test_empty_issue_queue_does_not_excuse_missing_generic_work(self): + result = self.check(assessments=[assessment(task) for task in [1, 2, 3]]) + self.assertNotEqual(result.returncode, 0) + self.assertIn("was not assessed", result.stderr) + + def test_issue_metadata_cannot_substitute_for_source_assessment(self): + command = "gh api repos/example/project/issues" + entries = [assessment(task, command, "repos/example/project/issues", "github") + for task in range(1, 11)] + result = self.check(assessments=entries, events=trace(command)) + self.assertNotEqual(result.returncode, 0) + self.assertIn("concrete source/test/docs/check", result.stderr) + + def test_unobserved_command_cannot_support_a_completion_claim(self): + result = self.check(events=[]) + self.assertNotEqual(result.returncode, 0) + self.assertIn("absent or failed command", result.stderr) + + def test_failed_shell_edit_does_not_become_success_from_tool_envelope(self): + result = self.check(events=trace( + "sed -n '1,160p' tests/Tests.fs", + "jq: Cannot index array with string\n", + )) + self.assertNotEqual(result.returncode, 0) + self.assertIn("absent or failed command", result.stderr) + + def test_failed_tool_is_not_evidence(self): + result = self.check(events=trace( + "sed -n '1,160p' tests/Tests.fs", success=False, + )) + self.assertNotEqual(result.returncode, 0) + + def test_running_command_is_not_completed_evidence(self): + result = self.check(events=trace( + "sed -n '1,160p' tests/Tests.fs", "Process still running", + )) + self.assertNotEqual(result.returncode, 0) + + def test_error_text_in_source_is_not_the_commands_exit_status(self): + result = self.check(events=trace( + "sed -n '1,160p' tests/Tests.fs", + 'let message = "exit code 5"\n', + )) + self.assertEqual(result.returncode, 0, result.stderr) + + def test_listing_files_is_not_source_inspection(self): + command = "ls tests/Tests.fs" + result = self.check( + assessments=[assessment(task, command) for task in range(1, 11)], + events=trace(command), + ) + self.assertNotEqual(result.returncode, 0) + + def test_task_selection_file_is_not_source_evidence(self): + command = "cat /tmp/gh-aw/task_selection.json" + result = self.check( + assessments=[assessment(task, command, "/tmp/gh-aw/task_selection.json") + for task in range(1, 11)], + events=trace(command), + ) + self.assertNotEqual(result.returncode, 0) + + def test_duplicate_tasks_cannot_hide_a_skipped_task(self): + result = self.check(assessments=[assessment(1), assessment(1)]) + self.assertNotEqual(result.returncode, 0) + self.assertIn("unique integers", result.stderr) + + def test_missing_result_or_evidence_is_rejected(self): + for field in ["result", "evidence"]: + with self.subTest(field=field): + entry = assessment(1) + del entry[field] + result = self.check(assessments=[entry]) + self.assertNotEqual(result.returncode, 0) + + def test_path_must_appear_in_recorded_command(self): + entries = [assessment(task, path="docs/nonexistent.md") for task in range(1, 11)] + result = self.check(assessments=entries) + self.assertNotEqual(result.returncode, 0) + self.assertIn("inspected path", result.stderr) + + def test_invalid_evidence_kind_is_rejected(self): + result = self.check(assessments=[assessment(1, kind="guess")]) + self.assertNotEqual(result.returncode, 0) + self.assertIn("invalid evidence kind", result.stderr) + + def test_maintenance_noop_needs_no_generic_sweep(self): + command = "gh api repos/example/project/issues" + result = self.check( + selection={"selected_tasks": [6, 1, 2], "generic_fallback_tasks": [], + "maintenance_only": True}, + assessments=[assessment(task, command, "repos/example/project/issues", "github") + for task in [6, 1, 2]], + events=trace(command), + ) + self.assertEqual(result.returncode, 0, result.stderr) + + def test_corrupted_selection_cannot_skip_assessments(self): + for selected, fallback in [([], []), ([1, 2, 3], []), ([1, 1, 2], [])]: + with self.subTest(selected=selected, fallback=fallback): + result = self.check(selection={ + "selected_tasks": selected, "generic_fallback_tasks": fallback, + "maintenance_only": False, + }) + self.assertNotEqual(result.returncode, 0) + + def test_explicit_stop_is_exempt_and_needs_no_session_or_selection(self): + result = self.check( + items=[{"type": "noop", "message": "Explicit stop: no action taken"}], + command="Do not do anything. No PR or comments.", omit="agent-session.jsonl", + ) + self.assertEqual(result.returncode, 0, result.stderr) + + def test_blank_command_is_still_normal_mode(self): + result = self.check(items=[{"type": "noop", "message": "Nothing to do"}], command=" \n") + self.assertNotEqual(result.returncode, 0) + + def test_incomplete_is_distinct_from_justified_noop(self): + result = self.check(items=[{ + "type": "report_incomplete", "message": "Memory could not be saved", + }], omit="agent-session.jsonl") + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("not a justified noop", result.stdout) + + def test_missing_evidence_artifact_fails_explicitly(self): + for name in ["task_selection.json", "agent-session.jsonl", "agent_output.json"]: + with self.subTest(name=name): + self.assertNotEqual(self.check(omit=name).returncode, 0) + + def test_label_work_requires_monthly_reporting(self): + result = self.check(items=[{"type": "add_labels", "labels": ["bug"]}]) + self.assertNotEqual(result.returncode, 0) + self.assertIn("Monthly Activity update", result.stderr) + + def test_label_changes_with_reporting_are_not_noops(self): + result = self.check(items=[ + {"type": "add_labels", "labels": ["bug"]}, self.report(), + ]) + self.assertEqual(result.returncode, 0, result.stderr) + + def test_monthly_bookkeeping_alone_cannot_bypass_assessment(self): + result = self.check(items=[self.report()]) + self.assertNotEqual(result.returncode, 0) + self.assertIn("bookkeeping alone", result.stderr) + + def test_mutation_and_noop_batch_is_rejected_even_with_report(self): + result = self.check(items=[ + {"type": "add_labels"}, self.report(), {"type": "noop", "message": "No PR"}, + ]) + self.assertNotEqual(result.returncode, 0) + self.assertIn("do not emit noop", result.stderr) + + def test_unrelated_issue_is_not_a_monthly_report(self): + result = self.check(items=[{ + "type": "create_issue", "title": "[repo-assist] Other issue", + "body": self.report()["body"], + }]) + self.assertNotEqual(result.returncode, 0) + + def test_new_monthly_tracker_allows_handler_applied_prefix(self): + month = datetime.now(timezone.utc).strftime("%Y-%m") + for prefix in ["", "[repo-assist] "]: + with self.subTest(prefix=prefix): + result = self.check(items=[ + {"type": "add_labels", "labels": ["bug"]}, + {"type": "create_issue", "title": f"{prefix}Monthly Activity {month}", + "body": self.report()["body"]}, + ]) + self.assertEqual(result.returncode, 0, result.stderr) + + def test_information_only_command_needs_no_monthly_reporting(self): + result = self.check(items=[{ + "type": "add_comment", "body": "Requested investigation", + }], command="Investigate and reply; do not open a PR") + self.assertEqual(result.returncode, 0, result.stderr) + + +class MemoryContractTests(unittest.TestCase): + def initial(self): + return { + "version": 1, "cursors": {"labelling_after": None, "investigation_after": None}, + "issues": [], "fixes": [], "checks": [], "completed_actions": [], "priorities": [], + } + + def validate(self, notes): + with tempfile.TemporaryDirectory() as directory: + Path(directory, "notes.json").write_text(json.dumps(notes)) + return subprocess.run( + ["node", "-e", 'const memoryRoot = process.env.MEMORY_ROOT;\n' + MEMORY], + env=dict(os.environ, MEMORY_ROOT=directory), + capture_output=True, text=True, check=False, + ) + + def test_prompt_examples_match_the_actual_memory_validator(self): + notes = self.initial() + table = WORKFLOW.split("### Exact memory contract\n", 1)[1].split( + "Use the actual current date", 1)[0] + for line in table.splitlines(): + if line.startswith("| `"): + name = line.split("`")[1] + notes[name] = [json.loads(line.split("`")[3])] + self.assertTrue(all(notes[name] for name in [ + "issues", "fixes", "checks", "completed_actions", "priorities", + ])) + result = self.validate(notes) + self.assertEqual(result.returncode, 0, result.stderr) + + def test_checks_are_an_array_not_a_map(self): + notes = self.initial() + notes["checks"] = {"issue_sweep": "looked at issue titles"} + result = self.validate(notes) + self.assertNotEqual(result.returncode, 0) + self.assertIn("checks must contain", result.stderr) + + def test_bad_issue_state_reports_allowed_enum(self): + notes = self.initial() + notes["issues"] = [{ + "number": 473, "state": "blocked by compiler", + "updated_at": "2026-11-09", "note": "Upstream fix pending", + }] + result = self.validate(notes) + self.assertNotEqual(result.returncode, 0) + self.assertIn("state must be one of", result.stderr) + self.assertIn("deferred", result.stderr) + + def test_schema_errors_name_missing_fields_without_erasing_state(self): + notes = self.initial() + notes["issues"] = [{"number": 473, "state": "deferred"}] + before = copy.deepcopy(notes) + result = self.validate(notes) + self.assertNotEqual(result.returncode, 0) + self.assertIn("requires exactly number, state, updated_at, note", result.stderr) + self.assertEqual(notes, before) + + +if __name__ == "__main__": + unittest.main() diff --git a/workflows/repo-assist.md b/workflows/repo-assist.md index f11dd24..4db7c70 100644 --- a/workflows/repo-assist.md +++ b/workflows/repo-assist.md @@ -13,6 +13,8 @@ description: | - Maintains a persistent memory of work done and what remains Always polite, constructive, and mindful of the project's goals. +intent: Reduce maintainer effort through verified repository improvements and useful triage without duplicate work or unsupported completion claims. + on: schedule: every 12h workflow_dispatch: @@ -64,7 +66,7 @@ checkout: tools: github: - toolsets: [all] + mode: gh-proxy min-integrity: none # This workflow is allowed to examine and comment on any issues or PRs bash: true repo-memory: @@ -107,9 +109,9 @@ tools: const issueStates = new Set(["commented", "awaiting_clarification", "resolution_recommended", "deferred", "awaiting_approval"]); if (!Array.isArray(data.issues) || data.issues.length > 100) fail("issues must be an array of at most 100 entries"); for (const [index, entry] of data.issues.entries()) { - if (!exactKeys(entry, ["number", "state", "updated_at", "note"])) fail(`invalid issues entry at index ${index}`); + if (!exactKeys(entry, ["number", "state", "updated_at", "note"])) fail(`issues[${index}] requires exactly number, state, updated_at, note`); if (!Number.isInteger(entry.number) || entry.number < 1) fail(`invalid issue number at index ${index}`); - if (!issueStates.has(entry.state)) fail(`invalid issue state at index ${index}`); + if (!issueStates.has(entry.state)) fail(`issues[${index}].state must be one of ${[...issueStates].join(", ")}`); if (!validDate(entry.updated_at)) fail(`invalid issue date at index ${index}`); if (!validText(entry.note, 300)) fail(`invalid issue note at index ${index}`); } @@ -118,7 +120,7 @@ tools: const fixStates = new Set(["open", "merged", "closed", "blocked"]); if (!Array.isArray(data.fixes) || data.fixes.length > 50) fail("fixes must be an array of at most 50 entries"); for (const [index, entry] of data.fixes.entries()) { - if (!exactKeys(entry, ["issue", "pr", "branch", "status", "updated_at", "note"])) fail(`invalid fixes entry at index ${index}`); + if (!exactKeys(entry, ["issue", "pr", "branch", "status", "updated_at", "note"])) fail(`fixes[${index}] requires exactly issue, pr, branch, status, updated_at, note`); if (!Number.isInteger(entry.issue) || entry.issue < 1) fail(`invalid fix issue at index ${index}`); if (entry.pr !== null && (!Number.isInteger(entry.pr) || entry.pr < 1)) fail(`invalid fix PR at index ${index}`); if (entry.branch !== null && !validText(entry.branch, 120)) fail(`invalid fix branch at index ${index}`); @@ -132,7 +134,7 @@ tools: const checkAreas = new Set(["dependencies", "ci", "tooling", "build", "code", "docs", "qa", "hygiene", "performance", "tests", "release", "repo_assist_prs"]); if (!Array.isArray(data.checks) || data.checks.length > checkAreas.size) fail("checks must contain at most one entry per area"); for (const [index, entry] of data.checks.entries()) { - if (!exactKeys(entry, ["area", "checked_at", "result", "follow_up"])) fail(`invalid checks entry at index ${index}`); + if (!exactKeys(entry, ["area", "checked_at", "result", "follow_up"])) fail(`checks[${index}] requires exactly area, checked_at, result, follow_up`); if (!checkAreas.has(entry.area)) fail(`invalid check area at index ${index}`); if (!validDate(entry.checked_at)) fail(`invalid check date at index ${index}`); if (!validText(entry.result, 300)) fail(`invalid check result at index ${index}`); @@ -150,7 +152,7 @@ tools: if (!Array.isArray(data.priorities) || data.priorities.length > 20) fail("priorities must be an array of at most 20 entries"); for (const [index, entry] of data.priorities.entries()) { - if (!exactKeys(entry, ["task", "item", "note"])) fail(`invalid priorities entry at index ${index}`); + if (!exactKeys(entry, ["task", "item", "note"])) fail(`priorities[${index}] requires exactly task, item, note`); if (!Number.isInteger(entry.task) || entry.task < 1 || entry.task > 10) fail(`invalid priority task at index ${index}`); if (!validText(entry.item, 100)) fail(`invalid priority item at index ${index}`); if (!validText(entry.note, 300)) fail(`invalid priority note at index ${index}`); @@ -182,8 +184,9 @@ safe-outputs: target: "*" required-title-prefix: "[repo-assist] " max: 4 + max-patch-size: 4096 protected-files: - policy: allowed + policy: request-review exclude: - CHANGELOG.md - README.md @@ -207,13 +210,104 @@ safe-outputs: jobs: safe_outputs: pre-steps: - - name: Download outputs for PR capacity check - if: github.event_name == 'schedule' + - name: Download outputs and assessment evidence uses: actions/download-artifact@v8 with: - pattern: "{agent,agent-output-fallback}" + pattern: "{agent,agent-output-fallback,repo-assist-selection}" merge-multiple: true path: ${{ runner.temp }}/repo-assist-capacity + - name: Validate Repo Assist outcome + env: + OUTPUT_DIR: ${{ runner.temp }}/repo-assist-capacity + REPO_ASSIST_COMMAND: ${{ needs.activation.outputs.text || inputs.command }} + run: | + node <<'EOF' + const fs = require("node:fs"); + const path = require("node:path"); + const read = name => fs.readFileSync(path.join(process.env.OUTPUT_DIR, name), "utf8"); + const fail = message => { throw new Error(`Repo Assist outcome: ${message}`); }; + const { items } = JSON.parse(read("agent_output.json")); + if (!Array.isArray(items)) fail("missing safe-output items"); + if ((process.env.REPO_ASSIST_COMMAND || "").trim()) { + console.log("Explicit command: normal-mode assessment and reporting rules do not apply."); + process.exit(0); + } + const diagnostics = new Set(["noop", "report_incomplete", "missing_tool", "missing_data", "report_missing_tool", "report_missing_data"]); + const mutations = items.filter(item => !diagnostics.has(item.type)); + const noops = items.filter(item => item.type === "noop"); + if (mutations.length) { + if (noops.length) fail("public changes, including labels, are substantive work; do not emit noop"); + const month = new Date().toISOString().slice(0, 7); + const isReport = item => + ["create_issue", "update_issue"].includes(item.type) && + (item.type !== "create_issue" || + (typeof item.title === "string" && item.title.replace(/^\[repo-assist\]\s*/, "") === `Monthly Activity ${month}`)) && + typeof item.body === "string" && + item.body.includes("## Suggested Actions for Maintainer") && + item.body.includes("## Run History"); + if (!items.some(isReport)) fail("substantive work requires a Monthly Activity update"); + if (mutations.every(item => isReport(item) || + (item.type === "update_issue" && item.status === "closed" && !item.body))) { + fail("Monthly Activity bookkeeping alone is not substantive work; assess tasks before noop"); + } + } else if (items.some(item => item.type === "report_incomplete")) { + console.log("Incomplete work explicitly reported; not a justified noop."); + } else { + if (noops.length !== 1) fail("expected one evidence-backed noop or report_incomplete"); + const selection = JSON.parse(read("task_selection.json")); + if (!Array.isArray(selection.selected_tasks) || selection.selected_tasks.length !== 3 || + new Set(selection.selected_tasks).size !== 3 || + selection.selected_tasks.some(task => !Number.isInteger(task) || task < 1 || task > 10) || + typeof selection.maintenance_only !== "boolean") fail("invalid task selection"); + const fallback = selection.maintenance_only ? [] : + [9, 8, 10, 7, 4, 5].filter(task => !selection.selected_tasks.includes(task)); + if (JSON.stringify(selection.generic_fallback_tasks) !== JSON.stringify(fallback)) fail("invalid fallback plan"); + const required = new Set([...selection.selected_tasks, ...selection.generic_fallback_tasks]); + const assessment = JSON.parse(noops[0].message); + if (assessment.outcome !== "justified_noop" || !Array.isArray(assessment.assessments)) { + fail("noop message must contain the structured justified_noop assessment"); + } + const events = read("agent-session.jsonl").split(/\r?\n/).filter(Boolean).map(line => JSON.parse(line)); + const requests = new Map(events.filter(event => event.type === "tool.execution_start") + .map(event => [event.data.toolCallId, event.data])); + const commands = new Set(events.filter(event => event.type === "tool.execution_complete") + .filter(event => { + const content = event.data.result?.content; + const completed = typeof content === "string" && + content.match(/(?:^|\n)]* completed with exit code (\d+)>\s*$/); + return event.data.success === true && typeof content === "string" && + completed && completed[1] === "0"; + }) + .map(event => requests.get(event.data.toolCallId)) + .filter(request => request?.toolName === "bash") + .map(request => request.arguments.command)); + const seen = new Set(); + for (const entry of assessment.assessments) { + if (!Number.isInteger(entry.task) || entry.task < 1 || entry.task > 10 || seen.has(entry.task)) { + fail("assessment task numbers must be unique integers from 1 to 10"); + } + seen.add(entry.task); + if (typeof entry.result !== "string" || !entry.result.trim() || + !Array.isArray(entry.evidence) || entry.evidence.length === 0) fail(`Task ${entry.task} needs results and evidence`); + for (const evidence of entry.evidence) { + if (!commands.has(evidence.command)) fail(`Task ${entry.task} references an absent or failed command`); + if (!["source", "test", "docs", "check", "github"].includes(evidence.kind)) fail(`Task ${entry.task} has an invalid evidence kind`); + if (typeof evidence.path !== "string" || !evidence.path.trim() || + !evidence.command.includes(evidence.path)) fail(`Task ${entry.task} needs an inspected path or GitHub API endpoint`); + } + if (entry.task >= 4 && entry.task !== 6 && !entry.evidence.some(evidence => + !path.isAbsolute(evidence.path) && !evidence.path.split(/[\\/]/).includes("..") && + !/^(?:https?:|repos\/)/.test(evidence.path) && + /^(?:source|test|docs|check)$/.test(evidence.kind) && + /\b(?:cat|sed|head|tail|rg|grep|dotnet|npm|npx|python3?|node|make|cargo|go)\b/.test(evidence.command) && + !/\b(?:ls|find)\b|--files\b/.test(evidence.command))) { + fail(`Task ${entry.task} needs concrete source/test/docs/check evidence, not just issue metadata`); + } + } + for (const task of required) if (!seen.has(task)) fail(`Task ${task} was not assessed`); + console.log(`Justified noop: ${seen.size} tasks have recorded command evidence.`); + } + EOF - name: Enforce scheduled PR capacity if: github.event_name == 'schedule' env: @@ -242,6 +336,15 @@ jobs: } EOF +post-steps: + - name: Upload Repo Assist task selection + uses: actions/upload-artifact@v7 + with: + name: repo-assist-selection + path: /tmp/gh-aw/task_selection.json + if-no-files-found: error + retention-days: 7 + steps: - name: Initialize Repo Assist memory env: @@ -273,7 +376,7 @@ steps: mkdir -p /tmp/gh-aw # Fetch open issues with labels (up to 500) - gh issue list --state open --limit 500 --json number,labels > /tmp/gh-aw/issues.json + gh issue list --state open --limit 500 --json number,title,labels > /tmp/gh-aw/issues.json # Fetch all open PRs so the capacity decision cannot miss older PRs. gh api --hostname "${GITHUB_SERVER_URL#https://}" --paginate "repos/$GITHUB_REPOSITORY/pulls?state=open&per_page=100" \ @@ -288,6 +391,7 @@ steps: with open('/tmp/gh-aw/prs.json') as f: prs = json.load(f) + issues = [i for i in issues if not i.get('title', '').startswith('[repo-assist] Monthly Activity ')] open_issues = len(issues) unlabelled = sum(1 for i in issues if not i.get('labels')) repo_assist_prs = sum(1 for p in prs if p['title'].startswith('[repo-assist]')) @@ -381,8 +485,18 @@ Take heed of **instructions**: "${{ steps.sanitized.outputs.text || inputs.comma If these are non-empty (not ""), then you have been triggered via `/repo-assist ` (or by the user setting `inputs.command` in a manual `workflow_dispatch`). Follow the user's instructions instead of the normal scheduled workflow. Focus exclusively on those instructions. Apply all the same guidelines (read AGENTS.md, run formatters/linters/tests, be polite, use AI disclosure). Skip the weighted task selection and Task 11 reporting, and instead directly do what the user requested. If no specific instructions were provided (empty or blank), proceed with the normal scheduled workflow below. +Distinguish a request for information, investigation, a report, or a proposed fix from permission to publish repository changes. For information-only requests, reply on the triggering issue/PR with `add_comment`; use the run summary for manual dispatches. A requested local report file is not, by itself, a request for a PR. Do not open a PR or issue unless the user asks for that publication. Honour explicit stop/no-action instructions with only `noop`, without public changes or memory edits. + Then exit - do not run the normal workflow after completing the instructions. +## Evidence and publication + +Use the pre-authenticated, read-only `gh` proxy for GitHub reads, always specifying this repository with `--repo` or a `repos///...` API endpoint. Do not export tokens, use unauthenticated `gh` as a fallback, or use `gh`/API writes; publish only through safe-output tools. + +An accepted safe-output call records a **queued request**, not a completed GitHub write. In final text and memory, say "queued" or "pending publication" until a later live query confirms the issue, comment, commit, or PR exists. At the start of the next applicable run, reconcile pending attempts with actual PR heads and safe-output failure reports before retrying or marking them complete. Do not repeat a rejected batch unchanged or claim a later human merge proves your earlier push succeeded. + +Only claim checks actually performed in this run. Listing filenames or counting API references is not test coverage; inspect assertions and relevant behavior. A type-check is not a full build, an attempted build is not a passing build, and a dependency query with `Current: MISSING` is not an installed/tested baseline. Record commands, their exit status, and concrete findings; stale memory alone cannot establish that source, dependencies, docs, or tests are unchanged. + ## Non-Command Mode You are Repo Assist for `${{ github.repository }}`. Your job is to make forward progress by investigating and clarifying issues, resolving issues where possible, identifying improvements, and fixing bugs and feature requests by creating pull requests. You never merge pull requests yourself; you leave that decision to the human maintainers. @@ -414,13 +528,30 @@ The schema stores only: - `completed_actions`: Monthly Activity actions checked off by a maintainer, so they are not proposed again - `priorities`: a short queue of concrete follow-up work +### Exact memory contract + +The root has exactly `version`, `cursors`, `issues`, `fixes`, `checks`, `completed_actions`, and `priorities`; `version` is `1`. `cursors` has exactly `labelling_after` and `investigation_after`, each a positive issue number or `null`. The other five fields are **arrays, never objects/maps**. Each entry has exactly these keys: + +| Array (maximum entries) | Entry | +|---|---| +| `issues` (100) | `{"number":130,"state":"deferred","updated_at":"2026-11-09","note":"Needs an interactive reproduction"}` | +| `fixes` (50) | `{"issue":130,"pr":null,"branch":null,"status":"blocked","updated_at":"2026-11-09","note":"Pending publication; verify the next run"}` | +| `checks` (12) | `{"area":"tests","checked_at":"2026-11-09","result":"Inspected cancellation assertions in tests/Tests.fs","follow_up":null}` | +| `completed_actions` (100) | `{"key":"review-pr-372","completed_at":"2026-11-09"}` | +| `priorities` (20) | `{"task":9,"item":"Cancellation coverage","note":"Check disposal after cancellation"}` | + +Use the actual current date in `YYYY-MM-DD` form. Issue/fix numbers are positive integers; `pr` may be `null`; `branch` may be `null` or a nonempty string of at most 120 characters. Issue states are exactly `commented`, `awaiting_clarification`, `resolution_recommended`, `deferred`, `awaiting_approval`. Fix statuses are exactly `open`, `merged`, `closed`, `blocked`; use `blocked` with a pending-publication note for unconfirmed attempts. Check areas are exactly `dependencies`, `ci`, `tooling`, `build`, `code`, `docs`, `qa`, `hygiene`, `performance`, `tests`, `release`, `repo_assist_prs`. All notes, check results, and non-null follow-ups are nonempty strings of at most 300 characters; action keys and priority items at most 100. Priority task numbers are integers 1-10. Issue numbers, fix issue numbers, non-null fix PRs, check areas, action keys, and priority task/item pairs are unique within their arrays. + Keep notes terse and current. Replace superseded entries, remove resolved issue records and closed fix records once they are no longer needed for duplicate prevention, and never store run-by-run narration, exhaustive label histories, stale PR inventories, copied GitHub content, or facts that can be cheaply queried again. Stay within the schema's array and text limits; do not create another memory file. -After every change to `notes.json`, run `jq empty /tmp/gh-aw/repo-memory/default/notes.json`, then call `push_repo_memory`. A successful tool result is required before finishing the run. If either check reports an error, repair `notes.json` and retry both checks. Prefer `jq` with a temporary file and atomic rename over manual partial JSON edits. +Before editing, save a backup **outside** the memory directory, for example `/tmp/gh-aw/notes.before.json`. Write a candidate outside that directory, check the shell exit code and `jq empty`, and only then atomically replace `notes.json`. Preserve unrelated entries. For example, upsert an array entry using `.checks |= (map(select(.area != $entry.area)) + [$entry])` with `--argjson entry`, not `.checks.tests = ...`. + +**Sequentially**, after the successful edit completes, call `push_repo_memory`; never run it in parallel with the edit. If validation fails, restore the backup, repair the specific entry, and retry the edit, parse, and push. Never empty arrays, reset cursors, or discard blockers merely to pass validation. A zero-byte patch does not prove an intended edit succeeded: verify the changed fields. If the intended state cannot be saved, use `report_incomplete`, not a successful noop. A failed shell command is a failure even if its tool envelope says success. **Important**: Memory may not be 100% accurate. Issues may have been created, closed, or commented on; PRs may have been created, merged, commented on, or closed since the last run. Always verify memory against current repository state — reviewing recent activity since your last run is wise before acting on stale assumptions. **Memory backlog tracking**: Records in `issues`, `fixes`, and `priorities` are **action items for you**, not just informational notes. Each run, prioritise applicable work from those fields and remove or replace entries as their state changes. +For clarification/deferred records, include the last reviewed human comment ID in `note` and compare it with live comments before reusing a blocker. A day-only `updated_at` value is not proof that no new reply arrived. ## Workflow @@ -453,6 +584,20 @@ If the selected tasks and their local fallbacks reveal no actionable work, do no - Keep this bounded to the six generic categories, each assessed at most once, within the existing run timeout. If all are exhausted or genuinely blocked, use an honest `noop` with the concrete checks and deferrals, and retain concise latest results in memory's `checks`. - Command mode remains exclusive: do not start this sweep after completing explicit instructions. Honour all output limits, maintainer holds and any maintenance-only/new-PR restrictions; the sweep is not permission to bypass a PR cap or other safety boundary. +### Completion contract + +Labels, label removals, comments, issues, and PR changes are all substantive work. After any such work, do Task 11 and do **not** emit `noop`, even if no code changed. The safe-output gate rejects public changes without a Monthly Activity update and rejects mutation-plus-noop batches. + +If work is genuinely exhausted, the **message string** passed to `noop` must be a JSON object with `outcome: "justified_noop"` and an `assessments` array. Include one entry for every selected task and every generic fallback task; local fallback tasks may also appear, once each. Each entry has `task`, a concise `result`, and a nonempty `evidence` array. Each evidence object has `command` (the exact successful bash command from this session), `path` (the inspected repository-relative path or GitHub API endpoint appearing in that command), and `kind` (`source`, `test`, `docs`, `check`, or `github`). Example of **one entry**, not a complete assessment: + +```json +{"task":9,"result":"Cancellation tests already assert disposal after early termination","evidence":[{"command":"sed -n '1,160p' tests/Tests.fs","path":"tests/Tests.fs","kind":"test"}]} +``` + +The gate compares the assessment with the uploaded task selection and actual completed tool calls. Tasks 4, 5, 7, 8, 9, and 10 require concrete source/test/docs/check evidence, not just issue queries, task-selection files, `ls`, or API-reference counts. This checks recorded assessment coverage, not whether an improvement must exist. Do not manufacture work to satisfy it. + +Missing access, failed memory updates, insufficient time, or skipped required assessments mean **`report_incomplete`**, with the uncompleted tasks and blockers, not `noop` or invented evidence. Command-mode noops, including explicit stop requests, are exempt from the normal-mode assessment contract. Scheduled maintenance-only noops need only Tasks 6, 1, and 2, respecting their restrictions. + The weighting scheme naturally adapts to repo state: - When unlabelled issues pile up, Task 1 (labelling) dominates. @@ -470,6 +615,7 @@ Always do Task 11 (Update Monthly Activity Summary Issue) after performing work Process as many unlabelled issues and PRs as possible each run. Resume from memory's backlog cursor. For each item, apply the best-fitting labels from: `bug`, `enhancement`, `help wanted`, `good first issue`, `documentation`, `question`, `duplicate`, `wontfix`, `spam`, `off topic`, `needs triage`, `needs investigation`, `breaking change`, `performance`, `security`, `refactor`. Remove misapplied labels. Apply multiple where appropriate; skip any you're not confident about. After labelling, post a brief comment if you have something genuinely useful to add. +Read the item's body before choosing labels, and relevant human comments when the classification is ambiguous. Titles alone are not sufficient evidence for a confident label. Update memory with labels applied and cursor position. @@ -477,6 +623,7 @@ Update memory with labels applied and cursor position. 1. List open issues sorted by creation date ascending (oldest first). Resume from your memory's backlog cursor; reset when you reach the end. 2. **Prioritise issues that have never received substantive Repo Assist action.** Read the issue and its comments, inspect the relevant code and tests, and check memory's `issues` and `fixes` records. Expect to act substantively on 1–3 issues per run; you may scan many more to find good candidates. Only re-engage on an already-addressed issue if new human comments or repository changes warrant further action. + Read the latest human reply before declaring an issue blocked or awaiting clarification. A reply choosing one of your proposed options is a specification decision, not a missing answer. Check linked reproductions and regression tests before asking for them again. If a fix still needs design work, record the remaining precise decision and assess another candidate; do not generalize one blocker to the whole backlog. 3. After investigation, choose the outcome that makes the most forward progress: a. **Resolve**: if the issue is already fixed, answered, a duplicate, unsupported, or no longer applicable, provide the evidence and recommend or apply the appropriate resolution and labels. b. **Fix**: for a fixable bug or feature request, implement it and create a tested draft PR. Follow Task 3's requirements for duplicate checks, branching, focused changes, tests, AI disclosure, issue linking, and memory updates. @@ -523,7 +670,9 @@ Check memory for already-submitted ideas; do not re-propose them. Create a fresh ### Task 6: Maintain Repo Assist PRs 1. List all open PRs with the `[repo-assist]` title prefix. -2. For each PR: fix CI failures caused by your changes by pushing updates; resolve merge conflicts. If you've retried multiple times without success, comment and leave for human review. +2. For each PR: query its current **head repository, head ref, and head SHA**, and fetch that exact head before editing. Fix CI failures caused by your changes and resolve merge conflicts. Before queueing an update, inspect the **entire transported delta against that PR head**, not just the file you intended to fix: `git diff --stat HEAD` and `git diff --name-only HEAD`. If a merge brings unrelated base history, start again from the current PR head and keep the correction focused. + Preflight a bundle of the actual range with `git bundle create /tmp/gh-aw/pr-update.bundle ..HEAD` and `wc -c /tmp/gh-aw/pr-update.bundle`; updates must fit the configured 4096 KiB limit. Do not queue an oversized bundle or a workflow-file change that the configured credential cannot publish. Do not increase limits, grant workflow scope, silently strip intended changes, or bypass protected-file review. Record the precise blocker or queue a concise maintainer comment instead. A successful safe-output call means the update is queued, not pushed. + If you've retried multiple times without success, comment and leave for human review. 3. Do not push updates for infrastructure-only failures — comment instead. 4. Update memory.