diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e724cb..47f4d8a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ ### Fixed +- Preserve per-repository smoke results and partial progress after acquisition or scanner operational failures instead of aborting and losing earlier rows. Error cases have null scores/findings; acquisition failures never match a clean expectation. Explicitly document the pinned Reaparr dangling instruction symlink as an expected typed operational rejection, separate from the historical clean root-file example. Expectation matching is not all-scans-completed or accuracy. + - Check simple full/collapsed Markdown reference links and images against document-relative local targets, reporting a used target at its definition line. Respect document-local, first-definition precedence, skip unused definitions and avoid treating supported definitions/link captions as extra prose paths. Code, ambiguous containers/layouts and unsupported Markdown forms retain explicit coverage limits. Stable rule IDs/penalties, runtime dependencies, published v0.24.0 artifacts and frozen research records are unchanged. ## v0.24.0 diff --git a/benchmarks/README.md b/benchmarks/README.md index 39ad93b..c1d4ba2 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -103,11 +103,20 @@ cannot judge, read the [score contract and known limitations](../docs/limitation | `openai/codex` | `d58d0e5` | 2 | B 82 | 3 absent `.rs` path occurrences; 1 context-size warning | | `anomalyco/opencode` | `9f69463` | 18 | A 94 | 1 context-size warning | | `browser-use/browser-use` | `d379a32` | 2 | B 88 | 1 context-size warning | -| `Reaparr/Reaparr` | `d9926d6` | 1 | A 100 | Prohibited `rm -rf` replacement-table example remains clean | +| `Reaparr/Reaparr` | `d9926d6` | Not scanned | No score | Expected whole-tree rejection: a selected instruction symlink has a missing target | | `olup/origan` | `95ac789` | 1 | B 88 | 1 active aggressive Docker cleanup error | | `restsharp/RestSharp` | `64ee129` | 3 | B 84 | Lowercase `agents.md`; 1 context warning and 1 active broad recursive-deletion error | -All nine findings in the recorded run were manually checked against their rule +The earlier Reaparr A 100 was recorded before strict selected-source validation. +The pinned `src/AppHost/ClientApp/AGENTS.md` is a symlink to `../../AGENTS.md` +(Git blob `b7e6491d3a0dda1d656cc3c4320f5c013cdbc02d`), resolving to absent +`src/AGENTS.md`. Whole-tree scanning must reject this input, not skip it and +produce a clean score. The corpus now explicitly expects `FileNotFoundError`, +errno `2`; that is an operational contract, not a clean prediction or an accuracy +label. The root file's prohibited replacement-table example remains a separate +regression case. The immutable earlier releases/reports are unchanged. + +All nine findings in the earlier completed scans were checked against their rule definitions. In particular, the reviewed output contains no `dead-path` finding for API symbols, package imports, documentation URLs, code-fence examples, platform paths, or generic filename conventions. @@ -130,6 +139,24 @@ The script clones but never executes code from the target repositories. It checks out the exact commits in `corpus.json`, scans them, compares stable finding fingerprints with the reviewed expectations, and exits non-zero on a mismatch. Use `--work-dir DIR` to retain the clones for inspection. + +The runner now records each repository as `scanned`, `acquisition_error` or +`scan_error` and continues after expected operational failures. Failed inputs +have null score/finding fields, not A 100 or an empty clean prediction. Only an +explicit, exact reviewed scan-error type/errno can match an expected rejection; +acquisition errors always fail. `all_matched` means maintained expectations +matched, **not** all repositories scanned or are safe. Inspect `summary` and +`all_scans_completed` separately. The present corpus expects five completed scans +and one operational rejection if acquisitions succeed; no independent accuracy +or user adoption is inferred. + +With `--output`, progress is retained after each repository and incomplete runs +remain `all_completed: false` / `all_matched: false`. Unexpected programming +errors still surface; they do not erase earlier saved results. Reports retain +the tool source identity, not just the declared version. These development +runner changes are not retroactive changes to the v0.24.0 package or frozen +research protocols. The [manual smoke workflow](../.github/workflows/real-world-smoke.yml) +can run the same corpus outside a locally blocked network and retains artifacts. Only one pinned commit is fetched for each repository. Stalled Git commands time out after 180 seconds with the failed command identified in the error. diff --git a/benchmarks/corpus.json b/benchmarks/corpus.json index c3ea4ba..f6800e1 100644 --- a/benchmarks/corpus.json +++ b/benchmarks/corpus.json @@ -76,12 +76,11 @@ "url": "https://github.com/Reaparr/Reaparr.git", "commit": "d9926d649ae6dc9d35b4c0faf183aa1d4d7b137b", "expected": { - "files": 1, - "score": 100, - "grade": "A", - "findings": [] + "status": "scan_error", + "error_type": "FileNotFoundError", + "error_errno": 2 }, - "review_note": "The safety section explicitly prohibits rm -rf and repeats it only in the left-hand side of a nearby replacement table; that example is not an active dangerous instruction." + "review_note": "Whole-tree scanning now explicitly rejects the pinned dangling instruction link src/AppHost/ClientApp/AGENTS.md -> ../../AGENTS.md (Git blob b7e6491d3a0dda1d656cc3c4320f5c013cdbc02d), whose src/AGENTS.md target is absent. This is an expected operational rejection, not A 100 or a clean prediction. The earlier isolated root AGENTS.md prohibition/replacement-table interpretation remains a separate regression case; no maintainer confirmation or current adoption is implied." }, { "name": "olup/origan", diff --git a/scripts/run_real_world_benchmark.py b/scripts/run_real_world_benchmark.py index 0cf4491..d0a10c5 100644 --- a/scripts/run_real_world_benchmark.py +++ b/scripts/run_real_world_benchmark.py @@ -11,6 +11,7 @@ from typing import Any from agent_config_score import __version__ +from agent_config_score.provenance import capture_tool_identity from agent_config_score.scanner import analyze @@ -73,21 +74,35 @@ def _fingerprints(findings: list[dict[str, Any]]) -> list[dict[str, Any]]: def _scan(repository: dict[str, Any], destination: Path) -> dict[str, Any]: - _clone(repository, destination) - report = analyze(destination).to_dict() + identity = {key: repository[key] for key in ("name", "url", "commit", "review_note")} + empty = {"files": None, "score": None, "grade": None, "estimated_tokens": None, "findings": None} + try: + _clone(repository, destination) + except (OSError, RuntimeError) as exc: + # Acquisition is never a prediction or an expected clean result. + return {**identity, **empty, "status": "acquisition_error", "error_type": type(exc).__name__, + "error_errno": getattr(exc, "errno", None), "matches_reviewed_expectation": False} expected = repository["expected"] + try: + report = analyze(destination).to_dict() + except OSError as exc: + matches = (expected.get("status") == "scan_error" + and type(exc).__name__ == expected["error_type"] + and exc.errno == expected["error_errno"]) + return {**identity, **empty, "status": "scan_error", "error_type": type(exc).__name__, + "error_errno": exc.errno, "matches_reviewed_expectation": matches} actual_findings = _fingerprints(report["findings"]) - expected_findings = _fingerprints(expected["findings"]) matches = ( - len(report["files"]) == expected["files"] - and report["score"] == expected["score"] - and report["grade"] == expected["grade"] - and actual_findings == expected_findings + expected.get("status", "scanned") == "scanned" + and len(report["files"]) == expected["files"] + and report["score"] == expected["score"] and report["grade"] == expected["grade"] + and actual_findings == _fingerprints(expected["findings"]) ) return { "name": repository["name"], "url": repository["url"], "commit": repository["commit"], + "status": "scanned", "files": len(report["files"]), "score": report["score"], "grade": report["grade"], @@ -98,6 +113,46 @@ def _scan(repository: dict[str, Any], destination: Path) -> dict[str, Any]: } +def _validate_expectations(repositories: list[dict[str, Any]]) -> None: + if not repositories: + raise ValueError("benchmark requires at least one repository") + for repository in repositories: + expected = repository["expected"] + status = expected.get("status", "scanned") + if status == "scan_error": + if (set(expected) != {"status", "error_type", "error_errno"} + or expected["error_type"] not in { + "OSError", "FileNotFoundError", "PermissionError", "NotADirectoryError", "IsADirectoryError"} + or type(expected["error_errno"]) is not int or expected["error_errno"] <= 0): + raise ValueError("invalid explicit scan-error expectation") + elif status != "scanned" or not all(key in expected for key in ("files", "score", "grade", "findings")): + raise ValueError("invalid scan expectation; acquisition errors cannot be expected successes") + + +def _result(repositories: list[dict[str, Any]], requested: int, corpus: Path, tool: dict) -> dict: + complete = len(repositories) == requested + return { + "schema_version": 1, "agent_config_score_version": __version__, "tool": tool, + "corpus": str(corpus), "all_completed": complete, + "all_scans_completed": complete and all(row["status"] == "scanned" for row in repositories), + "all_matched": complete and all(row["matches_reviewed_expectation"] for row in repositories), + "summary": { + "requested": requested, "recorded": len(repositories), + "scanned": sum(row["status"] == "scanned" for row in repositories), + "acquisition_errors": sum(row["status"] == "acquisition_error" for row in repositories), + "scan_errors": sum(row["status"] == "scan_error" for row in repositories), + "matched_expected_scan_errors": sum(row["status"] == "scan_error" and row["matches_reviewed_expectation"] + for row in repositories), + }, + "repositories": repositories, + } + + +def _write_result(path: Path, result: dict) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(result, indent=2) + "\n", encoding="utf-8") + + def _parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--corpus", type=Path, default=DEFAULT_CORPUS) @@ -138,6 +193,7 @@ def main() -> int: selected_repositories = _select_repositories( corpus["repositories"], args.repository ) + _validate_expectations(selected_repositories) except ValueError as exc: raise SystemExit(str(exc)) from exc temporary: tempfile.TemporaryDirectory[str] | None = None @@ -148,28 +204,24 @@ def main() -> int: work_dir = args.work_dir.resolve() work_dir.mkdir(parents=True, exist_ok=True) + repositories = [] + tool = capture_tool_identity().to_dict() try: - repositories = [ - _scan(repository, work_dir / repository["name"].replace("/", "--")) - for repository in selected_repositories - ] + for repository in selected_repositories: + repositories.append(_scan(repository, work_dir / repository["name"].replace("/", "--"))) + result = _result(repositories, len(selected_repositories), args.corpus, tool) + if args.output is not None: + _write_result(args.output, result) finally: if temporary is not None: temporary.cleanup() - result = { - "schema_version": 1, - "agent_config_score_version": __version__, - "corpus": str(args.corpus), - "all_matched": all(row["matches_reviewed_expectation"] for row in repositories), - "repositories": repositories, - } + result = _result(repositories, len(selected_repositories), args.corpus, tool) rendered = json.dumps(result, indent=2) + "\n" if args.output is None: print(rendered, end="") else: - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(rendered, encoding="utf-8") + _write_result(args.output, result) print(f"wrote {args.output}") return 0 if result["all_matched"] else 1 diff --git a/tests/test_benchmark.py b/tests/test_benchmark.py index aabb1eb..194ac06 100644 --- a/tests/test_benchmark.py +++ b/tests/test_benchmark.py @@ -1,11 +1,14 @@ import json import importlib.util +import contextlib +import io from pathlib import Path import re import subprocess import sys import tempfile import unittest +from types import SimpleNamespace from unittest.mock import patch @@ -18,6 +21,107 @@ class BenchmarkContractTests(unittest.TestCase): + def repository(self, expected=None): + return {"name": "owner/repo", "url": "https://github.com/owner/repo.git", + "commit": "0" * 40, "review_note": "Synthetic fixture", + "expected": expected if expected is not None else { + "files": 1, "score": 100, "grade": "A", "findings": []}} + + def report(self): + return SimpleNamespace(to_dict=lambda: { + "files": ["AGENTS.md"], "score": 100, "grade": "A", "estimated_tokens": 8, "findings": []}) + + def test_acquisition_failure_is_never_a_prediction_or_expected_success(self): + with patch.object(BENCHMARK, "_clone", side_effect=RuntimeError("private transport detail")), \ + patch.object(BENCHMARK, "analyze") as analyze: + result = BENCHMARK._scan(self.repository(), Path("unused")) + analyze.assert_not_called() + self.assertEqual(result["status"], "acquisition_error") + self.assertFalse(result["matches_reviewed_expectation"]) + for key in ("files", "score", "grade", "findings", "estimated_tokens"): + self.assertIsNone(result[key]) + self.assertNotIn("private transport detail", json.dumps(result)) + + def test_operational_failure_is_unscored_and_requires_exact_explicit_expectation(self): + expected = {"status": "scan_error", "error_type": "FileNotFoundError", "error_errno": 2} + for declared, error, matches in ( + (None, FileNotFoundError(2, "private path"), False), + (expected, FileNotFoundError(2, "private path"), True), + (expected, PermissionError(13, "private path"), False), + ): + with self.subTest(declared=declared, error=type(error).__name__): + with patch.object(BENCHMARK, "_clone"), patch.object(BENCHMARK, "analyze", side_effect=error): + result = BENCHMARK._scan(self.repository(declared), Path("unused")) + self.assertEqual(result["status"], "scan_error") + self.assertEqual(result["matches_reviewed_expectation"], matches) + self.assertIsNone(result["score"]) + self.assertIsNone(result["findings"]) + self.assertNotIn("private path", json.dumps(result)) + + def test_success_when_rejection_was_expected_remains_a_mismatch(self): + expected = {"status": "scan_error", "error_type": "FileNotFoundError", "error_errno": 2} + with patch.object(BENCHMARK, "_clone"), patch.object(BENCHMARK, "analyze", return_value=self.report()): + result = BENCHMARK._scan(self.repository(expected), Path("unused")) + self.assertEqual(result["status"], "scanned") + self.assertFalse(result["matches_reviewed_expectation"]) + + def test_expected_rejection_is_not_all_scans_completed(self): + row = {"status": "scan_error", "matches_reviewed_expectation": True} + result = BENCHMARK._result([row], 1, Path("corpus.json"), {}) + self.assertTrue(result["all_matched"]) + self.assertTrue(result["all_completed"]) + self.assertFalse(result["all_scans_completed"]) + self.assertEqual(result["summary"]["scanned"], 0) + self.assertEqual(result["summary"]["matched_expected_scan_errors"], 1) + + def test_empty_or_acquisition_success_expectations_are_rejected(self): + for repositories in ([], [self.repository({"status": "acquisition_error"})], + [self.repository({"status": "scan_error", "error_type": "OSError", "error_errno": True})]): + with self.assertRaises(ValueError): + BENCHMARK._validate_expectations(repositories) + + def test_failed_middle_repository_preserves_earlier_and_later_results(self): + with tempfile.TemporaryDirectory() as directory: + corpus, output = Path(directory) / "corpus.json", Path(directory) / "result.json" + repositories = [{**self.repository(), "name": f"owner/repo-{i}"} for i in range(3)] + corpus.write_text(json.dumps({"repositories": repositories}), encoding="utf-8") + snapshots = [] + writer = BENCHMARK._write_result + + def retain(path, result): + snapshots.append((result["summary"]["recorded"], result["all_completed"])) + writer(path, result) + + with patch.object(sys, "argv", [str(RUNNER), "--corpus", str(corpus), "--output", str(output)]), \ + patch.object(BENCHMARK, "_clone", side_effect=[None, RuntimeError("failure"), None]), \ + patch.object(BENCHMARK, "analyze", return_value=self.report()), \ + patch.object(BENCHMARK, "_write_result", side_effect=retain), \ + contextlib.redirect_stdout(io.StringIO()): + self.assertEqual(BENCHMARK.main(), 1) + result = json.loads(output.read_text(encoding="utf-8")) + self.assertEqual([row["status"] for row in result["repositories"]], + ["scanned", "acquisition_error", "scanned"]) + self.assertEqual(snapshots[:3], [(1, False), (2, False), (3, True)]) + self.assertFalse(result["all_matched"]) + self.assertFalse(result["all_scans_completed"]) + self.assertEqual(result["summary"]["scanned"], 2) + self.assertIn("source_fingerprint", result["tool"]) + + def test_unexpected_programming_error_leaves_explicitly_partial_report(self): + with tempfile.TemporaryDirectory() as directory: + corpus, output = Path(directory) / "corpus.json", Path(directory) / "result.json" + repositories = [{**self.repository(), "name": f"owner/repo-{i}"} for i in range(2)] + corpus.write_text(json.dumps({"repositories": repositories}), encoding="utf-8") + with patch.object(sys, "argv", [str(RUNNER), "--corpus", str(corpus), "--output", str(output)]), \ + patch.object(BENCHMARK, "_clone"), \ + patch.object(BENCHMARK, "analyze", side_effect=[self.report(), ValueError("bug")]), \ + self.assertRaisesRegex(ValueError, "bug"): + BENCHMARK.main() + result = json.loads(output.read_text(encoding="utf-8")) + self.assertEqual(result["summary"]["recorded"], 1) + self.assertFalse(result["all_completed"]) + self.assertFalse(result["all_matched"]) + def test_clone_fetches_only_the_pinned_snapshot(self): with tempfile.TemporaryDirectory() as directory: source = Path(directory) / "source" @@ -118,6 +222,11 @@ def test_corpus_uses_unique_pinned_commits_and_reviewed_expectations(self): self.assertTrue(repository["url"].startswith("https://github.com/")) self.assertTrue(repository["review_note"].strip()) expected = repository["expected"] + if expected.get("status") == "scan_error": + self.assertEqual(set(expected), {"status", "error_type", "error_errno"}) + self.assertEqual(expected["error_type"], "FileNotFoundError") + self.assertEqual(expected["error_errno"], 2) + continue self.assertGreater(expected["files"], 0) self.assertIn(expected["grade"], {"A", "B", "C", "D", "F"}) self.assertGreaterEqual(expected["score"], 0)