diff --git a/CHANGE_LOG.md b/CHANGE_LOG.md index 320f125..5643947 100644 --- a/CHANGE_LOG.md +++ b/CHANGE_LOG.md @@ -4,6 +4,16 @@ This file records completed project work in chronological order. ## 2026-07-02 +- Activated Phase 38 on `feature/p38-matrix-generated-model-evidence`, created parent issue #243 and + child issues #244 through #248, and scoped the phase around generic compact generated-model + evidence aggregation for FreshForge matrix runs. +- Implemented Phase 38 matrix evidence aggregation by adding compact matrix evidence records, + extraction/writer helpers, a `modelwright validation matrix-evidence` CLI command, public exports, + and validation-evidence documentation while keeping FABLE semantics and raw artifacts out of + Modelwright summaries. +- Verified the Phase 38 implementation locally with Ruff, focused evidence/CLI/API tests, full pytest + (`202` passed and `1` skipped benchmark), warning-clean Sphinx docs, Read the Docs theme + verification, release artifact checks, and `git diff --check`. - Updated the optional FreshForge integration dependency now that FreshForge is published on PyPI: `modelwright[freshforge]`, `dev`, and `test` install `freshforge>=0.1.0a5,<0.2` while the core Modelwright package remains FreshForge-free. diff --git a/ROADMAP.md b/ROADMAP.md index 90d8baa..b900023 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1058,6 +1058,10 @@ Phase 36 is complete on `main`: Modelwright now packages generic compact validat summaries for downstream automation without copying raw generated source, generated values, workbooks, or full validation reports. +Phase 38 is active on `feature/p38-matrix-generated-model-evidence`: Modelwright will aggregate +compact generated-model evidence across FreshForge matrix runs so downstream packages can summarize +repeated validation cases without parsing raw artifacts or encoding domain-specific semantics. + ## Phase 33: FreshForge Provider Pilot For Modelwright Workflows GitHub parent issue: #205 @@ -1559,3 +1563,51 @@ Verification evidence: - PyPI JSON listed `modelwright-0.1.0a7-py3-none-any.whl` and `modelwright-0.1.0a7.tar.gz`. - Clean PyPI install verified `modelwright[notebook]==0.1.0a7`, imported `modelwright 0.1.0a7`, imported pandas, and ran `modelwright --help`. + +## Phase 38: Matrix Generated-Model Evidence Aggregation + +GitHub parent issue: #243. + +Active branch: `feature/p38-matrix-generated-model-evidence`. + +Status: implemented locally; PR pending. + +Goal: aggregate compact generated-model evidence across FreshForge matrix runs without rerunning +FreshForge, Modelwright generation, or validation. + +- [x] P38.1 Define matrix evidence aggregation records. Child issue: #244. +- [x] P38.2 Add aggregation API and compact writers. Child issue: #245. +- [x] P38.3 Add CLI command for matrix evidence packaging. Child issue: #246. +- [x] P38.4 Update docs, examples, and tests. Child issue: #247. +- [ ] P38.5 Verify, PR, deploy docs, and close phase. Child issue: #248. + +Dependency note: this phase follows FreshForge `v0.1.0a5` and Modelwright Phase 36. It provides the +generic matrix evidence backend that FABLE Pyculator Phase 25 should consume. + +Acceptance boundary: + +- May summarize matrix-level generated-model validation evidence from compact FreshForge/Modelwright + artifacts. +- Must not run FreshForge matrices, generate models, validate workbooks, or infer domain semantics. +- Must not copy raw generated source, raw generated values, source workbooks, or full validation + reports into compact summaries. + +Implementation evidence: + +- Added `MatrixEvidencePaths`, `MatrixEvidenceCaseSummary`, and `MatrixEvidenceSummary`. +- Added matrix evidence extraction and writers that read FreshForge matrix JSON and optional per-case + generated-model artifacts or compact validation summaries. +- Added `modelwright validation matrix-evidence` with matrix run/summary input, artifact root, + output directory, required-evidence, and JSON options. +- Updated validation-evidence docs, CLI reference, public exports, and tests. + +Local verification: + +- `.venv/bin/python -m ruff check .` passed. +- `.venv/bin/python -m pytest tests/test_evidence.py tests/test_cli.py tests/test_public_api.py -q` + passed with 35 tests. +- `.venv/bin/python -m pytest` passed with 202 tests and 1 skipped benchmark. +- `.venv/bin/sphinx-build -q -b html docs _build/html -W` passed. +- `.venv/bin/python scripts/verify_docs_theme.py _build/html` passed. +- `scripts/check_release_artifacts.sh` passed. +- `git diff --check` passed. diff --git a/docs/guides/validation-evidence.rst b/docs/guides/validation-evidence.rst index 2cd5f2a..8e4b53d 100644 --- a/docs/guides/validation-evidence.rst +++ b/docs/guides/validation-evidence.rst @@ -55,6 +55,43 @@ the command should fail instead: --artifact-dir tmp/generated-model \ --require-artifacts +Matrix Evidence +--------------- + +FreshForge matrix runs can produce one generated-model workflow per case. Modelwright can aggregate +those saved matrix records into a compact matrix-level evidence package: + +.. code-block:: bash + + modelwright validation matrix-evidence \ + --evidence-id generated-model-matrix \ + --matrix-run tmp/matrix-run.json \ + --artifact-root tmp/generated-model-matrix \ + --output-dir tmp/validation-evidence/generated-model-matrix \ + --json + +The matrix command is also extraction-only. It does not run FreshForge, generate models, execute +generated code, or validate workbooks. It reads a saved FreshForge matrix run or matrix summary and +then looks for per-case generated-model artifacts under ``--artifact-root``. For each case, +Modelwright checks ``/`` and then ``/`` when the +FreshForge namespace is relative. + +The output is again compact: + +- matrix-level ``summary.json`` and ``summary.md``; +- one sanitized row per matrix case; +- evidence/equivalence status per case; +- comparable, match, and mismatch counts when available. + +Use ``--require-evidence`` when every matrix case must have generated-model evidence: + +.. code-block:: bash + + modelwright validation matrix-evidence \ + --matrix-summary tmp/matrix-summary.json \ + --artifact-root tmp/generated-model-matrix \ + --require-evidence + Status Rules ------------ @@ -79,8 +116,11 @@ Python API .. code-block:: python from modelwright import ( + extract_matrix_evidence, extract_validation_evidence, + matrix_evidence_paths, validation_evidence_paths, + write_matrix_evidence, write_validation_evidence, ) @@ -92,6 +132,15 @@ Python API summary = extract_validation_evidence(paths) write_validation_evidence(summary, paths) + matrix_paths = matrix_evidence_paths( + evidence_id="strategy-matrix", + matrix_run_path="tmp/matrix-run.json", + artifact_root="tmp/generated-model-matrix", + output_dir="tmp/validation-evidence/strategy-matrix", + ) + matrix_summary = extract_matrix_evidence(matrix_paths) + write_matrix_evidence(matrix_summary, matrix_paths) + Boundary -------- diff --git a/docs/reference/cli.rst b/docs/reference/cli.rst index f735b96..7056280 100644 --- a/docs/reference/cli.rst +++ b/docs/reference/cli.rst @@ -81,6 +81,13 @@ Validation Commands --output-dir tmp/validation-evidence/generated-model \ --json + modelwright validation matrix-evidence \ + --evidence-id generated-model-matrix \ + --matrix-run tmp/matrix-run.json \ + --artifact-root tmp/generated-model-matrix \ + --output-dir tmp/validation-evidence/generated-model-matrix \ + --json + The evaluation command executes the generated Python model, then builds cached-workbook and/or oracle-backed validation reports when those inputs are supplied. Verbose progress is written to stderr so stdout remains valid JSON for redirected reports. @@ -91,6 +98,10 @@ validation. Missing artifacts are reported as ``skipped`` by default; use ``--re make missing evidence fail. See :doc:`../guides/validation-evidence` for the conservative ``evidence_status`` and ``equivalence_status`` rules. +The matrix-evidence command aggregates compact generated-model evidence across an existing +FreshForge matrix run or matrix summary. It does not run the matrix; it only reads saved matrix JSON +and optional per-case generated-model artifacts or compact evidence summaries. + Conversion Commands ------------------- diff --git a/planning/phase-38-matrix-generated-model-evidence.md b/planning/phase-38-matrix-generated-model-evidence.md new file mode 100644 index 0000000..47494f0 --- /dev/null +++ b/planning/phase-38-matrix-generated-model-evidence.md @@ -0,0 +1,36 @@ +# Phase 38: Matrix Generated-Model Evidence Aggregation + +Phase 38 adds generic compact evidence aggregation for FreshForge matrix runs of generated-model +workflows. + +The motivating downstream workflow is FABLE Pyculator output-ref strategy and scenario-bundle +matrices, but the Modelwright implementation must remain domain-neutral. Modelwright should only +understand generated-model workflow evidence: case identifiers, run status, compact diagnostics, +artifact directories, comparison counts, and conservative equivalence status. + +## Intended Inputs + +- FreshForge matrix run JSON or matrix summary JSON. +- Optional per-case Modelwright compact validation-evidence directories. +- Optional artifact root containing per-case generated-model workflow artifacts. + +## Intended Outputs + +- A compact matrix `summary.json`. +- A compact matrix `summary.md`. +- Per-case status rows with evidence status, equivalence status, comparison counts, diagnostic + counts, and sanitized artifact references. + +## Boundaries + +- Do not run FreshForge matrices. +- Do not rerun Modelwright inference, generation, execution, or validation. +- Do not add FABLE output-ref strategy, workbook-version, or scenario semantics. +- Do not copy raw generated source, generated values, source workbook contents, or full validation + reports into summaries. + +## Downstream Dependency + +FABLE Pyculator Phase 25 should consume this generic aggregation layer for a FABLE-facing 2021 +benchmark matrix evidence cookbook. FreshForge Phase 13 should wait until this and other downstream +work expose concrete generic run/matrix ergonomics gaps. diff --git a/src/modelwright/__init__.py b/src/modelwright/__init__.py index ff55196..ab3fe85 100644 --- a/src/modelwright/__init__.py +++ b/src/modelwright/__init__.py @@ -20,10 +20,16 @@ evaluate_generated_model, ) from modelwright.evidence import ( + MatrixEvidenceCaseSummary, + MatrixEvidencePaths, + MatrixEvidenceSummary, ValidationEvidencePaths, ValidationEvidenceSummary, + extract_matrix_evidence, extract_validation_evidence, + matrix_evidence_paths, validation_evidence_paths, + write_matrix_evidence, write_validation_evidence, ) from modelwright.conversion import ( @@ -141,6 +147,9 @@ "GenerationDiagnostic", "GenerationResult", "MISSING_VALUE", + "MatrixEvidenceCaseSummary", + "MatrixEvidencePaths", + "MatrixEvidenceSummary", "ModelFacade", "NamedRangeRecord", "NotebookDependencyError", @@ -184,9 +193,11 @@ "evaluate_generated_model", "extract_validation_evidence", "extract_workbook", + "extract_matrix_evidence", "generate_python_module", "infer_generated_module_contract", "inputs_frame", + "matrix_evidence_paths", "load_validation_scenario", "normalize_cell_reference", "normalize_reference", @@ -199,4 +210,5 @@ "translate_formula_cell", "validation_evidence_paths", "write_validation_evidence", + "write_matrix_evidence", ] diff --git a/src/modelwright/cli.py b/src/modelwright/cli.py index 42a6bbf..3350944 100644 --- a/src/modelwright/cli.py +++ b/src/modelwright/cli.py @@ -13,8 +13,11 @@ from modelwright.conversion import BenchmarkRole, build_conversion_plan from modelwright.evaluation import evaluate_generated_model from modelwright.evidence import ( + extract_matrix_evidence, extract_validation_evidence, + matrix_evidence_paths, validation_evidence_paths, + write_matrix_evidence, write_validation_evidence, ) from modelwright.execution import execute_generated_model @@ -396,6 +399,83 @@ def validation_evidence( typer.echo(f"Missing artifacts: {len(missing)}") +@validation_app.command("matrix-evidence") +def validation_matrix_evidence( + evidence_id: str = typer.Option( + "generated-model-matrix", + "--evidence-id", + help="Stable identifier for this compact matrix evidence package.", + ), + matrix_run: Path | None = typer.Option( + None, + "--matrix-run", + exists=True, + dir_okay=False, + readable=True, + help="FreshForge matrix run JSON, such as output from freshforge matrix run --json.", + ), + matrix_summary: Path | None = typer.Option( + None, + "--matrix-summary", + exists=True, + dir_okay=False, + readable=True, + help="FreshForge matrix summary JSON.", + ), + artifact_root: Path | None = typer.Option( + None, + "--artifact-root", + help="Optional root containing per-case generated-model artifacts or compact summaries.", + ), + output_dir: Path | None = typer.Option( + None, + "--output-dir", + help="Directory where compact matrix summary.json and summary.md should be written.", + ), + require_evidence: bool = typer.Option( + False, + "--require-evidence", + help="Fail when a matrix case lacks generated-model evidence.", + ), + json_output: bool = typer.Option( + False, + "--json", + help="Emit command result as JSON.", + ), +) -> None: + """Package compact validation evidence for a FreshForge matrix run.""" + + try: + payload = _matrix_evidence_payload( + evidence_id=evidence_id, + matrix_run=matrix_run, + matrix_summary=matrix_summary, + artifact_root=artifact_root, + output_dir=output_dir, + require_evidence=require_evidence, + ) + except (FileNotFoundError, ValueError) as error: + error_payload: dict[str, JsonValue] = {"ok": False, "error": str(error)} + if json_output: + _emit_json(error_payload) + raise typer.Exit(1) from error + raise typer.BadParameter(str(error)) from error + + if json_output: + _emit_json(payload) + return + + summary = cast(dict[str, JsonValue], payload["summary"]) + typer.echo("Modelwright matrix validation evidence") + typer.echo(f"Evidence status: {summary['evidence_status']}") + typer.echo(f"Equivalence status: {summary['equivalence_status']}") + typer.echo(f"Cases: {summary['case_count']}") + typer.echo(f"Passing cases: {summary['pass_count']}") + typer.echo(f"Failing cases: {summary['fail_count']}") + typer.echo(f"Summary JSON: {payload['summary_json_path']}") + typer.echo(f"Summary Markdown: {payload['summary_markdown_path']}") + + @conversion_app.command("plan") def conversion_plan( workbook: Path = typer.Argument(..., exists=True, dir_okay=False, readable=True, help="Source workbook path."), @@ -624,6 +704,38 @@ def _validation_evidence_payload( } +def _matrix_evidence_payload( + *, + evidence_id: str, + matrix_run: Path | None, + matrix_summary: Path | None, + artifact_root: Path | None, + output_dir: Path | None, + require_evidence: bool, +) -> dict[str, JsonValue]: + paths = matrix_evidence_paths( + evidence_id=evidence_id, + matrix_run_path=matrix_run, + matrix_summary_path=matrix_summary, + artifact_root=artifact_root, + output_dir=output_dir, + ) + summary = extract_matrix_evidence(paths, require_evidence=require_evidence) + written = write_matrix_evidence(summary, paths) + return { + "ok": True, + "evidence_id": summary.evidence_id, + "evidence_status": summary.evidence_status, + "equivalence_status": summary.equivalence_status, + "summary_json_path": written["summary_json_path"], + "summary_markdown_path": written["summary_markdown_path"], + "case_count": summary.case_count, + "pass_count": summary.pass_count, + "fail_count": summary.fail_count, + "summary": summary.to_dict(), + } + + def _conversion_plan_payload( *, workbook: Path, diff --git a/src/modelwright/evidence.py b/src/modelwright/evidence.py index 8c4e6b8..e74abfa 100644 --- a/src/modelwright/evidence.py +++ b/src/modelwright/evidence.py @@ -74,6 +74,111 @@ def to_dict(self) -> dict[str, JsonValue]: } +@dataclass(frozen=True) +class MatrixEvidencePaths: + """Filesystem contract for compact matrix-evidence aggregation.""" + + evidence_id: str + output_dir: Path + matrix_run_path: Path | None = None + matrix_summary_path: Path | None = None + artifact_root: Path | None = None + summary_json_path: Path | None = None + summary_markdown_path: Path | None = None + + def __post_init__(self) -> None: + if self.matrix_run_path is None and self.matrix_summary_path is None: + raise ValueError("matrix_run_path or matrix_summary_path is required") + if self.summary_json_path is None: + object.__setattr__(self, "summary_json_path", self.output_dir / "summary.json") + if self.summary_markdown_path is None: + object.__setattr__(self, "summary_markdown_path", self.output_dir / "summary.md") + + +@dataclass(frozen=True) +class MatrixEvidenceCaseSummary: + """Sanitized generated-model evidence summary for one matrix case.""" + + case_id: str + namespace: str | None + run_status: str | None + evidence_status: EvidenceStatus + equivalence_status: EquivalenceStatus + comparable_output_count: int | None = None + match_count: int | None = None + mismatch_count: int | None = None + diagnostic_count: int = 0 + error_count: int = 0 + warning_count: int = 0 + artifact_dir: str | None = None + evidence_source: str = "missing" + notes: tuple[str, ...] = field(default_factory=tuple) + + def to_dict(self) -> dict[str, JsonValue]: + """Serialize the compact matrix case summary.""" + + return { + "case_id": self.case_id, + "namespace": self.namespace, + "run_status": self.run_status, + "evidence_status": self.evidence_status, + "equivalence_status": self.equivalence_status, + "comparable_output_count": self.comparable_output_count, + "match_count": self.match_count, + "mismatch_count": self.mismatch_count, + "diagnostic_count": self.diagnostic_count, + "error_count": self.error_count, + "warning_count": self.warning_count, + "artifact_dir": self.artifact_dir, + "evidence_source": self.evidence_source, + "notes": list(self.notes), + } + + +@dataclass(frozen=True) +class MatrixEvidenceSummary: + """Sanitized generated-model evidence summary for a FreshForge matrix.""" + + evidence_id: str + matrix_id: str | None + evidence_status: EvidenceStatus + equivalence_status: EquivalenceStatus + case_count: int + complete_count: int + incomplete_count: int + skipped_count: int + pass_count: int + fail_count: int + diagnostic_count: int = 0 + error_count: int = 0 + warning_count: int = 0 + artifacts: dict[str, str] = field(default_factory=dict) + cases: tuple[MatrixEvidenceCaseSummary, ...] = field(default_factory=tuple) + notes: tuple[str, ...] = field(default_factory=tuple) + + def to_dict(self) -> dict[str, JsonValue]: + """Serialize the compact matrix evidence summary.""" + + return { + "evidence_id": self.evidence_id, + "matrix_id": self.matrix_id, + "evidence_status": self.evidence_status, + "equivalence_status": self.equivalence_status, + "case_count": self.case_count, + "complete_count": self.complete_count, + "incomplete_count": self.incomplete_count, + "skipped_count": self.skipped_count, + "pass_count": self.pass_count, + "fail_count": self.fail_count, + "diagnostic_count": self.diagnostic_count, + "error_count": self.error_count, + "warning_count": self.warning_count, + "artifacts": self.artifacts, + "cases": [case.to_dict() for case in self.cases], + "notes": list(self.notes), + } + + def validation_evidence_paths( *, evidence_id: str = "generated-model", @@ -102,6 +207,26 @@ def validation_evidence_paths( ) +def matrix_evidence_paths( + *, + evidence_id: str = "generated-model-matrix", + output_dir: str | Path | None = None, + matrix_run_path: str | Path | None = None, + matrix_summary_path: str | Path | None = None, + artifact_root: str | Path | None = None, +) -> MatrixEvidencePaths: + """Build the generic path contract for matrix evidence aggregation.""" + + output_root = Path(output_dir) if output_dir is not None else Path("tmp/validation-evidence") / evidence_id + return MatrixEvidencePaths( + evidence_id=evidence_id, + output_dir=output_root, + matrix_run_path=Path(matrix_run_path) if matrix_run_path is not None else None, + matrix_summary_path=Path(matrix_summary_path) if matrix_summary_path is not None else None, + artifact_root=Path(artifact_root) if artifact_root is not None else None, + ) + + def extract_validation_evidence( paths: ValidationEvidencePaths | None = None, *, @@ -163,6 +288,62 @@ def extract_validation_evidence( ) +def extract_matrix_evidence( + paths: MatrixEvidencePaths | None = None, + *, + evidence_id: str = "generated-model-matrix", + matrix_run_path: str | Path | None = None, + matrix_summary_path: str | Path | None = None, + artifact_root: str | Path | None = None, + output_dir: str | Path | None = None, + require_evidence: bool = False, +) -> MatrixEvidenceSummary: + """Aggregate compact evidence from an existing FreshForge matrix run or summary.""" + + evidence_paths = paths or matrix_evidence_paths( + evidence_id=evidence_id, + matrix_run_path=matrix_run_path, + matrix_summary_path=matrix_summary_path, + artifact_root=artifact_root, + output_dir=output_dir, + ) + payload = _load_matrix_payload(evidence_paths) + matrix = _matrix_summary_object(payload) + cases = tuple( + _matrix_case_summary( + evidence_paths=evidence_paths, + case=_object(case), + require_evidence=require_evidence, + ) + for case in _sequence(matrix.get("cases")) + ) + if not cases and require_evidence: + raise FileNotFoundError("matrix evidence contains no cases") + + evidence_status = _matrix_evidence_status(cases) + equivalence_status = _matrix_equivalence_status(cases) + artifacts = _matrix_artifact_summary(evidence_paths) + notes = _matrix_notes(cases) + return MatrixEvidenceSummary( + evidence_id=evidence_paths.evidence_id, + matrix_id=_string(matrix.get("matrix_id")) or _string(_object(payload.get("run")).get("matrix_id")), + evidence_status=evidence_status, + equivalence_status=equivalence_status, + case_count=len(cases), + complete_count=sum(1 for case in cases if case.evidence_status == "complete"), + incomplete_count=sum(1 for case in cases if case.evidence_status == "incomplete"), + skipped_count=sum(1 for case in cases if case.evidence_status == "skipped"), + pass_count=sum(1 for case in cases if case.equivalence_status == "pass"), + fail_count=sum(1 for case in cases if case.equivalence_status == "fail"), + diagnostic_count=sum(case.diagnostic_count for case in cases), + error_count=sum(case.error_count for case in cases), + warning_count=sum(case.warning_count for case in cases), + artifacts=artifacts, + cases=cases, + notes=notes, + ) + + def write_validation_evidence( summary: ValidationEvidenceSummary, paths: ValidationEvidencePaths | None = None, @@ -190,6 +371,33 @@ def write_validation_evidence( } +def write_matrix_evidence( + summary: MatrixEvidenceSummary, + paths: MatrixEvidencePaths | None = None, + *, + output_dir: str | Path | None = None, +) -> dict[str, JsonValue]: + """Write compact matrix evidence ``summary.json`` and ``summary.md``.""" + + if paths is not None: + json_path = paths.summary_json_path or paths.output_dir / "summary.json" + markdown_path = paths.summary_markdown_path or paths.output_dir / "summary.md" + else: + output_root = Path(output_dir) if output_dir is not None else Path("tmp/validation-evidence") / summary.evidence_id + json_path = output_root / "summary.json" + markdown_path = output_root / "summary.md" + + json_path.parent.mkdir(parents=True, exist_ok=True) + payload = summary.to_dict() + json_path.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8") + markdown_path.write_text(_matrix_summary_markdown(summary), encoding="utf-8") + return { + "summary": payload, + "summary_json_path": str(json_path), + "summary_markdown_path": str(markdown_path), + } + + def _artifact_summary(paths: ValidationEvidencePaths) -> dict[str, str]: return { "artifact_dir": str(paths.artifact_dir), @@ -382,6 +590,228 @@ def _summary_markdown(summary: ValidationEvidenceSummary) -> str: return "\n".join(lines) +def _load_matrix_payload(paths: MatrixEvidencePaths) -> dict[str, JsonValue]: + if paths.matrix_run_path is not None: + return _load_json_object(paths.matrix_run_path) + if paths.matrix_summary_path is not None: + return _load_json_object(paths.matrix_summary_path) + raise ValueError("matrix_run_path or matrix_summary_path is required") + + +def _matrix_summary_object(payload: dict[str, JsonValue]) -> dict[str, JsonValue]: + summary = _object(payload.get("summary")) + if summary: + return summary + if "cases" in payload: + return payload + run = _object(payload.get("run")) + if run: + return run + return {} + + +def _matrix_case_summary( + *, + evidence_paths: MatrixEvidencePaths, + case: dict[str, JsonValue], + require_evidence: bool, +) -> MatrixEvidenceCaseSummary: + case_id = _case_id(case) + namespace = _case_namespace(case) + run_status = _string(case.get("status")) or _string(_object(case.get("summary")).get("status")) + diagnostic_count = _int(case.get("diagnostic_count")) or _diagnostic_count(case) + error_count = _int(case.get("error_count")) or _diagnostic_count(case, severity="error") + warning_count = _int(case.get("warning_count")) or _diagnostic_count(case, severity="warning") + + evidence = _case_validation_evidence( + evidence_paths=evidence_paths, + case_id=case_id, + namespace=namespace, + require_evidence=require_evidence, + ) + if evidence is None: + if require_evidence: + raise FileNotFoundError(f"missing matrix case evidence for {case_id}") + return MatrixEvidenceCaseSummary( + case_id=case_id, + namespace=namespace, + run_status=run_status, + evidence_status="skipped", + equivalence_status="incomplete", + diagnostic_count=diagnostic_count, + error_count=error_count, + warning_count=warning_count, + notes=("No per-case evidence directory was found.",), + ) + + comparison = evidence.comparison + return MatrixEvidenceCaseSummary( + case_id=case_id, + namespace=namespace, + run_status=run_status, + evidence_status=evidence.evidence_status, + equivalence_status=evidence.equivalence_status, + comparable_output_count=_int(comparison.get("comparable_output_count")), + match_count=_int(comparison.get("match_count")), + mismatch_count=_int(comparison.get("mismatch_count")), + diagnostic_count=diagnostic_count + _stage_diagnostic_count(evidence.stages), + error_count=error_count + _stage_diagnostic_count(evidence.stages, key="error_diagnostic_count"), + warning_count=warning_count, + artifact_dir=evidence.artifacts.get("artifact_dir"), + evidence_source="validation-summary" if _loaded_from_summary(evidence) else "artifact-extraction", + notes=evidence.notes, + ) + + +def _case_validation_evidence( + *, + evidence_paths: MatrixEvidencePaths, + case_id: str, + namespace: str | None, + require_evidence: bool, +) -> ValidationEvidenceSummary | None: + if evidence_paths.artifact_root is None: + return None + for artifact_dir in _candidate_case_dirs(evidence_paths.artifact_root, case_id, namespace): + summary_path = artifact_dir / "summary.json" + if summary_path.exists(): + return _validation_summary_from_dict(_load_json_object(summary_path)) + if artifact_dir.exists(): + paths = validation_evidence_paths( + evidence_id=f"{evidence_paths.evidence_id}:{case_id}", + artifact_dir=artifact_dir, + output_dir=artifact_dir / "validation-evidence", + ) + return extract_validation_evidence(paths, require_artifacts=require_evidence) + return None + + +def _candidate_case_dirs(artifact_root: Path, case_id: str, namespace: str | None) -> tuple[Path, ...]: + candidates = [artifact_root / case_id] + if namespace: + namespace_path = Path(namespace) + if not namespace_path.is_absolute() and ".." not in namespace_path.parts: + candidates.append(artifact_root / namespace_path) + unique: list[Path] = [] + for candidate in candidates: + if candidate not in unique: + unique.append(candidate) + return tuple(unique) + + +def _validation_summary_from_dict(data: dict[str, JsonValue]) -> ValidationEvidenceSummary: + return ValidationEvidenceSummary( + evidence_id=_string(data.get("evidence_id")) or "generated-model", + evidence_status=_evidence_status(data.get("evidence_status")), + equivalence_status=_equivalence_value(data.get("equivalence_status")), + missing_artifacts=tuple(str(item) for item in _sequence(data.get("missing_artifacts"))), + artifacts={str(key): str(value) for key, value in _object(data.get("artifacts")).items()}, + stages=_object(data.get("stages")), + comparison=_object(data.get("comparison")), + notes=tuple(str(item) for item in _sequence(data.get("notes"))), + ) + + +def _loaded_from_summary(summary: ValidationEvidenceSummary) -> bool: + return "summary_json" in summary.artifacts or "summary_markdown" in summary.artifacts + + +def _case_id(case: dict[str, JsonValue]) -> str: + case_id = _string(case.get("case_id")) or _string(case.get("id")) + if case_id: + return case_id + namespace = _case_namespace(case) + if namespace: + return namespace.strip("/").split("/")[-1] + workflow_id = _string(case.get("workflow_id")) + return workflow_id or "unknown-case" + + +def _case_namespace(case: dict[str, JsonValue]) -> str | None: + return _string(case.get("namespace")) or _string(case.get("run_namespace")) + + +def _matrix_evidence_status(cases: tuple[MatrixEvidenceCaseSummary, ...]) -> EvidenceStatus: + if not cases or all(case.evidence_status == "skipped" for case in cases): + return "skipped" + if all(case.evidence_status == "complete" for case in cases): + return "complete" + return "incomplete" + + +def _matrix_equivalence_status(cases: tuple[MatrixEvidenceCaseSummary, ...]) -> EquivalenceStatus: + if not cases: + return "incomplete" + if any(case.equivalence_status == "fail" for case in cases): + return "fail" + if all(case.equivalence_status == "pass" for case in cases): + return "pass" + return "incomplete" + + +def _matrix_artifact_summary(paths: MatrixEvidencePaths) -> dict[str, str]: + artifacts = { + "output_dir": str(paths.output_dir), + "summary_json": str(paths.summary_json_path or paths.output_dir / "summary.json"), + "summary_markdown": str(paths.summary_markdown_path or paths.output_dir / "summary.md"), + } + if paths.matrix_run_path is not None: + artifacts["matrix_run"] = str(paths.matrix_run_path) + if paths.matrix_summary_path is not None: + artifacts["matrix_summary"] = str(paths.matrix_summary_path) + if paths.artifact_root is not None: + artifacts["artifact_root"] = str(paths.artifact_root) + return artifacts + + +def _matrix_notes(cases: tuple[MatrixEvidenceCaseSummary, ...]) -> tuple[str, ...]: + notes: list[str] = [] + if not cases: + notes.append("Matrix evidence contained no cases.") + if any(case.evidence_status == "skipped" for case in cases): + notes.append("One or more matrix cases had no available generated-model evidence.") + if any(case.evidence_status == "incomplete" for case in cases): + notes.append("One or more matrix cases lacked explicit comparable/match/mismatch counts.") + if any(case.equivalence_status == "fail" for case in cases): + notes.append("One or more matrix cases reported explicit validation mismatches.") + return tuple(notes) + + +def _matrix_summary_markdown(summary: MatrixEvidenceSummary) -> str: + lines = [ + f"# Modelwright Matrix Evidence: {summary.evidence_id}", + "", + f"- Matrix id: `{summary.matrix_id}`", + f"- Evidence status: `{summary.evidence_status}`", + f"- Equivalence status: `{summary.equivalence_status}`", + f"- Cases: `{summary.case_count}`", + f"- Complete cases: `{summary.complete_count}`", + f"- Passing cases: `{summary.pass_count}`", + f"- Failing cases: `{summary.fail_count}`", + "", + "## Cases", + "", + "| Case | Namespace | Evidence | Equivalence | Comparable | Matches | Mismatches |", + "| --- | --- | --- | --- | ---: | ---: | ---: |", + ] + for case in summary.cases: + lines.append( + "| " + f"`{case.case_id}` | " + f"`{case.namespace}` | " + f"`{case.evidence_status}` | " + f"`{case.equivalence_status}` | " + f"`{case.comparable_output_count}` | " + f"`{case.match_count}` | " + f"`{case.mismatch_count}` |" + ) + if summary.notes: + lines.extend(["", "## Notes", ""]) + lines.extend(f"- {note}" for note in summary.notes) + lines.append("") + return "\n".join(lines) + + def _load_json_object(path: Path) -> dict[str, JsonValue]: data = json.loads(path.read_text(encoding="utf-8")) if not isinstance(data, dict): @@ -403,6 +833,30 @@ def _length(value: JsonValue) -> int | None: return None +def _string(value: JsonValue) -> str | None: + return value if isinstance(value, str) and value else None + + +def _int(value: JsonValue) -> int | None: + return value if isinstance(value, int) and not isinstance(value, bool) else None + + +def _evidence_status(value: JsonValue) -> EvidenceStatus: + return value if value in {"skipped", "incomplete", "complete"} else "incomplete" # type: ignore[return-value] + + +def _equivalence_value(value: JsonValue) -> EquivalenceStatus: + return value if value in {"pass", "fail", "incomplete"} else "incomplete" # type: ignore[return-value] + + +def _stage_diagnostic_count(stages: dict[str, JsonValue], *, key: str = "diagnostic_count") -> int: + return sum( + value + for stage in stages.values() + if isinstance(stage, dict) and isinstance((value := stage.get(key)), int) and not isinstance(value, bool) + ) + + def _diagnostic_count(data: dict[str, JsonValue], *, severity: str | None = None) -> int: diagnostics = _sequence(data.get("diagnostics")) if severity is None: diff --git a/tests/test_cli.py b/tests/test_cli.py index 314ecef..053eff1 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -494,6 +494,73 @@ def test_validation_evidence_uses_custom_artifact_output_and_scenario_paths(tmp_ assert json.loads((output_dir / "summary.json").read_text(encoding="utf-8"))["evidence_id"] == "synthetic" +def test_validation_matrix_evidence_help_is_available() -> None: + result = runner.invoke(app, ["validation", "matrix-evidence", "--help"]) + + assert result.exit_code == 0 + assert "Package compact validation evidence for a FreshForge matrix run" in result.stdout + + +def test_validation_matrix_evidence_json_output(tmp_path: Path) -> None: + matrix_run = tmp_path / "matrix-run.json" + artifact_root = tmp_path / "artifacts" + output_dir = tmp_path / "matrix-evidence" + case_dir = artifact_root / "strategy" / "output-columns" + _write_matrix_run(matrix_run, case_ids=("output-columns",)) + _write_validation_evidence_artifacts(case_dir, case_dir / "validation-scenario.json") + + result = runner.invoke( + app, + [ + "validation", + "matrix-evidence", + "--evidence-id", + "strategy-matrix", + "--matrix-run", + str(matrix_run), + "--artifact-root", + str(artifact_root), + "--output-dir", + str(output_dir), + "--json", + ], + ) + + assert result.exit_code == 0 + payload = json.loads(result.stdout) + assert payload["ok"] is True + assert payload["evidence_status"] == "complete" + assert payload["equivalence_status"] == "pass" + assert payload["case_count"] == 1 + assert payload["summary"]["cases"][0]["case_id"] == "output-columns" + assert (output_dir / "summary.json").exists() + assert (output_dir / "summary.md").exists() + + +def test_validation_matrix_evidence_require_evidence_fails(tmp_path: Path) -> None: + matrix_run = tmp_path / "matrix-run.json" + _write_matrix_run(matrix_run, case_ids=("output-columns",)) + + result = runner.invoke( + app, + [ + "validation", + "matrix-evidence", + "--matrix-run", + str(matrix_run), + "--artifact-root", + str(tmp_path / "missing"), + "--require-evidence", + "--json", + ], + ) + + assert result.exit_code != 0 + payload = json.loads(result.stdout) + assert payload["ok"] is False + assert "missing matrix case evidence" in payload["error"] + + def test_conversion_plan_command_outputs_plan_json(tmp_path: Path) -> None: workbook_path = build_workbook(tmp_path / "synthetic_model.xlsx") @@ -651,3 +718,68 @@ def _write_validation_evidence_artifacts(artifact_dir: Path, scenario_path: Path "diagnostics": [], }, ) + + +def _write_matrix_run(path: Path, *, case_ids: tuple[str, ...]) -> None: + _write_json( + path, + { + "ok": True, + "run": { + "matrix_id": "strategy", + "status": "success", + "cases": [ + { + "case_id": case_id, + "namespace": f"strategy/{case_id}", + "run": { + "workflow_id": f"workflow-{case_id}", + "run_namespace": f"strategy/{case_id}", + "status": "success", + "nodes": [], + "diagnostics": [], + }, + "summary": { + "workflow_id": f"workflow-{case_id}", + "run_namespace": f"strategy/{case_id}", + "status": "success", + "node_count": 0, + "diagnostic_count": 0, + "error_count": 0, + "warning_count": 0, + "artifact_count": 0, + "nodes": [], + }, + "diagnostics": [], + } + for case_id in case_ids + ], + "diagnostics": [], + }, + "summary": { + "matrix_id": "strategy", + "status": "success", + "case_count": len(case_ids), + "succeeded_count": len(case_ids), + "failed_count": 0, + "skipped_count": 0, + "diagnostic_count": 0, + "error_count": 0, + "warning_count": 0, + "cases": [ + { + "workflow_id": f"workflow-{case_id}", + "run_namespace": f"strategy/{case_id}", + "status": "success", + "node_count": 0, + "diagnostic_count": 0, + "error_count": 0, + "warning_count": 0, + "artifact_count": 0, + "nodes": [], + } + for case_id in case_ids + ], + }, + }, + ) diff --git a/tests/test_evidence.py b/tests/test_evidence.py index db95e89..534808b 100644 --- a/tests/test_evidence.py +++ b/tests/test_evidence.py @@ -4,8 +4,11 @@ import pytest from modelwright.evidence import ( + extract_matrix_evidence, extract_validation_evidence, + matrix_evidence_paths, validation_evidence_paths, + write_matrix_evidence, write_validation_evidence, ) @@ -105,6 +108,86 @@ def test_write_validation_evidence_writes_stable_json_and_markdown(tmp_path: Pat assert "Equivalence status: `pass`" in markdown +def test_matrix_evidence_aggregates_pass_and_fail_cases(tmp_path: Path) -> None: + matrix_run = tmp_path / "matrix-run.json" + artifact_root = tmp_path / "artifacts" + output_dir = tmp_path / "matrix-evidence" + _write_matrix_run(matrix_run, cases=("output-columns", "headline-only")) + _write_artifacts( + artifact_root / "strategy" / "output-columns", + comparison={"comparable_output_count": 2, "match_count": 2, "mismatch_count": 0}, + ) + _write_artifacts( + artifact_root / "strategy" / "headline-only", + comparison={"comparable_output_count": 2, "match_count": 1, "mismatch_count": 1}, + ) + paths = matrix_evidence_paths( + evidence_id="strategy-matrix", + matrix_run_path=matrix_run, + artifact_root=artifact_root, + output_dir=output_dir, + ) + + summary = extract_matrix_evidence(paths, require_evidence=True) + + assert summary.evidence_status == "complete" + assert summary.equivalence_status == "fail" + assert summary.case_count == 2 + assert summary.pass_count == 1 + assert summary.fail_count == 1 + assert [case.case_id for case in summary.cases] == ["output-columns", "headline-only"] + assert summary.cases[0].comparable_output_count == 2 + assert summary.cases[1].mismatch_count == 1 + + +def test_matrix_evidence_marks_missing_case_evidence_skipped(tmp_path: Path) -> None: + matrix_run = tmp_path / "matrix-run.json" + _write_matrix_run(matrix_run, cases=("output-columns",)) + + summary = extract_matrix_evidence(matrix_run_path=matrix_run, artifact_root=tmp_path / "missing") + + assert summary.evidence_status == "skipped" + assert summary.equivalence_status == "incomplete" + assert summary.cases[0].evidence_status == "skipped" + assert "No per-case evidence" in summary.cases[0].notes[0] + + +def test_matrix_evidence_missing_case_can_be_required(tmp_path: Path) -> None: + matrix_run = tmp_path / "matrix-run.json" + _write_matrix_run(matrix_run, cases=("output-columns",)) + + with pytest.raises(FileNotFoundError, match="missing matrix case evidence"): + extract_matrix_evidence(matrix_run_path=matrix_run, artifact_root=tmp_path / "missing", require_evidence=True) + + +def test_write_matrix_evidence_writes_sanitized_json_and_markdown(tmp_path: Path) -> None: + matrix_run = tmp_path / "matrix-run.json" + artifact_root = tmp_path / "artifacts" + paths = matrix_evidence_paths( + evidence_id="strategy-matrix", + matrix_run_path=matrix_run, + artifact_root=artifact_root, + output_dir=tmp_path / "matrix-evidence", + ) + _write_matrix_run(matrix_run, cases=("output-columns",)) + _write_artifacts( + artifact_root / "strategy" / "output-columns", + comparison_rows=[{"cell_ref": "Summary!B2", "matches": True, "generated": "generated secret"}], + ) + + summary = extract_matrix_evidence(paths) + written = write_matrix_evidence(summary, paths) + + payload = Path(str(written["summary_json_path"])).read_text(encoding="utf-8") + markdown = Path(str(written["summary_markdown_path"])).read_text(encoding="utf-8") + assert "strategy-matrix" in payload + assert "# Modelwright Matrix Evidence: strategy-matrix" in markdown + assert "source_code" not in payload + assert "output_values" not in payload + assert "Summary!B2" not in payload + assert "generated secret" not in payload + + def _write_artifacts( artifact_dir: Path, *, @@ -187,3 +270,72 @@ def _write_artifacts( def _write_json(path: Path, payload: object) -> None: path.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8") + + +def _write_matrix_run(path: Path, *, cases: tuple[str, ...]) -> None: + path.write_text( + json.dumps( + { + "ok": True, + "run": { + "matrix_id": "strategy", + "status": "success", + "cases": [ + { + "case_id": case_id, + "namespace": f"strategy/{case_id}", + "run": { + "workflow_id": f"workflow-{case_id}", + "run_namespace": f"strategy/{case_id}", + "status": "success", + "nodes": [], + "diagnostics": [], + }, + "summary": { + "workflow_id": f"workflow-{case_id}", + "run_namespace": f"strategy/{case_id}", + "status": "success", + "node_count": 0, + "diagnostic_count": 0, + "error_count": 0, + "warning_count": 0, + "artifact_count": 0, + "nodes": [], + }, + "diagnostics": [], + } + for case_id in cases + ], + "diagnostics": [], + }, + "summary": { + "matrix_id": "strategy", + "status": "success", + "case_count": len(cases), + "succeeded_count": len(cases), + "failed_count": 0, + "skipped_count": 0, + "diagnostic_count": 0, + "error_count": 0, + "warning_count": 0, + "cases": [ + { + "workflow_id": f"workflow-{case_id}", + "run_namespace": f"strategy/{case_id}", + "status": "success", + "node_count": 0, + "diagnostic_count": 0, + "error_count": 0, + "warning_count": 0, + "artifact_count": 0, + "nodes": [], + } + for case_id in cases + ], + }, + }, + indent=2, + sort_keys=True, + ), + encoding="utf-8", + ) diff --git a/tests/test_public_api.py b/tests/test_public_api.py index 00916c0..c2bfe69 100644 --- a/tests/test_public_api.py +++ b/tests/test_public_api.py @@ -19,6 +19,12 @@ def test_root_facade_exports_primary_entrypoints() -> None: assert "write_validation_evidence" in modelwright.__all__ assert "ValidationEvidencePaths" in modelwright.__all__ assert "ValidationEvidenceSummary" in modelwright.__all__ + assert "matrix_evidence_paths" in modelwright.__all__ + assert "extract_matrix_evidence" in modelwright.__all__ + assert "write_matrix_evidence" in modelwright.__all__ + assert "MatrixEvidencePaths" in modelwright.__all__ + assert "MatrixEvidenceSummary" in modelwright.__all__ + assert "MatrixEvidenceCaseSummary" in modelwright.__all__ assert "infer_generated_module_contract" in modelwright.__all__ assert "GeneratedContractInferenceResult" in modelwright.__all__ assert "ModelFacade" in modelwright.__all__