Add matrix generated-model evidence aggregation

CHANGE_LOG.md

@@ -4,6 +4,16 @@ This file records completed project work in chronological order.
 
 ## 2026-07-02
 
+- Activated Phase 38 on `feature/p38-matrix-generated-model-evidence`, created parent issue #243 and
+  child issues #244 through #248, and scoped the phase around generic compact generated-model
+  evidence aggregation for FreshForge matrix runs.
+- Implemented Phase 38 matrix evidence aggregation by adding compact matrix evidence records,
+  extraction/writer helpers, a `modelwright validation matrix-evidence` CLI command, public exports,
+  and validation-evidence documentation while keeping FABLE semantics and raw artifacts out of
+  Modelwright summaries.
+- Verified the Phase 38 implementation locally with Ruff, focused evidence/CLI/API tests, full pytest
+  (`202` passed and `1` skipped benchmark), warning-clean Sphinx docs, Read the Docs theme
+  verification, release artifact checks, and `git diff --check`.
 - Updated the optional FreshForge integration dependency now that FreshForge is published on PyPI:
   `modelwright[freshforge]`, `dev`, and `test` install `freshforge>=0.1.0a5,<0.2` while the core
   Modelwright package remains FreshForge-free.

ROADMAP.md

@@ -1058,6 +1058,10 @@ Phase 36 is complete on `main`: Modelwright now packages generic compact validat
 summaries for downstream automation without copying raw generated source, generated values,
 workbooks, or full validation reports.
 
+Phase 38 is active on `feature/p38-matrix-generated-model-evidence`: Modelwright will aggregate
+compact generated-model evidence across FreshForge matrix runs so downstream packages can summarize
+repeated validation cases without parsing raw artifacts or encoding domain-specific semantics.
+
 ## Phase 33: FreshForge Provider Pilot For Modelwright Workflows
 
 GitHub parent issue: #205
@@ -1559,3 +1563,51 @@ Verification evidence:
 - PyPI JSON listed `modelwright-0.1.0a7-py3-none-any.whl` and `modelwright-0.1.0a7.tar.gz`.
 - Clean PyPI install verified `modelwright[notebook]==0.1.0a7`, imported `modelwright 0.1.0a7`,
   imported pandas, and ran `modelwright --help`.
+
+## Phase 38: Matrix Generated-Model Evidence Aggregation
+
+GitHub parent issue: #243.
+
+Active branch: `feature/p38-matrix-generated-model-evidence`.
+
+Status: implemented locally; PR pending.
+
+Goal: aggregate compact generated-model evidence across FreshForge matrix runs without rerunning
+FreshForge, Modelwright generation, or validation.
+
+- [x] P38.1 Define matrix evidence aggregation records. Child issue: #244.
+- [x] P38.2 Add aggregation API and compact writers. Child issue: #245.
+- [x] P38.3 Add CLI command for matrix evidence packaging. Child issue: #246.
+- [x] P38.4 Update docs, examples, and tests. Child issue: #247.
+- [ ] P38.5 Verify, PR, deploy docs, and close phase. Child issue: #248.
+
+Dependency note: this phase follows FreshForge `v0.1.0a5` and Modelwright Phase 36. It provides the
+generic matrix evidence backend that FABLE Pyculator Phase 25 should consume.
+
+Acceptance boundary:
+
+- May summarize matrix-level generated-model validation evidence from compact FreshForge/Modelwright
+  artifacts.
+- Must not run FreshForge matrices, generate models, validate workbooks, or infer domain semantics.
+- Must not copy raw generated source, raw generated values, source workbooks, or full validation
+  reports into compact summaries.
+
+Implementation evidence:
+
+- Added `MatrixEvidencePaths`, `MatrixEvidenceCaseSummary`, and `MatrixEvidenceSummary`.
+- Added matrix evidence extraction and writers that read FreshForge matrix JSON and optional per-case
+  generated-model artifacts or compact validation summaries.
+- Added `modelwright validation matrix-evidence` with matrix run/summary input, artifact root,
+  output directory, required-evidence, and JSON options.
+- Updated validation-evidence docs, CLI reference, public exports, and tests.
+
+Local verification:
+
+- `.venv/bin/python -m ruff check .` passed.
+- `.venv/bin/python -m pytest tests/test_evidence.py tests/test_cli.py tests/test_public_api.py -q`
+  passed with 35 tests.
+- `.venv/bin/python -m pytest` passed with 202 tests and 1 skipped benchmark.
+- `.venv/bin/sphinx-build -q -b html docs _build/html -W` passed.
+- `.venv/bin/python scripts/verify_docs_theme.py _build/html` passed.
+- `scripts/check_release_artifacts.sh` passed.
+- `git diff --check` passed.

docs/guides/validation-evidence.rst

@@ -55,6 +55,43 @@ the command should fail instead:
      --artifact-dir tmp/generated-model \
      --require-artifacts
 
+Matrix Evidence
+---------------
+
+FreshForge matrix runs can produce one generated-model workflow per case. Modelwright can aggregate
+those saved matrix records into a compact matrix-level evidence package:
+
+.. code-block:: bash
+
+   modelwright validation matrix-evidence \
+     --evidence-id generated-model-matrix \
+     --matrix-run tmp/matrix-run.json \
+     --artifact-root tmp/generated-model-matrix \
+     --output-dir tmp/validation-evidence/generated-model-matrix \
+     --json
+
+The matrix command is also extraction-only. It does not run FreshForge, generate models, execute
+generated code, or validate workbooks. It reads a saved FreshForge matrix run or matrix summary and
+then looks for per-case generated-model artifacts under ``--artifact-root``. For each case,
+Modelwright checks ``<artifact-root>/<case-id>`` and then ``<artifact-root>/<namespace>`` when the
+FreshForge namespace is relative.
+
+The output is again compact:
+
+- matrix-level ``summary.json`` and ``summary.md``;
+- one sanitized row per matrix case;
+- evidence/equivalence status per case;
+- comparable, match, and mismatch counts when available.
+
+Use ``--require-evidence`` when every matrix case must have generated-model evidence:
+
+.. code-block:: bash
+
+   modelwright validation matrix-evidence \
+     --matrix-summary tmp/matrix-summary.json \
+     --artifact-root tmp/generated-model-matrix \
+     --require-evidence
+
 Status Rules
 ------------
 
@@ -79,8 +116,11 @@ Python API
 .. code-block:: python
 
    from modelwright import (
+       extract_matrix_evidence,
        extract_validation_evidence,
+       matrix_evidence_paths,
        validation_evidence_paths,
+       write_matrix_evidence,
        write_validation_evidence,
    )
 
@@ -92,6 +132,15 @@ Python API
    summary = extract_validation_evidence(paths)
    write_validation_evidence(summary, paths)
 
+   matrix_paths = matrix_evidence_paths(
+       evidence_id="strategy-matrix",
+       matrix_run_path="tmp/matrix-run.json",
+       artifact_root="tmp/generated-model-matrix",
+       output_dir="tmp/validation-evidence/strategy-matrix",
+   )
+   matrix_summary = extract_matrix_evidence(matrix_paths)
+   write_matrix_evidence(matrix_summary, matrix_paths)
+
 Boundary
 --------
 

docs/reference/cli.rst

@@ -81,6 +81,13 @@ Validation Commands
      --output-dir tmp/validation-evidence/generated-model \
      --json
 
+   modelwright validation matrix-evidence \
+     --evidence-id generated-model-matrix \
+     --matrix-run tmp/matrix-run.json \
+     --artifact-root tmp/generated-model-matrix \
+     --output-dir tmp/validation-evidence/generated-model-matrix \
+     --json
+
 The evaluation command executes the generated Python model, then builds cached-workbook and/or
 oracle-backed validation reports when those inputs are supplied. Verbose progress is written to stderr so
 stdout remains valid JSON for redirected reports.
@@ -91,6 +98,10 @@ validation. Missing artifacts are reported as ``skipped`` by default; use ``--re
 make missing evidence fail. See :doc:`../guides/validation-evidence` for the conservative
 ``evidence_status`` and ``equivalence_status`` rules.
 
+The matrix-evidence command aggregates compact generated-model evidence across an existing
+FreshForge matrix run or matrix summary. It does not run the matrix; it only reads saved matrix JSON
+and optional per-case generated-model artifacts or compact evidence summaries.
+
 Conversion Commands
 -------------------
 

planning/phase-38-matrix-generated-model-evidence.md

@@ -0,0 +1,36 @@
+# Phase 38: Matrix Generated-Model Evidence Aggregation
+
+Phase 38 adds generic compact evidence aggregation for FreshForge matrix runs of generated-model
+workflows.
+
+The motivating downstream workflow is FABLE Pyculator output-ref strategy and scenario-bundle
+matrices, but the Modelwright implementation must remain domain-neutral. Modelwright should only
+understand generated-model workflow evidence: case identifiers, run status, compact diagnostics,
+artifact directories, comparison counts, and conservative equivalence status.
+
+## Intended Inputs
+
+- FreshForge matrix run JSON or matrix summary JSON.
+- Optional per-case Modelwright compact validation-evidence directories.
+- Optional artifact root containing per-case generated-model workflow artifacts.
+
+## Intended Outputs
+
+- A compact matrix `summary.json`.
+- A compact matrix `summary.md`.
+- Per-case status rows with evidence status, equivalence status, comparison counts, diagnostic
+  counts, and sanitized artifact references.
+
+## Boundaries
+
+- Do not run FreshForge matrices.
+- Do not rerun Modelwright inference, generation, execution, or validation.
+- Do not add FABLE output-ref strategy, workbook-version, or scenario semantics.
+- Do not copy raw generated source, generated values, source workbook contents, or full validation
+  reports into summaries.
+
+## Downstream Dependency
+
+FABLE Pyculator Phase 25 should consume this generic aggregation layer for a FABLE-facing 2021
+benchmark matrix evidence cookbook. FreshForge Phase 13 should wait until this and other downstream
+work expose concrete generic run/matrix ergonomics gaps.

src/modelwright/__init__.py

@@ -20,10 +20,16 @@
     evaluate_generated_model,
 )
 from modelwright.evidence import (
+    MatrixEvidenceCaseSummary,
+    MatrixEvidencePaths,
+    MatrixEvidenceSummary,
     ValidationEvidencePaths,
     ValidationEvidenceSummary,
+    extract_matrix_evidence,
     extract_validation_evidence,
+    matrix_evidence_paths,
     validation_evidence_paths,
+    write_matrix_evidence,
     write_validation_evidence,
 )
 from modelwright.conversion import (
@@ -141,6 +147,9 @@
     "GenerationDiagnostic",
     "GenerationResult",
     "MISSING_VALUE",
+    "MatrixEvidenceCaseSummary",
+    "MatrixEvidencePaths",
+    "MatrixEvidenceSummary",
     "ModelFacade",
     "NamedRangeRecord",
     "NotebookDependencyError",
@@ -184,9 +193,11 @@
     "evaluate_generated_model",
     "extract_validation_evidence",
     "extract_workbook",
+    "extract_matrix_evidence",
     "generate_python_module",
     "infer_generated_module_contract",
     "inputs_frame",
+    "matrix_evidence_paths",
     "load_validation_scenario",
     "normalize_cell_reference",
     "normalize_reference",
@@ -199,4 +210,5 @@
     "translate_formula_cell",
     "validation_evidence_paths",
     "write_validation_evidence",
+    "write_matrix_evidence",
 ]

src/modelwright/cli.py

@@ -13,8 +13,11 @@
 from modelwright.conversion import BenchmarkRole, build_conversion_plan
 from modelwright.evaluation import evaluate_generated_model
 from modelwright.evidence import (
+    extract_matrix_evidence,
     extract_validation_evidence,
+    matrix_evidence_paths,
     validation_evidence_paths,
+    write_matrix_evidence,
     write_validation_evidence,
 )
 from modelwright.execution import execute_generated_model
@@ -396,6 +399,83 @@ def validation_evidence(
         typer.echo(f"Missing artifacts: {len(missing)}")
 
 
+@validation_app.command("matrix-evidence")
+def validation_matrix_evidence(
+    evidence_id: str = typer.Option(
+        "generated-model-matrix",
+        "--evidence-id",
+        help="Stable identifier for this compact matrix evidence package.",
+    ),
+    matrix_run: Path | None = typer.Option(
+        None,
+        "--matrix-run",
+        exists=True,
+        dir_okay=False,
+        readable=True,
+        help="FreshForge matrix run JSON, such as output from freshforge matrix run --json.",
+    ),
+    matrix_summary: Path | None = typer.Option(
+        None,
+        "--matrix-summary",
+        exists=True,
+        dir_okay=False,
+        readable=True,
+        help="FreshForge matrix summary JSON.",
+    ),
+    artifact_root: Path | None = typer.Option(
+        None,
+        "--artifact-root",
+        help="Optional root containing per-case generated-model artifacts or compact summaries.",
+    ),
+    output_dir: Path | None = typer.Option(
+        None,
+        "--output-dir",
+        help="Directory where compact matrix summary.json and summary.md should be written.",
+    ),
+    require_evidence: bool = typer.Option(
+        False,
+        "--require-evidence",
+        help="Fail when a matrix case lacks generated-model evidence.",
+    ),
+    json_output: bool = typer.Option(
+        False,
+        "--json",
+        help="Emit command result as JSON.",
+    ),
+) -> None:
+    """Package compact validation evidence for a FreshForge matrix run."""
+
+    try:
+        payload = _matrix_evidence_payload(
+            evidence_id=evidence_id,
+            matrix_run=matrix_run,
+            matrix_summary=matrix_summary,
+            artifact_root=artifact_root,
+            output_dir=output_dir,
+            require_evidence=require_evidence,
+        )
+    except (FileNotFoundError, ValueError) as error:
+        error_payload: dict[str, JsonValue] = {"ok": False, "error": str(error)}
+        if json_output:
+            _emit_json(error_payload)
+            raise typer.Exit(1) from error
+        raise typer.BadParameter(str(error)) from error
+
+    if json_output:
+        _emit_json(payload)
+        return
+
+    summary = cast(dict[str, JsonValue], payload["summary"])
+    typer.echo("Modelwright matrix validation evidence")
+    typer.echo(f"Evidence status: {summary['evidence_status']}")
+    typer.echo(f"Equivalence status: {summary['equivalence_status']}")
+    typer.echo(f"Cases: {summary['case_count']}")
+    typer.echo(f"Passing cases: {summary['pass_count']}")
+    typer.echo(f"Failing cases: {summary['fail_count']}")
+    typer.echo(f"Summary JSON: {payload['summary_json_path']}")
+    typer.echo(f"Summary Markdown: {payload['summary_markdown_path']}")
+
+
 @conversion_app.command("plan")
 def conversion_plan(
     workbook: Path = typer.Argument(..., exists=True, dir_okay=False, readable=True, help="Source workbook path."),
@@ -624,6 +704,38 @@ def _validation_evidence_payload(
     }
 
 
+def _matrix_evidence_payload(
+    *,
+    evidence_id: str,
+    matrix_run: Path | None,
+    matrix_summary: Path | None,
+    artifact_root: Path | None,
+    output_dir: Path | None,
+    require_evidence: bool,
+) -> dict[str, JsonValue]:
+    paths = matrix_evidence_paths(
+        evidence_id=evidence_id,
+        matrix_run_path=matrix_run,
+        matrix_summary_path=matrix_summary,
+        artifact_root=artifact_root,
+        output_dir=output_dir,
+    )
+    summary = extract_matrix_evidence(paths, require_evidence=require_evidence)
+    written = write_matrix_evidence(summary, paths)
+    return {
+        "ok": True,
+        "evidence_id": summary.evidence_id,
+        "evidence_status": summary.evidence_status,
+        "equivalence_status": summary.equivalence_status,
+        "summary_json_path": written["summary_json_path"],
+        "summary_markdown_path": written["summary_markdown_path"],
+        "case_count": summary.case_count,
+        "pass_count": summary.pass_count,
+        "fail_count": summary.fail_count,
+        "summary": summary.to_dict(),
+    }
+
+
 def _conversion_plan_payload(
     *,
     workbook: Path,