Skip to content

feat: phase-2 harness — feature map, fixture packs, verdict replay - #2

Merged
aisona-lab merged 1 commit into
mainfrom
harness/phase-2-feature-map-fixtures-replay
Sep 29, 2026
Merged

aisona-lab merged 1 commit into
mainfrom
harness/phase-2-feature-map-fixtures-replay

Conversation

@aisona-lab

Copy link
Copy Markdown
Owner

Summary

Phase 2 harness-only work for lazycoder:

  • docs/FEATURE_MAP.md — maps core / eval / CLI surfaces to tests and fixture packs; linked from README.
  • fixtures/{clean,deny,empty,malicious,boundary}/ — eval case bodies rehomed from config/evals.json. The config file is now a thin fixture_packs + scoring wrapper (loader refuses embedded cases to avoid dual-source drift). Fixtures are bundled in the wheel next to config_defaults.
  • lazycoder replay LOG — recomputes derive_verdict from each decision-log JSONL record with no model call; exit 1 on drift. Documented in README; covered by pytest (also exercised via the existing decision-log replay unit test).
  • README — short sibling section: lazycoder = trailer, agent-action-gate = thesis (no dependency claimed).

Out of scope (unchanged)

Stage 2 corpus, Action fail-on default, rubric/policy taste, LLM prompts/models, SARIF, sandbox.

Test plan

  • uv sync --extra dev && pytest -q (111 passed)
  • ruff check . && black --check . && mypy src
  • Manual lazycoder replay on a fake-client decision log
  • CI green on this PR

Rehome eval cases under fixtures/{clean,deny,empty,malicious,boundary}
with config/evals.json as a thin fixture_packs wrapper (no dual source).
Add `lazycoder replay LOG` to recompute derive_verdict from a decision
log without calling the model. Document coverage in docs/FEATURE_MAP.md
and clarify lazycoder (trailer) vs agent-action-gate (thesis).
@github-actions

Copy link
Copy Markdown

lazycoder review — 💥 ERROR

Downloading anthropic (1.3MiB)
Downloading pydantic-core (2.0MiB)
 Downloaded pydantic-core
 Downloaded anthropic
Installed 16 packages in 37ms
error: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'Your credit balance is too low to access the Anthropic API. Please go to Plans & Billing to upgrade or purchase credits.'}, 'request_id': 'req_011CfXa7Y4HXKLgsLuQ5dnE6'}

0 finding(s), 0 abstention(s), 0 rule error(s) — R1..R17 rubric via lazycoder

@aisona-lab
aisona-lab merged commit c89390c into main Sep 29, 2026
1 of 2 checks passed
@aisona-lab
aisona-lab deleted the harness/phase-2-feature-map-fixtures-replay branch September 29, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant