feat: phase-2 harness — feature map, fixture packs, verdict replay - #2
Merged
Merged
Conversation
Rehome eval cases under fixtures/{clean,deny,empty,malicious,boundary}
with config/evals.json as a thin fixture_packs wrapper (no dual source).
Add `lazycoder replay LOG` to recompute derive_verdict from a decision
log without calling the model. Document coverage in docs/FEATURE_MAP.md
and clarify lazycoder (trailer) vs agent-action-gate (thesis).
lazycoder review — 💥 ERROR0 finding(s), 0 abstention(s), 0 rule error(s) — R1..R17 rubric via lazycoder |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 2 harness-only work for lazycoder:
docs/FEATURE_MAP.md— maps core / eval / CLI surfaces to tests and fixture packs; linked from README.fixtures/{clean,deny,empty,malicious,boundary}/— eval case bodies rehomed fromconfig/evals.json. The config file is now a thinfixture_packs+ scoring wrapper (loader refuses embeddedcasesto avoid dual-source drift). Fixtures are bundled in the wheel next toconfig_defaults.lazycoder replay LOG— recomputesderive_verdictfrom each decision-log JSONL record with no model call; exit 1 on drift. Documented in README; covered by pytest (also exercised via the existing decision-log replay unit test).Out of scope (unchanged)
Stage 2 corpus, Action
fail-ondefault, rubric/policy taste, LLM prompts/models, SARIF, sandbox.Test plan
uv sync --extra dev && pytest -q(111 passed)ruff check . && black --check . && mypy srclazycoder replayon a fake-client decision log