Report injection-family holdout and provenance limits

[jonathanjasare]

Summary

• Characterize the canonical split accurately as holding out user-task/injection-task combinations rather than whole attack families.
• Add a deterministic secondary stress test that keeps complete injection-task families together.
• Elevate the mixed-provenance finding to the README opening.
• Document the exact normal banking sequence responsible for the false-positive ceiling.
• Remove the track-record phrase “production readiness” while retaining scope limitations.

Why

The canonical (suite, user task, injection task) group key prevents exact combinations from crossing partitions, but every injection-task identity appears in both development and holdout. Its 35.3% recall therefore measures generalisation to unseen combinations, not unseen attacks.

The dominant empirical finding is broader than TW-001: eight of eleven canonical holdout false negatives carry attacker-controlled strings inside transaction records returned by tools classified as internal. Tool-result-level provenance cannot represent mixed-trust records accurately.

Measured impact

The canonical result remains unchanged:

Metric	Result
Recall	6/17 (35.3%)
Precision	6/7 (85.7%)
False-positive rate	1/37 (2.7%)
Confusion counts	TP 6, FP 1, TN 36, FN 11

The secondary family-isolated split deterministically holds out injection_task_7 and injection_task_8 and recalls 6/13 successful attacks, or 46.2% (Wilson 95%: 23.2%–70.9%). This is higher than the canonical result, not lower.

Methodology caveat

The family result is explicitly reported as a post-hoc stress test, not an unbiased unseen-attack estimate. TW-001 was frozen before this analysis, but its original development partition contained examples from every injection family.

False-positive finding

Two of three development false positives are benign none/none.json sessions. The remaining development false positive and the holdout false positive are injected runs whose benchmark goals failed. All four share the legitimate sequence read_file → get_scheduled_transactions → update_scheduled_transaction, establishing an honest specificity ceiling for this rule shape.

Scope

No rule, taxonomy, corpus, label, canonical split, or dependency changed. No additional rule was added.

Verification

• uv run tripwire verify: TP 6, FP 1, TN 36, FN 11
• pytest: 29 passed
• mypy: passed
• ruff check .: passed
• uv build: source distribution and wheel built
• git diff --check: passed

Review

A fresh read-only reviewer approved the family-split methodology and reporting. Its one blocking documentation-accuracy finding was fixed, and the requested regression coverage now asserts the exact benign source entries and common tool sequence across all four false positives.

Review guidance

Focus on whether the family stress test is clearly distinguished from a clean holdout and whether the README gives the provenance finding appropriate prominence without overstating generalisation.

[jonathanjasare]

Verification and review note:

• Head commit: cb91a51 (Report injection-family holdout)
• Canonical measurement reproduced unchanged: TP 6, FP 1, TN 36, FN 11.
• Secondary whole-family stress test: 6/13 recall (46.2%; Wilson 95%: 23.2%–70.9%) on injection_task_7 and injection_task_8.
• 29 tests, mypy, Ruff, package build, and uv run tripwire verify passed.
• Fresh read-only review approved the split methodology. The review-driven false-positive wording fix and regression assertions are included.

No rule, taxonomy, corpus, label, canonical split, or dependency changed.