Clarify four V2.1 rubric edge cases

[MagMueller]

Task 185's absence-branch clarification below is superseded by merged #51, which introduces the validated playback contract. Before merging this PR, rebase it to retain #51 and remove the old 185 changes; the proposed 020/084/099 clarifications remain separate.

Three task rubrics could treat timestamp mistakes, incomplete research or honest unavailable-data rows as grounds for a whole-task fabrication penalty. The video task also did not explicitly describe its existing evidenced-absence outcome. This PR clarifies those boundaries in bu2-020, bu2-084, bu2-099 and bu2-185.

• 020: accept a real clock reading at the close of the observation window; reserve global flags for concrete fabricated evidence.
• 084: distinguish disclosed unattempted scopes from claims that each query returned a response. Missing query coverage still loses credit.
• 099: internal todo completion alone is insufficient for a fabrication flag; source verification is still required.
• 185: explicitly permit a sufficiently evidenced no-player outcome in the task. Incomplete searches still fail the search item; independent reporting credit does not imply successful playback.

Six changed files: encrypted dataset and review scenarios, revision hashes/notes, README and existing integrity assertions. Only these four tasks have content changes; only 185 changes agent-facing instructions. All 200 IDs, item IDs, weights, canaries, execution defaults, judge prompt and scoring code remain unchanged. Historical scores are not rewritten.

Validation: seven local artifact-integrity tests pass. GitHub Actions also passes the full test suite, revision validator, Ruff and actionlint; CodeQL and GitGuardian checks pass. A direct comparison against the parent confirms exactly four content changes. Ten encrypted review scenarios cover honest failure and genuine fabrication (34 total); these are review inputs, not claims of measured judge accuracy. Saved-trace judge replays remain pending.

[cubic-dev-ai]

1 issue found across 6 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/test_rubric_revision.py">

<violation number="1" location="tests/test_rubric_revision.py:91">
P3: The new 34-case pin verifies only the aggregate count, not that the ten added scenarios cover the four clarified tasks. All four tasks declare `review_status: pending_saved_trace_review`, so `validate_revision()` permits each to have zero cases, and the count would pass even if the new scenarios were attached to the wrong tasks or missing entirely. Assert at least one review case for each of bu2-020, bu2-084, bu2-099 and bu2-185 so the claimed "ten scenarios cover the four cases" is actually enforced.</violation>
</file>

<sub>Heads up: you’re close to your flex budget. Increase your flex budget so reviews don’t pause.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P3: The new 34-case pin verifies only the aggregate count, not that the ten added scenarios cover the four clarified tasks. All four tasks declare review_status: pending_saved_trace_review, so validate_revision() permits each to have zero cases, and the count would pass even if the new scenarios were attached to the wrong tasks or missing entirely. Assert at least one review case for each of bu2-020, bu2-084, bu2-099 and bu2-185 so the claimed "ten scenarios cover the four cases" is actually enforced.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At tests/test_rubric_revision.py, line 91:

<comment>The new 34-case pin verifies only the aggregate count, not that the ten added scenarios cover the four clarified tasks. All four tasks declare `review_status: pending_saved_trace_review`, so `validate_revision()` permits each to have zero cases, and the count would pass even if the new scenarios were attached to the wrong tasks or missing entirely. Assert at least one review case for each of bu2-020, bu2-084, bu2-099 and bu2-185 so the claimed "ten scenarios cover the four cases" is actually enforced.</comment>

<file context>
@@ -84,7 +88,7 @@ def test_only_declared_contracts_change(self):
             self.assertTrue(changed_fields <= allowed)
             self.assertEqual(after[task_id]["weights"], before[task_id]["weights"])
-        self.assertEqual(len(cases["cases"]), 24)
+        self.assertEqual(len(cases["cases"]), 34)
         self.assertEqual(manifest["status"], "main_not_regraded")
 
</file context>

Suggested change

	self.assertEqual(len(cases["cases"]), 34)
	clarified = {"bu2-020", "bu2-084", "bu2-099", "bu2-185"}
	covered = {case["task_id"] for case in cases["cases"]}
	self.assertTrue(clarified <= covered, f"missing review cases for {clarified - covered}")
	self.assertEqual(len(cases["cases"]), 34)