feat(trace): add investigation submission endpoint and The Debrief challenge - #573
Open
stealthwhizz wants to merge 8 commits into
Open
Conversation
Scores a student's forensic investigation submission (server, tool, directive) against the expected answer in detector_config, with independent partial credit per field (33/33/34 by default). Server and tool match exactly; the free-text directive uses substring matching by default, or exact when directive_match is set. Scoring is deterministic and never calls an LLM, so it behaves identically under any model backend. Evidence records the score and which fields were correct but never the expected answer, so completion evidence cannot spoil the challenge. 19 unit tests cover the 33/66/100 thresholds, zero-score submissions, extra fields that must not inflate the score, directive fuzzy/exact modes, normalization, the tool_arguments fallback, and config validation.
…allenge
Wire the PurpleTeamDetector into a playable forensic challenge:
- POST /ctf/api/v1/toolkit/investigation accepts a {server, tool,
directive} answer and emits business.investigation.submitted with a
fresh workflow_id per submission, so each attempt is tracked.
- The Debrief challenge is a purple team follow-up to Toxic Transfer.
It names finstripe/create_transfer as the poisoned tool and grades the
directive by meaning, not verbatim text.
- Upgrade directive matching from substring to token overlap with light
prefix stemming so students can paraphrase. A configurable
directive_threshold (default 0.6) controls how much of the expected
answer must appear.
24 unit tests pass, including reordered-paraphrase acceptance and
unrelated-text rejection. Full submit-to-score flow verified against the
challenge config.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds PurpleTeamEvaluator, which scores forensic reasoning instead of agent behavior: a student submits {server, tool, directive} identifying a compromised MCP server, the poisoned tool, and the directive it carried, and each field scores independently against a known seeded answer. Also adds the investigation-submission API endpoint and "The Debrief" challenge that uses it.
Test plan