Repository navigation
Conversation
Verify agents return verdict fields only, so organize.py read empty skill, severity, file and title off verified.jsonl. The security index came out empty and the tracking issue comment listed finding IDs. Join verified entries to all-findings.jsonl by findingId, as getsentry#559 does for create_issue.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent: claude
|
@tlq5l is attempting to deploy a commit to the Sentry Team on Vercel. A member of the Team first needs to authorize it. |
There was a problem hiding this comment.
Looks good to me. The verify prompt only returns verdict fields, so building the index off verified.jsonl alone could never match a security skill. Joining on findingId with the verified entry winning fixes that and matches what #559 does. I ran organize_test.py against main and it fails both cases; on this branch both pass.
One thing for a follow-up, not this PR: generate_report.py (the severity breakdown and the verified findings table in summary.md/report.json) and index_prs.py (get_verified_files) read severity, skill, file and title off verified.jsonl too, so they'll still show up empty. If you take that on, moving read_verified_findings into _utils.py would let all three scripts and create_issue.py share one join.
There was a problem hiding this comment.
Looks good to me. verify-prompt.md only returns verdict fields, so joining on findingId with the verified entry winning is the right fix, and it matches what #559 does in create_issue.py. I ran organize_test.py against this branch and against main's organize.py. Both tests fail on main and pass here, so they cover the empty-index failure and the PR list falling back to bare IDs.
One non-blocking follow-up: generate_report.py, which organize.py runs in step 5, still reads verified.jsonl directly. That means the Verified Findings table in summary.md, the severity breakdown, and the per-PR severity in report.json will still come out blank or info. The Security Findings section there is fixed now because it reads security/index.jsonl. If you move read_verified_findings() into _utils.py, generate_report.py could use it too, either in this PR or the next one.
What changed
organize.pybuiltsecurity/index.jsonland the tracking issue's "Sweep Complete" comment fromverified.jsonl. Verify agents' entries there carry noskill,severityortitle. No security finding reached the security index, and the PR list showed finding IDs with no severity.A new
read_verified_findings()merges each verified entry over itsall-findings.jsonlrecord, joined byfindingId. Fields in the verified entry still win, as in #559's change tocreate_issue.py. Both the security index and the tracking issue use it. A verified entry with no match behaves as before.Adds
organize_test.py: end-to-end tests that run the realextract_findings.pyon a warden log, thenorganize.pywith a stubgh.Verification
Both new tests fail on
mainand pass on this branch.organize.pyis the only caller of the new helper, andorganize-phase.mdruns it.Ran live (agent-run) at cf2e91c:
uv run python -B skills/warden-sweep/scripts/organize_test.py: exit 0, last output lineOK.Follow-up to #559.