Skip to content

fix(sweep): Build the security index from all-findings - #560

Open
tlq5l wants to merge 1 commit into
getsentry:mainfrom
tlq5l:fix/sweep-security-index-from-all-findings
Open

tlq5l wants to merge 1 commit into
getsentry:mainfrom
tlq5l:fix/sweep-security-index-from-all-findings

Conversation

@tlq5l

@tlq5l tlq5l commented Nov 4, 2026 •

Copy link
Copy Markdown

What changed

organize.py built security/index.jsonl and the tracking issue's "Sweep Complete" comment from verified.jsonl. Verify agents' entries there carry no skill, severity or title. No security finding reached the security index, and the PR list showed finding IDs with no severity.

A new read_verified_findings() merges each verified entry over its all-findings.jsonl record, joined by findingId. Fields in the verified entry still win, as in #559's change to create_issue.py. Both the security index and the tracking issue use it. A verified entry with no match behaves as before.

Adds organize_test.py: end-to-end tests that run the real extract_findings.py on a warden log, then organize.py with a stub gh.

Verification

Both new tests fail on main and pass on this branch. organize.py is the only caller of the new helper, and organize-phase.md runs it.

Ran live (agent-run) at cf2e91c: uv run python -B skills/warden-sweep/scripts/organize_test.py: exit 0, last output line OK.

Follow-up to #559.

Verify agents return verdict fields only, so organize.py read empty
skill, severity, file and title off verified.jsonl. The security index
came out empty and the tracking issue comment listed finding IDs.
Join verified entries to all-findings.jsonl by findingId, as getsentry#559 does
for create_issue.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent: claude
@vercel

vercel Bot commented Nov 4, 2026

Copy link
Copy Markdown

@tlq5l is attempting to deploy a commit to the Sentry Team on Vercel.

A member of the Team first needs to authorize it.

@sentry-junior sentry-junior Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. The verify prompt only returns verdict fields, so building the index off verified.jsonl alone could never match a security skill. Joining on findingId with the verified entry winning fixes that and matches what #559 does. I ran organize_test.py against main and it fails both cases; on this branch both pass.

One thing for a follow-up, not this PR: generate_report.py (the severity breakdown and the verified findings table in summary.md/report.json) and index_prs.py (get_verified_files) read severity, skill, file and title off verified.jsonl too, so they'll still show up empty. If you take that on, moving read_verified_findings into _utils.py would let all three scripts and create_issue.py share one join.

@sentry-junior sentry-junior Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. verify-prompt.md only returns verdict fields, so joining on findingId with the verified entry winning is the right fix, and it matches what #559 does in create_issue.py. I ran organize_test.py against this branch and against main's organize.py. Both tests fail on main and pass here, so they cover the empty-index failure and the PR list falling back to bare IDs.

One non-blocking follow-up: generate_report.py, which organize.py runs in step 5, still reads verified.jsonl directly. That means the Verified Findings table in summary.md, the severity breakdown, and the per-PR severity in report.json will still come out blank or info. The Security Findings section there is fixed now because it reads security/index.jsonl. If you move read_verified_findings() into _utils.py, generate_report.py could use it too, either in this PR or the next one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant