[sentry-junior]

Optional security eval suite: does each wrdn-* skill catch bugs builtin security-review misses?

It runs security-review plus wrdn-authz, wrdn-pii, wrdn-data-exfil, wrdn-code-execution and wrdn-dos-review (warden-skills pinned at daf01f9) over the same code. Every lane is scored against the same known bugs, and each wrdn-* lane gets one verdict.

What's in it

The suite is a new package, packages/security-evals. Nothing in packages/evals or .github/workflows/evals.yml changes, so pnpm evals, the run-evals label and the PR Evals workflow never run it.

• Scenarios: one per entry in sentry-vulnerability-corpus.json (86), plus 47 cases in cases/<skill>/ (25 bugs, 22 clean twins). That makes 133 entries in 53 shards. Cases come from real fix commits in getsentry/sentry, getsentry/sentry-python, getsentry/sentry-wizard and getsentry/dotagents. Every entry pins repo, SHA and targets. A hit on a clean case is a false positive.
• Repo context: --repo owner/name=<path>, repeatable, with --sentry-repo kept as an alias. Each SHA gets one git worktree add --detach, reused across lanes, entries and repeats. The only network step is a setup-time git fetch for a missing SHA. Without a clone, setup stops with an error.
• Skills: --skills-repo points at a local warden-skills clone. Setup makes one worktree at the pin and loads each skill once. If that fails, the run stops before any pass.
• Execution: one Warden pass per shard, with every selected skill on the same file-mode context. The judge applies the corpus rule: same bug, roughly the same location. A failed pass, chunk or judge call counts as an error, never a miss.

Proving a skill

The thresholds are in VERDICT_RULES (src/verdict.ts) and the README:

• earns-keep: at least one bug the skill caught in 2 of 3 repeats or more, that security-review caught in none, with both lanes at 3+ repeats.
• no-unique-catches: no such bug, nothing unsettled, and 5+ owned bug cases with 3+ repeats in both lanes.
• insufficient-evidence: anything else. That covers any errored entry, too few cases, too few repeats, or a catch that security-review gets only some of the time.

The report leads with these verdicts. It never prints a bare "0 caught only by X".

Adding a case

pnpm evals:security:add https://github.com/getsentry/sentry/pull/<n> --skill wrdn-dos-review

The command writes the bug case at the fix's parent and the clean twin at the fix. Targets come from the non-test .py/.ts files the fix modified. Then fill in summary and run the free validator, pnpm -C packages/security-evals test, which is part of regular CI. It checks:

• the schema and unique ids
• no overlap with the corpus
• a known skill directory
• no TODO summary
• a clean twin or a noClean reason
• baseline entries still exist at the same ref

After that, run --entries <ids> --repeats 3 for real.

Baseline

Each lane in baseline/results.json records its runtime, model, effort, aux model, judge, warden version and skill ref. If any of that differs from the current run, the lane is dropped with a warning. The baseline starts empty, because the old imported benchmark lane could never match. baseline:update <results.json> folds in a clean run and refuses results that contain errors.

Running it

git clone --filter=blob:none https://github.com/getsentry/sentry.git ~/src/sentry
git clone https://github.com/getsentry/warden-skills.git ~/src/warden-skills
pnpm evals:security --repo getsentry/sentry=$HOME/src/sentry --skills-repo ~/src/warden-skills \
  --skills security-review,wrdn-dos-review --entries 'wrdn-dos-review/*' --repeats 3

A full pass also needs --repo clones of getsentry/sentry-python, getsentry/sentry-wizard and getsentry/dotagents (see the README). The workflow prepares all four with scripts/prepare-clone.sh <owner/name> <dir>.

Defaults match the production lane: pi, openrouter/x-ai/grok-4.5 at high effort, openrouter/openai/gpt-5.6-luna at high effort for aux, and maxTurns 100. CI runs it through the workflow_dispatch-only Security Evals workflow.

Not done

• No live run of this package yet. It needs an OpenRouter key and the clones.
• Owned bug cases: authz 5, dos-review 6, data-exfil 5, code-execution 3, pii 6. Every skill except code-execution can now reach any verdict. Code-execution stays insufficient-evidence until it has 5: fixed, attacker-reachable command, template or deserialization bugs are rare in getsentry history, and most "injection" fixes there are workflow YAML or latent hardening.
• Next step, not built: harvest candidate cases from production Warden PR comments that led to a fix.
• Locally, packages/warden's remote.test.ts fails because the sandbox git hook rejects the test's commit author. This PR doesn't touch that code.

via @dcramer.

--

View Session [Sentry]

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
warden-prod	Ready	Preview	Nov 7, 2026 2:53am UTC

[semgrep-code-getsentry]

Semgrep found 1 ssc-d25b9656-2752-480c-98ff-eb392322b6e1 finding:

• packages/security-evals/src/baseline.ts
  • L24 - Triage

🟠 High severity issue identified in your code:

Risk: Affected versions of zod are vulnerable to Allocation of Resources Without Limits or Throttling. Zod parses every element of an array, set or map schema and records one validation issue per invalid element, with no cap or early stop. Upper size bounds (.max(), .length(), .size(), or z.maxLength()/z.length()/z.maxSize()/z.size() checks) only ran after all elements had been parsed, so they did not limit the work. An attacker who submits a very large collection (e.g. a million invalid items for z.array(z.string().uuid())) forces the allocation of millions of issue objects, which can exhaust memory and crash the process. Upgrade to zod 4.6.6 or later, which rejects oversized input before parsing its elements when the schema has an upper bound, and add an upper bound (e.g. .max(n)) to every collection schema that parses untrusted input: unbounded schemas stay exposed even after the upgrade.

Fix: Upgrade this library to at least version 4.6.6 at warden/pnpm-lock.yaml:6596.

Reference(s): GHSA-mrhx-c7xc-m3w5, CVE-2023-54404