Skip to content

feat: nightly drift watch with a security-drift issue - #12

Open
VincentJerico wants to merge 2 commits into
mainfrom
feat/nightly-drift-watch
Open

VincentJerico wants to merge 2 commits into
mainfrom
feat/nightly-drift-watch

Conversation

@VincentJerico

Copy link
Copy Markdown
Owner

What & why

Second of three security-automation PRs. The checks currently run only when someone pushes. Both images are pinned, so the target doesn't change between runs. What does change is what's known about it: Trivy's database adds new CVEs for packages Juice Shop already ships. This adds a nightly job that notices and tells you.

  • .github/workflows/nightly.yml: runs daily at 19:30 UTC (03:30 PH, after performance-testing's nightly run) or by hand. It re-runs the pinned Playwright tests, the ZAP baseline and the Trivy scan, then compares the results with the baseline.
  • baseline/findings.json: the known findings from a real scan on 2026-11-06: 130 CVE IDs and 9 ZAP alerts. Regenerate it with npm run baseline:update.
  • scripts/drift.ts: the comparison. New CVEs or ZAP alerts exit 1. Findings that disappear are listed but don't fail; updating the baseline stays a deliberate PR.
  • scripts/drift-issue.sh: keeps one issue labelled security-drift.
    • Drift or a failing pinned test: opens the issue, or comments on it if it's already open.
    • Clean run: comments and closes it.
    • It creates the label if missing. The job gets issues: write and nothing more.
  • scripts/zap-baseline.sh fix (own commit): a ZAP crash, for example a missing config, used to print "a FAIL rule triggered". It now deletes old reports first and exits 3 with "scan error" when no report was written.

How it was tested

Locally (Node 22.23.4, Docker), against Juice Shop v18.0.0:

  • tests/drift.selftest.spec.ts, 5 passed: collects findings correctly, no drift on a match, a planted new CVE and a planted new ZAP alert are drift, disappeared findings aren't, and the CLI exits 1 or 0 accordingly.
  • tests/drift-issue.selftest.spec.ts, 4 passed: runs the issue script with a fake gh and checks the exact calls and exit code for clean with no issue, clean with an open issue, drift with no issue, and a failing test with an open issue.
  • Baseline against its own scan: "No drift: the scans match the baseline", exit 0.
  • zap-baseline.sh: with a missing config, exit 3 "scan error, no zap/report.json was written (exit 1)". A normal scan exits 0 (8 WARN, 0 FAIL).
  • Full suite: 38 passed. Lint, format and typecheck pass. actionlint and shellcheck report no issues.

Not tested yet: the workflow on GitHub itself. A manual run needs the workflow file on main. After merge I'd trigger it once by hand; with no drift, that run passes and opens nothing.

🤖 Generated with Claude Code

VincentJerico and others added 2 commits November 6, 2026 21:53
zap-baseline.py exits 1 when it crashes, for example on a missing config
file, so the script blamed a FAIL rule. It now deletes old reports before
scanning and exits 3 with a clear message when no report.json was written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Re-runs the pinned tests, ZAP and Trivy every night and compares the
scans with baseline/findings.json (130 CVEs and 9 ZAP alerts from the
2026-11-06 scan). New findings or a failing test open or update one
issue labelled security-drift; a clean run closes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant