Skip to content

feat: integrate security scanning checks (QEMETRICS-2383)#26

Closed
Benkapner wants to merge 2 commits into
RHEcosystemAppEng:mainfrom
Benkapner:QEMETRICS-2383/security-scan
Closed

feat: integrate security scanning checks (QEMETRICS-2383)#26
Benkapner wants to merge 2 commits into
RHEcosystemAppEng:mainfrom
Benkapner:QEMETRICS-2383/security-scan

Conversation

@Benkapner

@Benkapner Benkapner commented May 26, 2026

Copy link
Copy Markdown

Hey Guy, this PR is the security scanning gate we talked about in QEMETRICS-2380. It adds a new pipeline step that catches dangerous skill submissions before they reach scaffolding and container builds.

What it does

The scanner reads all SKILL.md files in a submission and checks for:

  • Prompt injection (20 patterns) - "ignore previous instructions", jailbreak attempts, prompt leaks, encoding tricks, markdown image exfiltration, curl/wget data exfiltration, and more
  • Sensitive file references (16 patterns) - ~/.ssh/, ~/.aws/credentials, /etc/shadow, ~/.kube/config, ~/.gnupg/, etc.
  • Secret env vars (12 patterns) - $OPENAI_API_KEY, $AWS_SECRET_ACCESS_KEY, $GITHUB_TOKEN, etc.
  • Dangerous commands (5 patterns) - sudo, chmod 777, rm -rf /, curl | sh

The scanner is context-aware: patterns inside code blocks or after "for example" become warnings instead of errors. Warnings don't block, errors do.

Where it fits

Runs after validation and before scaffold, in parallel with ai-review. Feature flag enable-security-scan (default on). No LLM calls, no new dependencies.

Files

  • abevalflow/security_scanner.py - core scanning module
  • scripts/security_scan.py - CLI entry point
  • tests/test_security_scan.py - 58 tests
  • pipeline/tasks/security_scan.yaml - Tekton task
  • pipeline/pipeline.yaml - added parameter, task, updated scaffold runAfter

All 58 tests pass, full suite has no regressions, ruff clean.

Related: QEMETRICS-2383

…ICS-2383)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Benkapner

This comment was marked as spam.

…on checks

Port security detection patterns from harness-eval-lab's eval-setup-security
into the ABEvalFlow security scanner. Adds four new finding categories:

- Data exfiltration (8 patterns): curl/wget post with command substitution,
  DNS tunneling, webhook exfil, base64/archive piped to network
- Reverse shells (10 patterns): bash, netcat, ncat, python, perl, ruby, php,
  socat, named pipe, powershell
- Code obfuscation (6 patterns): eval+decode, fromCharCode, hex/unicode
  escape sequences, dynamic exec
- Hidden content (unicode deception): zero-width chars, RTL overrides,
  homoglyphs

Also downgrades prompt injection from ERROR to WARNING since several
patterns (act as, simulate mode, translate) are broad enough to match
legitimate skill instructions.

110 tests, all passing, no new dependencies.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Benkapner

Copy link
Copy Markdown
Author

Update: expanded security scanner with patterns from harness-eval-lab

Added four new detection categories ported from harness-eval-lab's eval-setup-security command, adapted to fit ABEvalFlow's scanner structure:

New blocking checks (ERROR):

  • Data exfiltration (8 patterns): curl/wget with -d/--data + command substitution, wget --post-data, DNS tunneling (dig/nslookup TXT), webhook exfil (pipedream/requestbin/ngrok), base64 or tar archives piped to curl/wget/nc
  • Reverse shells (10 patterns): bash /dev/tcp, netcat, ncat, python/perl/ruby/php socket shells, socat exec, named pipes, PowerShell TCPClient
  • Code obfuscation (6 patterns): eval(atob(...)), eval(base64.b64decode(...)), String.fromCharCode(), hex/unicode escape sequences, exec(compile(...)), charCode round-trips
  • Unicode deception (44 chars): zero-width characters (6), RTL overrides (9), Cyrillic/Greek homoglyphs (29, WARNING severity)

Changed:

  • Prompt injection downgraded from ERROR to WARNING (non-blocking). Some patterns like "act as", "simulate mode", "translate" are broad enough to match legitimate skills.

Deferred for future PR:

  • Hidden instruction patterns (HTML/markdown comments with system/override keywords, base64 blobs, data URIs). The base64 blob regex ([A-Za-z0-9+/]{40,}) would false-positive on SHA hashes, JWT tokens, and long identifiers. Worth revisiting with a higher threshold or requiring padding.

All context-aware (code fences downgrade to WARNING). 110 tests, no new dependencies, ruff clean.

@Benkapner Benkapner closed this Jun 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant