diff --git a/AGENTS.MD b/AGENTS.MD index 433393f..73e4773 100644 --- a/AGENTS.MD +++ b/AGENTS.MD @@ -55,7 +55,7 @@ Codex CLI output: avoid Markdown tables by default; they render poorly there. Us - CI: `gh run list/view`; rerun/fix until green when asked. - `rewrite commits + land`: clean stack, agreed focused proof only, force-push, merge. No Codex review, PR-body proof polish, or CI babysitting unless asked. - Pre-land/pre-commit code changes: use `$autoreview` until no accepted/actionable findings remain, unless equivalent manual review already done, trivial/docs-only, or user opts out. -- API work: also run `$autoreview --preset claude-opus` (Claude Opus 4.8) before merge/ship. +- API work: also run `$autoreview --engine claude --model claude-opus-4-8` before merge/ship. (`--preset` is gone; upstream autoreview uses `--engine`/`--model`.) - Replies: cite fix + file/line; resolve threads only after fix lands. - Issue fixed on `main` with proof: comment proof + commit/PR, then close. - User-facing fixes/landed PRs: changelog unless pure test/internal. diff --git a/CHANGELOG.md b/CHANGELOG.md index 7bdc988..f3fbff9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,12 @@ summary: Timeline of guardrail helper changes mirrored from Sweetistics and rela # Changelog +## 2026-07-22 — Autoreview From Canonical Source +- Replaced the vendored `autoreview` skill with the canonical `openclaw/agent-skills` copy at `c4ab5e7` (helper 2,485 -> 12,001 lines), including its scripts, test suite, fixtures, and the skill-level `AGENTS.md` sync rule. +- Gains TruffleHog secret scanning over the reviewed diff, a Scope Governor that classifies findings as in-scope blocker / follow-up / stop-and-escalate, oversized-bundle handling, release-branch rules, and Codex `gpt-5.6-sol` with an access-only fallback to `gpt-5.6-terra`. +- Engines that cannot be fully isolated (`droid`, `copilot`, `opencode`, `cursor`) are now refused rather than run; `--preset` is gone, so the AGENTS rule for API work uses `--engine claude --model claude-opus-4-8`. +- TruffleHog is a hard requirement: autoreview exits early when it is not on PATH. + ## 2026-07-22 — Validator UTF-8 Fix - `scripts/validate-skills` reads `SKILL.md` as UTF-8 instead of inheriting the process locale, so validation and the pre-commit hook stop failing with `invalid byte sequence in US-ASCII` in shells without `LANG` set. diff --git a/skills/autoreview/AGENTS.md b/skills/autoreview/AGENTS.md new file mode 100644 index 0000000..5a0173d --- /dev/null +++ b/skills/autoreview/AGENTS.md @@ -0,0 +1,6 @@ +# Autoreview Skill + +- Canonical source: `openclaw/agent-skills`, under `skills/autoreview`. +- Before editing any copy, fast-forward a checkout of `openclaw/agent-skills` from `origin/main`. +- Make and validate shared changes in canonical `skills/autoreview` first, then sync the complete directory into downstream repos. +- Never create repo-local behavior variants; downstream differences belong in repo-level validation, not the skill. diff --git a/skills/autoreview/CLAUDE.md b/skills/autoreview/CLAUDE.md new file mode 120000 index 0000000..47dc3e3 --- /dev/null +++ b/skills/autoreview/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/skills/autoreview/SKILL.md b/skills/autoreview/SKILL.md index 71e1648..775f103 100644 --- a/skills/autoreview/SKILL.md +++ b/skills/autoreview/SKILL.md @@ -1,20 +1,24 @@ --- name: autoreview -description: "Run a structured code review (Codex default, optional Claude, Pi, Copilot, or OpenCode) as a closeout check on a local or PR branch before commit or ship." +description: "Pre-commit/ship code review: Codex default; optional Claude or Pi." --- # Auto Review -Run the bundled structured review helper as a closeout check. This is code review, not approval routing. +Run the bundled structured review helper as a closeout check. This is code review, not Guardian `auto_review` approval routing. -Codex review is the default when no engine is set. Bram's copy uses the local CLI/configured model by default; pass `--model` or `AUTOREVIEW__MODEL` only when a specific model is required. Optional engines are used only when installed and explicitly selected. +Codex review is the default when no engine is set. It uses `gpt-5.6-sol` with `high` reasoning by default, then retries once with `gpt-5.6-terra` only when the account cannot access Sol. Claude review is optional and uses `claude-fable-5` by default. + +For user-visible behavior, pair autoreview with `behavior-validator`. Autoreview is source-aware and judges the change bundle; behavior validation is source-blind and judges the running product or tool against a behavior contract. A clean autoreview is not proof that a UI, CLI, API, or generated artifact works from the user's perspective. Use when: -- user asks for Codex review / Claude review / Pi review / Copilot review / OpenCode review / autoreview / second-model review +- user asks for Codex review / Claude review / Pi review / autoreview / second-model review - after non-trivial code edits, before final/commit/ship - reviewing a local branch or PR branch after fixes +Do not require autoreview for a change whose entire diff is prose-only internal notes or `SKILL.md` documentation. Still inspect the diff directly and run the repository's lightweight documentation validation, if any. This exception does not cover user-facing documentation, executable examples, configuration, scripts, generated files, or behavior changes. + ## Contract - Treat review output as advisory. Never blindly apply it. @@ -24,18 +28,20 @@ Use when: - Prefer small fixes at the right ownership boundary; no refactor unless it clearly improves the bug class. - When an accepted finding shows a bug class or repeated pattern, inspect the current PR scope for sibling instances before fixing. - Fix the scoped bug class at once when practical; stop at touched surfaces, owner boundaries, and clear follow-up territory. -- Keep going until structured review returns no accepted/actionable findings. +- Keep going until structured review returns no accepted/actionable findings only while the work remains inside the original task scope. - If a review-triggered fix changes code, rerun focused tests and rerun the structured review helper. - For security-audit suppression changes, verify accepted findings remain auditable: suppressed findings stay in structured output, active output keeps an unsuppressible suppression notice, and aggregate findings cannot hide unrelated active risk. -- Never switch or override the requested review engine/model. If the review hits model capacity, retry the same command a few times with the same engine/model. +- Never switch or override the requested review engine/model except for the documented Codex Sol-to-Terra account-access fallback. Capacity, rate-limit, and unrelated failures keep the same engine/model. - Be patient with large bundles. Structured review can take up to 30 minutes while the model call is active, especially with Codex tools or web search. -- Treat heartbeat lines like `review still running: ... elapsed=... pid=...` as healthy progress, not a hang. Let the helper continue while heartbeats are advancing. Pass `--stream-engine-output` when live engine text is useful; Codex and Claude filter tool/file chatter, other engines pass raw output through. +- Treat heartbeat lines like `review still running: ... elapsed=... pid=...` as healthy progress, not a hang. Let the helper continue while heartbeats are advancing. Pass `--stream-engine-output` when live engine text is useful; Codex and Claude filter tool/file chatter, other runnable engines pass raw output through. - Do not kill a review just because it has been quiet for 2-5 minutes, or because it is still running under the 30-minute window. Inspect the process only after missing multiple expected heartbeats, after 30 minutes, or after an obviously failed subprocess; prefer letting the same helper command finish. -- Tools are useful in review mode. The helper allows read-only inspection tools and web search by default so reviewers can check dependency contracts, upstream docs, and current behavior. +- Tools are useful in review mode. Codex receives the validated bundle in an empty workspace so ignored files and linked-worktree metadata remain unreadable; web search stays available for dependency contracts and upstream docs. - Security perspective is always included, but it should not cripple legitimate functionality. Report security findings only when the change creates a concrete, actionable risk or removes an important safety check. +- Reviewer subprocesses preserve engine authentication and non-credentialed proxy variables needed by headless or restricted-network environments while stripping process-injection, Git override, and credentialed proxy values. +- Before engine invocation, autoreview runs TruffleHog over temporary snapshots of the exact added or modified content under review. It intentionally matches TruffleHog's low-false-positive pre-commit policy (`verified,unknown`); it does not classify arbitrary password-like strings or rescan unchanged history. Install TruffleHog using its official platform-neutral instructions; autoreview fails with that link when the binary is unavailable and never auto-installs it. Repositories should also run TruffleHog in pull-request CI as a backup outside autoreview; repository-local Git hooks are optional. Review bundles still omit security-sensitive paths or files, and explicit prompt and dataset inputs remain checked before engine invocation. Safe large diffs are sent as one pass while they fit the aggregate prompt limit, then partitioned into complete bounded passes without truncation. - For regression provenance, keep roles separate: blamed code author, blamed PR author, PR merger/committer, current PR author, and PR/date. If no blamed PR is traceable, use the blamed commit as the provenance: commit SHA, date, and author username. Do not guess a merger or frame missing PR metadata as a separate finding. -- If the blamed PR was merged by automation, identify the human trigger when practical. Check timeline/comments first; if rate-limited, use gitcrawl/cache or public PR HTML. Report `automerge triggered by @login`; if not found, say trigger unknown. -- Do not invoke built-in `codex review`, nested reviewers, or reviewer panels from inside the review. The helper builds one bundle, calls one selected engine, validates one structured result, and stops. +- If the blamed PR was merged by `clawsweeper[bot]` or another automation, identify the human trigger when practical. Check timeline/comments first; if rate-limited, use gitcrawl/cache or public PR HTML. Look for maintainer commands such as `@clawsweeper automerge`, `/landpr`, or labels/status comments that armed automerge. Report `automerge triggered by @login`; if not found, say trigger unknown. +- Do not invoke built-in `codex review`, nested reviewers, or reviewer panels from inside the review. The helper builds one validated bundle, calls the selected engine once for normal inputs or once per complete bounded chunk for oversized inputs, validates the structured results, and stops. - Stop as soon as the helper exits 0 with no accepted/actionable findings. Do not run an extra review just to get a nicer "clean" line, a second opinion, or clearer closeout wording. - Treat the helper's successful exit plus absence of actionable findings as the clean review result, even if the underlying Codex CLI output is terse. - Multi-reviewer panels are opt-in only. Use them when explicitly requested or when risk justifies the extra spend; the main agent still verifies every accepted finding before fixing. @@ -44,6 +50,42 @@ Use when: - If Gitcrawl reports a portable manifest mismatch, source/runtime DB health error, or stale portable-store checkout, run `gitcrawl doctor --json` and inspect `source_db_health`, `runtime_db_health`, and `portable_store_status` before falling back to live GitHub. - Do not push just to review. Push only when the user requested push/ship/PR update. +## Scope Governor + +Autoreview is a closeout gate, not permission to rewrite the task. + +Before the first review, freeze a scope baseline: original request or issue, target branch, intended behavior, owner boundary, changed files, and non-test LOC. For inherited or already-bloated branches, use the intended PR diff as the baseline rather than accepting all existing branch drift. + +Before patching a finding, classify it: + +- **In-scope blocker**: the finding is introduced by the current diff, affects the same owner boundary, and can be fixed without changing the task's contract. +- **Follow-up**: the finding is real but belongs to an adjacent bug class, sibling surface, cleanup, or broader hardening track. +- **Stop-and-escalate**: the finding requires a new protocol/config/storage/public API contract, a different owner boundary, a release-process change, or a design choice outside the original request. + +Stop patching and report the scope break instead of continuing when: + +- a narrow PR turns into an architecture change, protocol change, migration, or release-process change; +- the diff grows past 2x the original files or non-test LOC without explicit approval to expand scope; +- two review-triggered patch cycles have not converged; pause and reclassify every remaining finding before another edit; +- the best fix is "define the canonical contract first" rather than another local inference layer; +- fixing the accepted finding would make the PR no longer describe the same behavior, issue, or owner boundary. + +After the two-cycle pause, continue only when every remaining accepted finding is still an in-scope blocker. Otherwise preserve the useful analysis, identify the smallest safe landed subset if one exists, and open or request a follow-up for the larger fix. Do not keep committing speculative fixes just to satisfy the reviewer. + +Do not stack or push review-triggered fix commits while scope classification or focused proof is unresolved. Keep exploratory edits local until the cycle is proven in scope; if scope breaks, remove them from the landing lane instead of preserving them as branch history. + +Critical exceptions must be explicit: active data loss, crash, broken install/upgrade, release blocker, or concrete security exposure. If the exception is not one of those, it is not critical enough to blow up scope. + +## Release Branches And Release Process + +On release, beta, stable, hotfix, signing, notarization, appcast, package-publish, or release-check work, use freeze discipline even when the branch name is not release-like: + +- Fix only release blockers, failed release infrastructure, exact backports, install/upgrade breakage, data loss, crashes, or concrete security exposure. +- Treat non-blocking autoreview findings as follow-ups for `main`, not reasons to broaden the release branch. +- Do not introduce new product behavior, config surface, protocol shape, migration, plugin ownership, docs narrative, or process policy unless it directly unblocks the release. +- Keep proof tied to the release target: exact branch/ref, failing check or shipped-risk reason, smallest command/proof, and whether the fix must also forward-port to `main`. +- If review discovers a real but non-critical design problem during release closeout, stop with a follow-up issue/PR plan; do not use the release branch as the refactor lane. + ## Skill Path (set once) Set the skill script paths once, then use `"$AUTOREVIEW"` and `"$AUTOREVIEW_HARNESS"` in the examples below. @@ -51,21 +93,57 @@ Set the skill script paths once, then use `"$AUTOREVIEW"` and `"$AUTOREVIEW_HARN Choose one: ```bash -# This agent-scripts checkout: -export AUTOREVIEW="$HOME/Projects/agent-scripts/skills/autoreview/scripts/autoreview" -export AUTOREVIEW_HARNESS="$HOME/Projects/agent-scripts/skills/autoreview/scripts/test-review-harness" +# Project-local skill in the current repo for Codex and other agents: +export AUTOREVIEW=".agents/skills/autoreview/scripts/autoreview" +export AUTOREVIEW_HARNESS=".agents/skills/autoreview/scripts/test-review-harness" ``` ```bash -# Current repo copy: +# Claude Code project-local skill in the current repo: +export AUTOREVIEW=".claude/skills/autoreview/scripts/autoreview" +export AUTOREVIEW_HARNESS=".claude/skills/autoreview/scripts/test-review-harness" +``` + +```bash +# Source checkout of openclaw/agent-skills: export AUTOREVIEW="skills/autoreview/scripts/autoreview" export AUTOREVIEW_HARNESS="skills/autoreview/scripts/test-review-harness" ``` ```bash -# Codex global skill: -export AUTOREVIEW="$HOME/.codex/skills/autoreview/scripts/autoreview" -export AUTOREVIEW_HARNESS="$HOME/.codex/skills/autoreview/scripts/test-review-harness" +# Global skill: +export AGENTS_HOME="${AGENTS_HOME:-$HOME/.agents}" +export AUTOREVIEW="$AGENTS_HOME/skills/autoreview/scripts/autoreview" +export AUTOREVIEW_HARNESS="$AGENTS_HOME/skills/autoreview/scripts/test-review-harness" +``` + +When using Claude Code, set `AGENTS_HOME="$HOME/.claude"` for global skills. + +On native Windows, choose the matching pair: + +```powershell +# Project-local skill in the current repo for Codex and other agents: +$AUTOREVIEW = ".agents\skills\autoreview\scripts\autoreview" +$AUTOREVIEW_HARNESS = ".agents\skills\autoreview\scripts\test-review-harness.ps1" +``` + +```powershell +# Claude Code project-local skill in the current repo: +$AUTOREVIEW = ".claude\skills\autoreview\scripts\autoreview" +$AUTOREVIEW_HARNESS = ".claude\skills\autoreview\scripts\test-review-harness.ps1" +``` + +```powershell +# Source checkout of openclaw/agent-skills: +$AUTOREVIEW = "skills\autoreview\scripts\autoreview" +$AUTOREVIEW_HARNESS = "skills\autoreview\scripts\test-review-harness.ps1" +``` + +```powershell +# Global skill: +$AgentsHome = if ($env:AGENTS_HOME) { $env:AGENTS_HOME } else { Join-Path $HOME ".agents" } +$AUTOREVIEW = Join-Path $AgentsHome "skills\autoreview\scripts\autoreview" +$AUTOREVIEW_HARNESS = Join-Path $AgentsHome "skills\autoreview\scripts\test-review-harness.ps1" ``` ## Pick Target @@ -83,35 +161,16 @@ or branch diff instead; do not force dirty modes just because the helper docs mention dirty work first. A clean local review only proves there is no local patch. -Risky untracked paths such as `.env`, keys, tokens, credentials, and logs are omitted by default. Include them only when intentionally needed: - -```bash -"$AUTOREVIEW" --mode local --include-risky-paths -``` - Branch/PR work: ```bash "$AUTOREVIEW" --mode branch --base origin/main ``` -Optional review context is first-class: - -```bash -"$AUTOREVIEW" --mode branch --base origin/main --prompt-file /tmp/review-notes.md --dataset /tmp/evidence.json -``` - -Loop closeout can run the default Codex review, then a separate Claude Opus review: - -```bash -"$AUTOREVIEW" --mode branch --base origin/main -"$AUTOREVIEW" --preset claude-opus --mode branch --base origin/main -``` - -Claude Opus only: +Optional review context is first-class. Prompt files and datasets must be repo-relative so review bundles cannot pull arbitrary host files: ```bash -"$AUTOREVIEW" --preset claude-opus --mode branch --base origin/main +"$AUTOREVIEW" --mode branch --base origin/main --prompt-file review-notes.md --dataset evidence.json ``` If an open PR exists, use its actual base: @@ -132,6 +191,29 @@ clean `main` against `origin/main` is usually an empty diff after push. For a small stack, review each commit explicitly or review the branch before merging with `--base`. +## Oversized Bundles + +The helper scans the full patch before partitioning it. A safe bundle that fits +the aggregate prompt limit remains one integrated review pass. Larger bundles +are split at bundle sections and file boundaries where possible; an oversized +single-file block is split at line boundaries with repeated file/hunk context +and an absolute new- or old-file line offset. Untracked snapshots use +injection-safe source-line records so continuation passes retain reportable +locations. A single physical diff line split across passes also retains its +original addition, deletion, or context marker. +Every original bundle byte appears exactly once across the pass sequence, and +all validated reports are merged before required-finding and exit-status checks. +The helper caps one run at eight bounded passes so an unexpectedly huge branch +cannot create unbounded model calls; split still-larger work into coherent review +targets. + +Chunking makes large-diff review usable, but it cannot give one model call every +cross-file implementation detail. For architecture-heavy changes, still prefer +a coherent branch or PR shape whose semantic decision surface fits one pass. +Removing verified non-authoritative generated noise remains useful, but never +drop lockfiles, generated clients, policies, manifests, schemas, or other +independently semantic artifacts merely to shrink the review. + ## Parallel Closeout Format first if formatting can change line locations. Then it is OK to run tests and review in parallel: @@ -143,6 +225,29 @@ Format first if formatting can change line locations. Then it is OK to run tests On Windows, the default `--parallel-tests` shell preserves the platform `cmd.exe` semantics used by Python `shell=True`. Use `--parallel-tests-shell powershell` or `--parallel-tests-shell pwsh` when the focused test command is PowerShell-specific. +Parallel tests inherit only a small allowlist of ordinary OS, CI, and toolchain +variables. Put additional non-secret project controls directly in the test command. +Home and standard config directories point to a temporary isolated root that is +removed after the command exits. Do not put secrets in the command because it is +printed before execution. Set `OPENCLAW_TESTBOX=1` on the autoreview process, not +inside the test command, because the environment snapshot and credential staging +happen before the test shell starts: + +```bash +OPENCLAW_TESTBOX=1 "$AUTOREVIEW" --parallel-tests "pnpm check:changed" +``` + +On POSIX, the helper puts this isolated Testbox home under the short, sticky +system `/tmp`; Blacksmith creates an SSH control socket below that home, and a +long macOS `TMPDIR` can exceed the Unix-socket path limit. With an older helper, +prefix the outer autoreview process with `TMPDIR=/tmp`. Setting `TMPDIR` inside +the quoted test command is too late because the isolated home already exists. + +This is the narrow trusted-maintainer-code exception: it stages only the Blacksmith +credential file into the temporary home so the command can delegate remotely. Never +use this credential-hydrated path for untrusted contributor or fork code. Run other +secret-bearing or credentialed tests separately in an appropriately isolated remote +runner. Tradeoff: tests may force code changes that stale the review. If tests or review lead to code edits, rerun the affected tests and rerun review until no accepted/actionable findings remain. Once that rerun exits cleanly, stop; do not spend another long review cycle on redundant confirmation. @@ -151,113 +256,126 @@ Tradeoff: tests may force code changes that stale the review. If tests or review Run multiple reviewers against one frozen bundle: ```bash -"$AUTOREVIEW" --reviewers codex,claude,pi,opencode -``` - -Named presets are available for common closeout shapes: - -```bash -"$AUTOREVIEW" --preset claude-opus +"$AUTOREVIEW" --reviewers codex,claude,pi ``` -`claude-opus` runs the Claude reviewer pinned to `claude-opus-4-8`. Normal standalone `autoreview` still defaults to Codex. Run both commands when a loop requires double review. If Claude Opus is unavailable after the normal same-command retries, treat the second review as blocked rather than accepting a Codex-only result. - `--panel` is shorthand for Codex plus Claude unless `--engine` changes the first reviewer: ```bash "$AUTOREVIEW" --panel ``` -Set reviewer models and thinking/effort explicitly when needed: +Set reviewer models and thinking/effort explicitly: ```bash -"$AUTOREVIEW" --reviewers codex,claude --model codex= --thinking codex=high --model claude= --thinking claude=max +"$AUTOREVIEW" --reviewers codex,claude --model codex=gpt-5.6-sol --thinking codex=high --model claude=claude-fable-5 --thinking claude=max ``` Inline syntax is also supported for simple model IDs: ```bash -"$AUTOREVIEW" --reviewers codex::high,claude::max +"$AUTOREVIEW" --reviewers codex:gpt-5.6-sol:high,claude:claude-fable-5:max ``` For models with slashes or extra colons, prefer keyed form: ```bash "$AUTOREVIEW" --engine pi --model anthropic/claude-sonnet-4 --thinking high -"$AUTOREVIEW" --engine opencode --model opencode/north-mini-code-free --thinking high -"$AUTOREVIEW" --reviewers codex,pi --model codex= --model pi=anthropic/claude-sonnet-4 -"$AUTOREVIEW" --reviewers codex,opencode --model codex= --model opencode=opencode/north-mini-code-free +"$AUTOREVIEW" --reviewers codex,pi --model codex=gpt-5.6-sol --model pi=anthropic/claude-sonnet-4 ``` +`--reviewers all` covers Codex, Claude, and Pi. Droid, Copilot, Cursor, and OpenCode selections fail closed because their current CLI contracts cannot confine project instructions, filesystem reads, or network fetches to the review boundary. + ## Models and thinking The helper accepts `--model` globally or per engine (`engine=model`) and `--thinking` globally or per engine (`engine=level`). Repeat either flag for multiple reviewers. -Model defaults: +Recommended model defaults: -- Codex and Claude use their local CLI/configured model by default. -- CLI flags and environment variables override those defaults. -- Copilot, Pi, and OpenCode also use their provider defaults unless a model is specified. -- Droid is disabled in Bram's copy until a verified read-only isolation flag set is configured. +| Engine | Default model | Source note | +| ------------------- | -------------------------------------------------- | ----------------------------------------------------- | +| **codex** (default) | `gpt-5.6-sol` -> `gpt-5.6-terra` on access failure | OpenClaw org review default | +| **claude** | `claude-fable-5` | Anthropic's most capable widely released Claude model | -| Engine | Model flag | Example model IDs | Thinking flag | Accepted levels | -|--------|------------|-------------------|---------------|-----------------| -| **codex** (default) | `codex --model X exec ...` | local Codex model IDs | `-c model_reasoning_effort=Y` | `none`, `minimal`, `low`, `medium`, `high`, `xhigh` | -| **claude** | `claude --model X` | local Claude model IDs | `--effort Y` | `low`, `medium`, `high`, `xhigh`, `max` | -| **copilot** | `copilot --model X` | `gpt-5.2`, Copilot model aliases | not supported | n/a | -| **pi** | `pi --model X` | `anthropic/claude-sonnet-4`, `openai/gpt-4o` | `--thinking Y` | `off`, `minimal`, `low`, `medium`, `high`, `xhigh` | -| **opencode** | `opencode run -m X` | `opencode/north-mini-code-free`, OpenCode provider/model IDs | `--variant Y` | `minimal`, `low`, `medium`, `high`, `max` | +CLI flags and environment variables override these defaults. Pi does not get a built-in model default because its provider catalog may vary by installation. Droid, Copilot, Cursor, and OpenCode are currently refused. + +| Engine | Model flag | Example model IDs | Thinking flag | Accepted levels | +| ------------------- | -------------------------- | ---------------------------------------------------------------------------- | ----------------------------- | ---------------------------------------------------------- | +| **codex** (default) | `codex --model X exec ...` | `gpt-5.6-sol`, then `gpt-5.6-terra` on Sol access failure | `-c model_reasoning_effort=Y` | `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max` | +| **claude** | `claude --model X` | `claude-fable-5`, `claude-opus-4-8`, `claude-sonnet-4-6`, `claude-haiku-4-5` | `--effort Y` | `low`, `medium`, `high`, `xhigh`, `max` | +| **droid** | currently refused | Factory model IDs | `-r, --reasoning-effort Y` | `off`, `none`, `low`, `medium`, `high`, `xhigh`, `max` | +| **copilot** | currently refused | Copilot model aliases | not supported | n/a | +| **pi** | `pi --model X` | `anthropic/claude-sonnet-4`, `openai/gpt-4o` | `--thinking Y` | `off`, `minimal`, `low`, `medium`, `high`, `xhigh` | +| **cursor** | currently refused | Cursor model aliases | not supported | n/a | +| **opencode** | currently refused | OpenCode provider/model IDs | not supported | n/a | Claude also supports `--fallback-model a,b` for availability-based fallback chains ([model-config](https://code.claude.com/docs/en/model-config)). Current Claude docs note that auth, billing, rate-limit, request-size, and transport errors do not trigger fallback, and the changelog documents interactive-session support in `v2.1.166`. -Examples: +[OpenAI's model guidance](https://developers.openai.com/api/docs/guides/latest-model) identifies Sol as the GPT-5.6 frontier-capability route and documents `max` support. Autoreview keeps `high` as its default; use `max` only for the hardest quality-first reviews after comparing its latency and cost with `xhigh` on representative changes. + +Examples matching current `main` behavior: ```bash # Codex with explicit model and reasoning -"$AUTOREVIEW" --engine codex --model --thinking high +"$AUTOREVIEW" --engine codex --model gpt-5.6-sol --thinking high -# Claude Code aliases or full model names, with optional availability fallback -"$AUTOREVIEW" --engine claude --model --thinking max -"$AUTOREVIEW" --engine claude --model --fallback-model +# Codex fast mode (priority service tier); needs a model whose catalog lists the tier, silently standard otherwise +"$AUTOREVIEW" --engine codex --codex-speed fast + +# Safe Codex model/response tuning overrides (--codex-speed wins over a service_tier here) +"$AUTOREVIEW" --engine codex --codex-config 'service_tier="fast"' -# GitHub Copilot (model only; no thinking knob) -"$AUTOREVIEW" --engine copilot --model gpt-5.2 +# Claude Code aliases or full model names, with optional availability fallback +"$AUTOREVIEW" --engine claude --model claude-fable-5 --thinking max +"$AUTOREVIEW" --engine claude --model claude-fable-5 --fallback-model claude-opus-4-8,claude-sonnet-4-6 # Pi with explicit model and thinking level "$AUTOREVIEW" --engine pi --model anthropic/claude-sonnet-4 --thinking high --pi-bin pi -# OpenCode with explicit provider/model and variant -"$AUTOREVIEW" --engine opencode --model opencode/north-mini-code-free --thinking high ``` +`--cursor-agent-bin` and `CURSOR_AGENT_BIN` remain compatibility aliases for +`--cursor-bin` and `CURSOR_BIN`. + ### Environment defaults CLI flags take precedence over environment variables. -| Variable | Purpose | -|----------|---------| -| `AUTOREVIEW_MODEL` | Set `--model` for all engines | -| `AUTOREVIEW_THINKING` | Default `--thinking` for all engines | -| `AUTOREVIEW_FALLBACK_MODEL` | Default Claude `--fallback-model` chain | -| `AUTOREVIEW__MODEL` | Per-engine model override | -| `AUTOREVIEW__THINKING` | Per-engine thinking override | -| `AUTOREVIEW_CLAUDE_FALLBACK_MODEL` | Claude-only fallback chain | -| `AUTOREVIEW_PRESET` | Reviewer preset such as `claude-opus` | +Store persistent personal defaults in your shell startup file or launcher +environment. For repository-local defaults, use an existing local environment +loader such as an untracked `.envrc`; the helper does not write a config file. -Codex maps thinking to `model_reasoning_effort`. Claude maps thinking to `--effort`. Pi maps thinking to `--thinking`. OpenCode maps thinking to `--variant`. Copilot rejects `--thinking`. Only Claude accepts `--fallback-model`; global CLI/env fallback requires at least one Claude reviewer, and engine-specific fallback overrides require that reviewer to be selected. Non-Claude fallback overrides, including `AUTOREVIEW__FALLBACK_MODEL`, fail closed instead of being silently ignored. +| Variable | Purpose | +| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| `AUTOREVIEW_MODEL` | Override the built-in default `--model` for all engines | +| `AUTOREVIEW_THINKING` | Default `--thinking` for all engines | +| `AUTOREVIEW_FALLBACK_MODEL` | Default Claude `--fallback-model` chain | +| `AUTOREVIEW__MODEL` | Per-engine model override, for example `AUTOREVIEW_CODEX_MODEL=gpt-5.6-sol` | +| `AUTOREVIEW__THINKING` | Per-engine thinking override | +| `AUTOREVIEW_CODEX_CONFIG` | Safe Codex model/response tuning overrides, semicolon-separated, e.g. `service_tier="fast"`; capability-bearing keys fail closed | +| `AUTOREVIEW_CODEX_SPEED` | Codex service tier override: `fast` (priority), `flex`, or `default`; silently standard when the model does not list the tier | +| `AUTOREVIEW_CLAUDE_FALLBACK_MODEL` | Claude-only fallback chain | +| `AUTOREVIEW_PROVIDER_ENV_ALLOW` | Comma-separated custom Pi/OpenCode credential variable names; names must end in a recognized credential suffix | + +Codex maps thinking to `model_reasoning_effort`. Claude maps thinking to `--effort`. Pi maps thinking to `--thinking`. Only Claude accepts `--fallback-model`; global CLI/env fallback requires at least one Claude reviewer, and engine-specific fallback overrides require that reviewer to be selected. Non-Claude fallback overrides, including `AUTOREVIEW__FALLBACK_MODEL`, fail closed instead of being silently ignored. ## Review engine isolation When autoreview runs inside the repository under review, external reviewer CLIs must not load project-local trust or configuration that the branch controls. -| Engine | Isolation flags | Reference | -|--------|-----------------|-----------| -| **codex** | `-c project_doc_max_bytes=0`, repo `trust_level="untrusted"`, `exec --ignore-rules`, plus read-only sandbox | Codex CLI `exec --help` | -| **claude** | `--safe-mode --setting-sources user --strict-mcp-config --disallowedTools mcp__*` plus explicit `--allowedTools` (`--safe-mode` requires Claude Code `v2.1.169+`) | Claude Code [CLI reference](https://code.claude.com/docs/en/cli-reference) | -| **pi** | `--no-approve --no-session --no-context-files --no-extensions --no-skills --no-prompt-templates --no-themes`, plus read-only tool allowlist | Pi CLI `--help`; requires Pi `v0.79.0+` | -| **opencode** | `opencode run --dir --pure --format json`, prompt over stdin, neutral subprocess cwd, injected deny-by-default permissions, project config disabled | OpenCode CLI `--help` | +| Engine | Isolation flags | Reference | +| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------- | +| **codex** | Auth-only config overrides, isolated workspace, `exec --ignore-user-config --ignore-rules --skip-git-repo-check`, plus read-only sandbox | Codex CLI `exec --help` | +| **claude** | `--safe-mode --setting-sources user --strict-mcp-config --disallowedTools mcp__*`; auto-memory and filesystem/shell tools disabled; empty external workspace; WebSearch by default (`v2.1.169+`) | Claude Code [CLI reference](https://code.claude.com/docs/en/cli-reference) | +| **droid** | Fails closed: current CLI cannot disable both project instructions and all tools | Droid CLI `exec --help` and `--list-tools` | +| **copilot** | Fails closed: repository read tools also expose ignored files outside the reviewed bundle | GitHub Copilot CLI command reference | +| **pi** | `--no-approve --no-session --no-context-files --no-extensions --no-skills --no-prompt-templates --no-themes --no-tools` | Pi CLI `--help`; requires Pi `v0.79.0+` | +| **opencode** | Fails closed: project/global config isolation and private-network fetch denial are not both proven | OpenCode CLI contract | +| **cursor** | Fails closed: documented read permissions can target absolute host paths and no proven repository-only filesystem sandbox is exposed | Cursor CLI [permissions](https://cursor.com/docs/cli/reference/permissions) | + +Codex `--ignore-user-config` skips config loading for the exec run. Autoreview reconstructs only the documented `cli_auth_credentials_store`, `forced_login_method`, and `forced_chatgpt_workspace_id` settings from `CODEX_HOME/config.toml`, keeping authentication usable without forwarding unrelated user configuration. Codex runs in an empty temporary workspace: the validated bundle is its sole repository input, ignored files and linked-worktree metadata remain unreadable, and the zero project-doc budget keeps workspace instructions out of the prompt. `--ignore-rules` skips user/project execpolicy rules. Claude `--safe-mode` disables project hooks, skills, plugins, MCP servers, and CLAUDE.md; autoreview supplies WebSearch by default, permits only explicitly domain-constrained WebFetch rules, and exposes no filesystem or shell tools. Pi runs from a neutral temporary directory with project resources disabled and `--no-tools`. Droid, Copilot, Cursor, and OpenCode fail closed because their current CLI contracts cannot isolate untrusted review input from host, project, or private-network trust surfaces. -The explicit repo trust override and zero project-doc budget keep reviewed-repo `AGENTS.md` and `.codex/` trust surfaces out of the review prompt while preserving the local Codex CLI's configured model/provider. `--ignore-rules` skips user/project execpolicy rules. Claude `--safe-mode` disables project hooks, skills, plugins, MCP servers, and CLAUDE.md while preserving normal authentication, model selection, built-in tools, and permissions; managed settings policy can still apply. `--setting-sources user` avoids project/local settings from the reviewed checkout, and current Claude Code docs note the project-skill blocking behavior was fixed in `v2.1.69`. `--strict-mcp-config` and `--disallowedTools mcp__*` keep MCP unavailable to the review run. `--bare` is not used here because Claude's headless docs say it skips OAuth and keychain reads. Pi `--no-approve` ignores project-local files for one run; the helper requires Pi `v0.79.0+` plus help output that advertises every required isolation flag because older legacy binaries can ignore unknown flags. The current package is `@earendil-works/pi-coding-agent`; deprecated `@mariozechner/pi-coding-agent` `0.73.x` is intentionally rejected. Pi version/help probes and the review command run from neutral temporary directories, not the reviewed repo. Pi `--no-context-files` removes `AGENTS.md`/`CLAUDE.md`, the resource-disable flags keep `.pi` extensions, skills, prompts, and themes out of the run, `--no-session` avoids writing review sessions, and the read-only allowlist omits `bash`, `edit`, and `write`. OpenCode starts from a neutral temporary directory, points at the reviewed repo with `--dir`, disables project config through `OPENCODE_DISABLE_PROJECT_CONFIG=1`, and injects `OPENCODE_CONFIG_CONTENT`; permissions default to deny, allow read/grep/glob, preserve OpenCode's `.env` ask rules, and gate `websearch`/`webfetch` with `--no-web-search`. The injected config also clears command/instruction/plugin arrays and disables write/edit/bash/task/skill/todowrite tools without changing user auth storage. The helper sends the review prompt over stdin rather than argv and extracts the final structured JSON from `type: "text"` events. OpenCode rejects `--no-tools`. +Codex uses a named permission profile that grants read access only to an empty temporary workspace. This is narrower than repository-root access, which would expose ignored credentials, and narrower than the legacy `read-only` sandbox, which permits reads across the host filesystem. ## Context Efficiency @@ -280,13 +398,13 @@ The smoke harness has thin shell wrappers over a shared Python implementation: On native Windows, invoke the extensionless Python helper through Python: ```powershell -python skills\autoreview\scripts\autoreview --help +python $AUTOREVIEW --help ``` and the smoke harness: ```powershell -skills\autoreview\scripts\test-review-harness.ps1 -Fixture benign -Engine codex +& $AUTOREVIEW_HARNESS -Fixture benign -Engine codex ``` The helper: @@ -295,22 +413,21 @@ The helper: - accepts `--mode uncommitted` as an alias for `--mode local` - otherwise uses current PR base if `gh pr view` works - otherwise uses `origin/main` for non-main branches -- supports `--engine codex`, `claude`, `copilot`, `pi`, and `opencode`; default is `AUTOREVIEW_ENGINE` or `codex`; Codex should remain the default when nothing is set -- resolves bare `git`, `gh`, reviewer, and PowerShell shell commands from absolute `PATH` entries only, never from the reviewed checkout; explicit relative `--*-bin` paths are resolved from the reviewed repository root +- does not fetch automatically during branch review; the selected base ref must already resolve locally +- recognizes `--engine droid`, `copilot`, `cursor`, and `opencode` only to fail closed with isolation errors; runnable engines are `codex`, `claude`, and `pi`; default is `AUTOREVIEW_ENGINE` or `codex` +- resolves bare `git`, `gh`, reviewer, and PowerShell shell commands from absolute `PATH` entries only, never from the reviewed checkout; explicit `--*-bin` paths are interpreted from the reviewed repository root when relative and accepted only when both the supplied path and resolved target stay outside the reviewed repository - use `--mode commit --commit ` for already-committed work, especially clean `main` after landing +- scans safe Git patches in full, recognizes synthetic fixture values tied to their credential field, reviews them in one pass up to the aggregate prompt limit, and automatically uses complete bounded passes above it - should be left in `--mode auto` or forced to `--mode branch` for PR/branch work; do not force `--mode local` after committing - writes only to stdout unless `--output`, `--json-output`, or live streamed engine stderr is set -- supports `--dry-run`, `--parallel-tests`, `--parallel-tests-shell`, `--prompt`, `--prompt-file`, `--dataset`, `--no-tools`, `--no-web-search`, `--include-risky-paths`, and commit refs -- fails closed when reviewed diffs or included untracked files exceed the bundle cap; narrow the review target instead of accepting a truncated clean result +- supports `--dry-run`, `--parallel-tests`, `--parallel-tests-shell`, `--prompt`, repo-relative `--prompt-file`, repo-relative `--dataset`, `--no-tools`, `--no-web-search`, repeatable Codex-only safe model/response tuning with `--codex-config key=value`, Codex-only `--codex-speed fast|flex|default`, and commit refs - supports `--stream-engine-output` or `AUTOREVIEW_STREAM_ENGINE_OUTPUT=1` for live engine text while preserving structured validation; Codex and Claude hide tool/file event details, emit compact activity summaries, and report usage at turn completion - supports opt-in review panels with `--panel` / `--reviewers`, plus per-engine `--model`, `--thinking`, and Claude `--fallback-model` -- supports `--preset claude-opus`; run the default command separately when both Codex and Claude reviews are required -- uses each reviewer CLI's configured model by default; honors `AUTOREVIEW_MODEL`, `AUTOREVIEW_THINKING`, `AUTOREVIEW_FALLBACK_MODEL`, and per-engine `AUTOREVIEW__MODEL` / `AUTOREVIEW__THINKING` environment overrides when CLI flags are omitted -- allows read-only tools and web search by default where the selected CLI supports them; forbids nested review in the prompt; Codex is run through `codex exec` with read-only sandbox, reviewed-repo instruction/config/rule isolation flags, and structured output -- runs Claude with `--safe-mode` (`v2.1.169+`), `--setting-sources user`, MCP disabled, explicit allowed tools, and `--fallback-model` when set, so reviewed-repo hooks/skills/MCP do not affect the review run while normal auth still works; managed settings policy can still apply -- fails closed for Droid until a verified read-only isolation flag set is configured -- runs Pi `v0.79.0+` from neutral temporary directories with `--no-approve`, `--no-session`, disabled Pi context/resource loading, and built-in read-only tools (`read,grep,find,ls`) when tools are enabled -- runs OpenCode with `opencode run --dir --pure --format json` from a neutral temporary directory, forwards `--model` and `--variant`, injects deny-by-default permissions, disables project config loading, and passes the review prompt over stdin +- uses built-in defaults `codex=gpt-5.6-sol` with `high` reasoning and an access-only `gpt-5.6-terra` retry, plus `claude=claude-fable-5`; honors `AUTOREVIEW_MODEL`, `AUTOREVIEW_THINKING`, `AUTOREVIEW_FALLBACK_MODEL`, and per-engine `AUTOREVIEW__MODEL` / `AUTOREVIEW__THINKING` environment overrides when CLI flags are omitted +- gives Codex the bundle in an empty workspace with web search available; Claude receives the bundle plus WebSearch by default and optional domain-constrained WebFetch, and Pi receives the bundle with no tools +- runs Claude with `--safe-mode` (`v2.1.169+`), `--setting-sources user`, MCP and auto-memory disabled, no filesystem/shell tools, an empty external workspace, and `--fallback-model` when set +- refuses Droid, Copilot, Cursor, and OpenCode reviews until their CLIs expose the required project, filesystem, and network isolation +- runs Pi `v0.79.0+` from neutral temporary directories with `--no-approve`, `--no-session`, disabled Pi context/resource loading, and `--no-tools` because its built-in read tools are not repository-confined - prints `review still running: elapsed=s pid=` to stderr at long-running intervals while waiting for the selected review engine, unless streamed output or compact Codex activity has been visible recently - prints `autoreview clean: no accepted/actionable findings reported` when the selected review command exits 0 - exits nonzero when accepted/actionable findings are present diff --git a/skills/autoreview/scripts/autoreview b/skills/autoreview/scripts/autoreview index c4b7cb2..4aae660 100755 --- a/skills/autoreview/scripts/autoreview +++ b/skills/autoreview/scripts/autoreview @@ -2,43 +2,809 @@ from __future__ import annotations import argparse +import ast +import base64 +import binascii +import bisect import concurrent.futures import copy +import functools +import hashlib +import io import json +import math import os import queue import re +import shutil +import stat import subprocess import sys import tempfile import textwrap import threading import time -from pathlib import Path -from typing import Any, Callable - - -ENGINES = ("codex", "claude", "droid", "copilot", "pi", "opencode") +import unicodedata +import urllib.parse +from pathlib import Path, PurePosixPath +from typing import Any, Callable, NamedTuple + + +ENGINES = ("codex", "claude", "droid", "copilot", "pi", "opencode", "cursor") +ENGINE_ALIASES = {"cursor-agent": "cursor"} +ENGINE_CHOICES = (*ENGINES, *ENGINE_ALIASES) +ALL_REVIEWERS = ("codex", "claude", "pi") +SAFE_GIT_CONFIG_ARGS = ( + "-c", + "core.fsmonitor=false", + "-c", + "core.pager=cat", + "-c", + "diff.external=", + "-c", + "diff.renames=false", + "-c", + "pager.diff=cat", + "-c", + "pager.log=cat", + "-c", + "pager.show=cat", +) +SAFE_DIFF_FLAGS = ("--no-ext-diff", "--no-textconv", "--no-renames") +DIFF_HUNK_CONTENT_BOUNDARY = "\0autoreview-diff-hunk-boundary\0" +ENGINE_GIT_CONFIG_OVERRIDES = ( + ("core.fsmonitor", "false"), + ("core.pager", "cat"), + ("diff.external", ""), + ("diff.renames", "false"), + ("pager.diff", "cat"), + ("pager.log", "cat"), + ("pager.show", "cat"), +) +SENSITIVE_PATH_PARTS = { + ".aws", + ".azure", + ".config/gcloud", + ".docker", + ".gnupg", + ".ssh", + "private", +} +TRACKED_SENSITIVE_PATH_PARTS = SENSITIVE_PATH_PARTS - { + "private", + ".docker", +} +TRACKED_CREDENTIAL_DIR_PATTERN = re.compile( + r"^(?:.*[._-])?" + r"(secret|secrets|credential|credentials|service[-_]?account|private[-_]?key|api[-_]?key)" + r"(?:[._-].*)?$", + re.IGNORECASE, +) +CREDENTIAL_FILE_PATTERN = re.compile( + r"(^|/)(?:\.netrc|\.git-credentials)$", + re.IGNORECASE, +) +SENSITIVE_NAME_PATTERNS = [ + CREDENTIAL_FILE_PATTERN, + re.compile(r"(^|/)\.env($|[._/-])", re.IGNORECASE), + re.compile(r"(^|/)(id_rsa|id_dsa|id_ecdsa|id_ed25519)(\.pub)?$", re.IGNORECASE), + re.compile(r"\.(pem|p12|pfx|key)$", re.IGNORECASE), + re.compile( + r"(^|/)[^/]*(secret|token|credential|credentials|service[-_]?account|private[-_]?key|apikey|api[-_]?key)[^/]*$", + re.IGNORECASE, + ), +] +TRACKED_SENSITIVE_NAME_PATTERNS = [ + CREDENTIAL_FILE_PATTERN, + re.compile( + r"(^|/)\.env(?:$|/|[._-](?!(?:example|sample|template)$)[^/]*)", + re.IGNORECASE, + ), + re.compile(r"(^|/)(id_rsa|id_dsa|id_ecdsa|id_ed25519)(\.pub)?$", re.IGNORECASE), + re.compile(r"\.(pem|p12|pfx|key)$", re.IGNORECASE), + re.compile( + r"(^|/)(secret|secrets|credential|credentials|service[-_]?account|private[-_]?key|api[-_]?key|token|tokens)$", + re.IGNORECASE, + ), + re.compile( + r"(^|/)(?:[^/]*[._-])?" + r"(secret|secrets|credential|credentials|service[-_]?account|private[-_]?key|api[-_]?key|token|tokens)" + r"(?:[._-][^/]*)?\.(json|ya?ml|toml|ini|conf|config|txt|csv)$", + re.IGNORECASE, + ), +] +TRACKED_TOKEN_CREDENTIAL_STEMS = { + "access", + "account", + "auth", + "cache", + "credentials", + "credential", + "device", + "id", + "prod", + "production", + "refresh", + "secret", + "secrets", + "session", + "store", + "token", + "tokens", + "user", +} +TRACKED_TOKEN_CREDENTIAL_EXTENSIONS = { + "", + ".conf", + ".config", + ".csv", + ".dat", + ".db", + ".enc", + ".ini", + ".json", + ".jsonl", + ".jwt", + ".sqlite", + ".sqlite3", + ".txt", + ".toml", + ".yaml", + ".yml", +} +SECRET_KEY_NAME_PATTERN = ( + r"(?:api[_-]?key|aws[_-]?secret[_-]?access[_-]?key" + r"|client[_-]?secret|refresh[_-]?token|access[_-]?token" + r"|auth[_-]?token|id[_-]?token|token|secret|password" + r"|credentials?|private[_-]?key)" +) +SECRET_SEPARATED_KEY_NAME_PATTERN = ( + rf"(?:[A-Za-z0-9]{{1,64}}" + rf"(?:[_-][A-Za-z0-9]{{1,64}}){{0,15}}[_-]" + rf"{SECRET_KEY_NAME_PATTERN})" +) +SECRET_LOWER_KEY_NAME_PATTERN = ( + r"(?-i:[a-z][a-z0-9]*" + r"(?:apikey|awssecretaccesskey|clientsecret|refreshtoken" + r"|accesstoken|authtoken|idtoken|token|secret|password" + r"|credential|credentials|privatekey))" +) +SECRET_CAMEL_KEY_NAME_PATTERN = ( + r"(?-i:[A-Za-z][A-Za-z0-9]*" + r"(?:ApiKey|APIKey|AwsSecretAccessKey|AWSSecretAccessKey" + r"|ClientSecret|RefreshToken|AccessToken|AuthToken|IdToken|IDToken" + r"|Token|Secret|Password" + r"|Credential|Credentials|PrivateKey))" +) +SECRET_UPPER_KEY_NAME_PATTERN = ( + r"(?-i:[A-Z][A-Z0-9]*" + r"(?:APIKEY|AWSSECRETACCESSKEY|CLIENTSECRET|REFRESHTOKEN" + r"|ACCESSTOKEN|AUTHTOKEN|IDTOKEN|TOKEN|SECRET|PASSWORD" + r"|CREDENTIAL|CREDENTIALS|PRIVATEKEY))" +) +SECRET_ASSIGNMENT_KEY_NAME_PATTERN = ( + rf"(?:{SECRET_SEPARATED_KEY_NAME_PATTERN}" + rf"|{SECRET_LOWER_KEY_NAME_PATTERN}" + rf"|{SECRET_CAMEL_KEY_NAME_PATTERN}" + rf"|{SECRET_UPPER_KEY_NAME_PATTERN}" + rf"|{SECRET_KEY_NAME_PATTERN})" +) +SECRET_ASSIGNMENT_KEY_PATTERN = ( + rf"(?:[\"']{SECRET_ASSIGNMENT_KEY_NAME_PATTERN}[\"']" + rf"|(?[^\"\r\n]{8,})\"|" + r"'(?P[^'\r\n]{8,})'|" + r"`(?P[^`\r\n]{8,})`|" + r"(?P[A-Za-z_$][A-Za-z0-9_$]*" + r"(?:(?:\?\.|\.)[A-Za-z_$][A-Za-z0-9_$]*)*)(?=[ \t]*\()|" + r"(?P[A-Za-z_$][A-Za-z0-9_$]*" + r"(?:(?:\?\.|\.)[A-Za-z_$][A-Za-z0-9_$]*" + r"|(?:\?\.)?\[(?:[A-Za-z_$][A-Za-z0-9_$]*" + r"|[\"'][A-Za-z_$][A-Za-z0-9_$]*[\"']|[0-9]+)\])+)" + r"(?![A-Za-z0-9_./+=:@#$%&*!?-])|" + r"(?P[A-Za-z0-9_./+=:@#$%&*!?-]{8,}))" +) +SECRET_ASSIGNMENT_PREFIX_PATTERN = re.compile( + rf"(?i){SECRET_ASSIGNMENT_KEY_PATTERN}" + r"\s*(?:=(?!=|>)|:(?![:=]))\s*" +) +SECRET_BOOLEAN_DECLARATION_PATTERN = re.compile( + r"(?im)^[ \t]*(?:(?:abstract|const|declare|export|final|internal|lateinit|" + r"open|override|private|protected|public|readonly|static)\s+)*" + rf"(?:val|var|let|const)\s+" + rf"(?P{SECRET_ASSIGNMENT_KEY_PATTERN})\s*:\s*" + rf"(?P{SECRET_BOOLEAN_TYPE_PATTERN})" + r"(?=\s*(?:=|[,;)\]}]|$))" +) +PRIVATE_KEY_BOUNDARY_PREFIX = r"-----" +PRIVATE_KEY_BEGIN_PATTERN = re.compile( + PRIVATE_KEY_BOUNDARY_PREFIX + + r"BEGIN (?:RSA |DSA |EC |OPENSSH |PGP |ENCRYPTED )?" + r"PRIVATE KEY(?: BLOCK)?-----" +) +PRIVATE_KEY_END_PATTERN = re.compile( + PRIVATE_KEY_BOUNDARY_PREFIX + + r"END (?:RSA |DSA |EC |OPENSSH |PGP |ENCRYPTED )?" + r"PRIVATE KEY(?: BLOCK)?-----" +) +PRIVATE_KEY_BODY_PATTERN = re.compile( + r"(?" + r"(?:[A-Za-z0-9+/]{4,}={0,2}|[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)" + r")(?![A-Za-z0-9+/=])" +) +BEARER_CREDENTIAL_PATTERN = re.compile( + r"(?i)bearer\s+(?P[A-Za-z0-9._~+/-]{20,}=*)" +) +SECRET_FALLBACK_LITERAL_PATTERNS = ( + re.compile(r'"(?P[^"\r\n]+)"'), + re.compile(r"'(?P[^'\r\n]+)'"), + re.compile(r"`(?P[^`\r\n]+)`"), +) +SECRET_FALLBACK_UNQUOTED_PATTERN = re.compile( + r"(?[A-Za-z0-9_./+=:@#$%&*!?-]{4,})" + r"(?![A-Za-z0-9_./+=:@#$%&*!?-])" +) +CONFIG_PATH_SEGMENT_PATTERN = ( + r"(?:[a-z][A-Za-z0-9]{0,63}|[a-z][a-z0-9]*(?:-[a-z0-9]+)+" + r"|\$\{[A-Za-z_$][A-Za-z0-9_$]*(?:\.[A-Za-z_$][A-Za-z0-9_$]*)*\})" +) +CANONICAL_CONFIG_PATH_REFERENCE_PATTERN = re.compile( + rf"{CONFIG_PATH_SEGMENT_PATTERN}(?:\.{CONFIG_PATH_SEGMENT_PATTERN}){{2,}}" + r"\.[a-z][A-Za-z0-9]{0,63}" +) +CONFIG_PATH_CREDENTIAL_FIELD_PATTERN = re.compile( + r"(?:apiKey|password|Password|secret|Secret|token|Token|credential|Credential" + r"|keyRef|KeyRef|privateKey|PrivateKey|serviceAccount)" +) +SECRET_VALUE_PATTERNS = [ + PRIVATE_KEY_BEGIN_PATTERN, + BEARER_CREDENTIAL_PATTERN, + re.compile(r"\b(?:sk|rk|pk|org|proj)-[A-Za-z0-9_-]{20,}\b"), + re.compile(r"\bgithub_pat_[A-Za-z0-9_]{20,}\b"), + re.compile(r"\bgh[pousr]_[A-Za-z0-9_]{20,}\b"), + re.compile(r"\bglpat-[A-Za-z0-9_-]{20,}\b"), + re.compile(r"\bnpm_[A-Za-z0-9]{20,}\b"), + re.compile(r"\bxox[baprs]-[A-Za-z0-9-]{20,}\b"), + re.compile(r"\b(?:A3T|AKIA|ASIA)[A-Z0-9]{16}\b"), + re.compile(r"\bAIza[0-9A-Za-z_-]{35}\b"), + re.compile(r"\bya29\.[0-9A-Za-z_-]{20,}\b"), + re.compile(r"\beyJ[A-Za-z0-9_-]{7,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b"), +] +BASIC_AUTHORIZATION_PATTERN = re.compile( + r"(?i)(?:^|[^A-Za-z0-9_])[\"']?authorization[\"']?" + r"\s*[:=]\s*[\"']?" + r"basic\s+(?P[A-Za-z0-9+/]{8,}={0,2})" + r"(?![A-Za-z0-9+/=])" +) +URI_SCHEME_PATTERN = re.compile( + r"\b[A-Za-z][A-Za-z0-9+.-]*:(?:\\?/){2}", + re.IGNORECASE, +) +URI_PASSWORD_REFERENCE_PATTERNS = ( + re.compile(r"^\$[A-Za-z_][A-Za-z0-9_]*$"), + re.compile(r"^\$\{[A-Za-z_][A-Za-z0-9_]*\}$"), + re.compile(r"^\{[A-Za-z_][A-Za-z0-9_]*\}$"), + re.compile( + r"^\$\{[A-Za-z_$][A-Za-z0-9_$]*" + r"(?:(?:\?\.|\.)[A-Za-z_$][A-Za-z0-9_$]*" + r"|\[(?:[0-9]+|[\"'][A-Za-z_$][A-Za-z0-9_$]*[\"'])\])+\}$" + ), + re.compile( + r"^\{[A-Za-z_$][A-Za-z0-9_$]*" + r"(?:(?:\?\.|\.)[A-Za-z_$][A-Za-z0-9_$]*" + r"|\[(?:[0-9]+|[\"'][A-Za-z_$][A-Za-z0-9_$]*[\"'])\])+\}$" + ), +) +URI_CREDENTIAL_REFERENCE_TEXT = ( + r"[A-Za-z_$][A-Za-z0-9_$]*" + r"(?:(?:\?\.|\.)[A-Za-z_$][A-Za-z0-9_$]*" + r"|\[(?:[0-9]+|[\"'][A-Za-z_$][A-Za-z0-9_$]*[\"'])\])*" +) +URI_CREDENTIAL_REFERENCE_PATTERN = re.compile( + rf"^{URI_CREDENTIAL_REFERENCE_TEXT}$" +) +URI_COMPUTED_REFERENCE_PATTERN = re.compile( + rf"^{URI_CREDENTIAL_REFERENCE_TEXT}" + rf"\(\s*{URI_CREDENTIAL_REFERENCE_TEXT}\s*\)$" +) +POWERSHELL_ENV_REFERENCE_PATTERN = re.compile( + r"^\$env:[A-Za-z_][A-Za-z0-9_]*$", + re.IGNORECASE, +) +MAX_BUNDLE_TEXT_BYTES = 180_000 +MAX_REVIEW_PROMPT_BYTES = 512_000 +MAX_REVIEW_CHUNK_CONTEXT_BYTES = 64_000 +MAX_REVIEW_PASSES = 8 + + +class ReviewChunk(NamedTuple): + content: str + context: str = "" +SECRET_PLACEHOLDER_VALUES = { + "__openclaw_redacted__", + "changeme", + "decoy-token", + "dummy", + "example", + "fake", + "gateway-token", + "matrix_qa_e2ee_cli_gateway", + "matrix_qa_e2ee_thread", + "matrix_qa_e2ee_verify_notice", + "not-a-real", + "not-a-valid-matrix-recovery-key", + "placeholder", + "redacted", + "sample", + "secret-token", + "test-auth-token", + "test-key", + "test-secret", + "test-token", + "test-token-placeholder", + "token-oversized", + "clawrouter-e2e-secret", + "config-token", + "very-long-browser-token-0123456789", +} +SYNTHETIC_SECRET_PREFIXES = frozenset( + {"dummy", "example", "fake", "fixture", "mock", "sample", "test"} +) +FETCH_CREDENTIAL_MODE_VALUES = {"include", "omit", "same-origin"} +URI_PASSWORD_PLACEHOLDER_VALUES = { + "clawrouter-e2e-secret", + "dummy", + "example", + "fake", + "not-a-real", + "placeholder", + "redacted", + "sample", + "test-auth-token", + "test-token-placeholder", + "token-oversized", + "very-long-browser-token-0123456789", +} +URI_CREDENTIAL_NAME_PATTERN = re.compile( + r"(?:api[_-]?key|auth|credential|pass(?:word)?|pwd|secret|token)", + re.IGNORECASE, +) +SHELL_COMMAND_WRAPPERS = {"command", "env", "sudo"} +NON_SHELL_COMMAND_WORDS = { + "assert", + "await", + "case", + "catch", + "class", + "const", + "def", + "else", + "except", + "export", + "finally", + "for", + "from", + "function", + "if", + "import", + "include", + "interface", + "let", + "match", + "new", + "print", + "raise", + "require", + "return", + "switch", + "throw", + "try", + "type", + "var", + "while", + "with", + "yield", +} +PUBLIC_PROMPT_TARGETS = {"getpass.getpass", "input", "prompt"} +GENERIC_CREDENTIAL_PROMPT_PATTERN = re.compile( + r"(?i)\s*(?:(?:enter|type|provide)\s+(?:(?:your|the)\s+)?)?" + r"(?:password|passphrase|api[\s_-]*(?:key|token))" + r"(?:\s+for\s+(?:the\s+)?" + r"(?P[A-Za-z][A-Za-z0-9 _-]{0,48}))?" + r"\s*[:?]?\s*" +) +PROMPT_SECRET_THEME_WORDS = frozenset( + { + "admin", + "autumn", + "fall", + "password", + "secret", + "spring", + "summer", + "vacation", + "welcome", + "winter", + } +) +CSHARP_STANDALONE_REFERENCE_PATTERN = re.compile( + r"(?:credential|credentials|pass|passwd|password|pwd|secret|token)", + re.IGNORECASE, +) +CSHARP_METHOD_MODIFIERS_PATTERN = ( + r"(?:(?:async|extern|internal|new|override|partial|private|protected" + r"|public|sealed|static|unsafe|virtual)\s+)*" +) +CSHARP_ATTRIBUTE_PATTERN = r"(?:\[[^\[\]{};]*\]\s*)*" +CSHARP_TYPE_MODIFIERS_PATTERN = ( + r"(?:(?:abstract|file|internal|new|partial|private|protected|public" + r"|readonly|ref|sealed|static|unsafe)\s+)*" +) +CSHARP_TYPE_PREFIX_PATTERN = ( + rf"{CSHARP_ATTRIBUTE_PATTERN}" + rf"{CSHARP_TYPE_MODIFIERS_PATTERN}" + r"(?:class|interface|namespace|record(?:\s+(?:class|struct))?|struct)\s+" + r"[A-Za-z_][A-Za-z0-9_.]*(?:<[^{};]+>)?[^{;]*\{" +) +CSHARP_RETURN_TYPE_PATTERN = ( + r"(?:(?:ref\s+(?:readonly\s+)?|scoped\s+)?" + r"(?:(?:[A-Za-z_][A-Za-z0-9_]*::)?" + r"[A-Za-z_][A-Za-z0-9_.]*" + r"(?:<[^{};]+>)?" + r"|\([^{};]+\))(?:\?|\*|\[[,\s]*\])*)" +) +CSHARP_METHOD_PREFIX_PATTERN = ( + rf"{CSHARP_ATTRIBUTE_PATTERN}" + rf"{CSHARP_METHOD_MODIFIERS_PATTERN}" + r"(?!function\b)" + rf"{CSHARP_RETURN_TYPE_PATTERN}\s+" + r"[A-Za-z_][A-Za-z0-9_]*(?:<[^(){};]+>)?\s*" + r"\([^{};]*\)\s*" + r"(?:where\s+[^{;]+)?\{" +) +CSHARP_EVIDENCE_WINDOW = 8192 +SOURCE_CODE_REFERENCE_ROOT_VALUES = { + "accountConfig", + "attemptAuthProfileStore", + "baseConfig", + "merged", +} +SOURCE_CODE_REFERENCE_ROOT_PATTERN = re.compile( + r"(?[\"'])op://[^`\"'\r\n]+(?P=op_quote)" + r")$" + ), +) +BACKTICK_TEMPLATE_INTERPOLATION_PATTERN = re.compile(r"\$\{([^{}\r\n]+)\}") +BACKTICK_TEMPLATE_SAFE_LITERAL_PATTERN = re.compile( + r"(?i)(?:(?:Bearer|Basic)[ \t]+|[ \t:./,_-]*)" +) +BACKTICK_TEMPLATE_FIXTURE_PREFIX_VALUES = {"matrix-qa-"} +SOURCE_CODE_REFERENCE_VALUES = { + "cliDevice.accessToken", + "context.driverAccessToken", + "context.driverPassword", + "context.observerAccessToken", + "context.observerPassword", + "context.sutAccessToken", + "context.sutPassword", + "driverAccount.accessToken", + "driverAccount.password", + "driverPassword", + "data.session?.access_token", + "providerConfig.api_key", + "providerConfig.api_secret", + "recoveryDevice.accessToken", + "recoveryDevice.password", + "resolution.value", +} +SOURCE_CODE_REFERENCE_PATTERN = re.compile( + r"(? subprocess.CompletedProcess[str]: +def run( + args: list[str], + cwd: Path, + *, + input_text: str | None = None, + check: bool = True, + env: dict[str, str] | None = None, + text_errors: str = SUBPROCESS_TEXT_ERRORS, +) -> subprocess.CompletedProcess[str]: result = subprocess.run( args, cwd=cwd, input=input_text, text=True, + encoding=SUBPROCESS_TEXT_ENCODING, + errors=text_errors, stdout=subprocess.PIPE, stderr=subprocess.PIPE, + env=env, ) if check and result.returncode != 0: cmd = " ".join(args) @@ -118,24 +895,688 @@ def subprocess_env(extra: dict[str, str] | None) -> dict[str, str] | None: return merged +def safe_git_env(repo: Path) -> dict[str, str]: + platform_keys = ("COMSPEC", "PATHEXT", "SYSTEMROOT", "TEMP", "TMP", "TMPDIR", "WINDIR") + env = { + key: os.environ[key] + for key in platform_keys + if key in os.environ + } + env.update({ + "GIT_CONFIG_GLOBAL": os.devnull, + "GIT_CONFIG_NOSYSTEM": "1", + "GIT_CONFIG_SYSTEM": os.devnull, + "GIT_OPTIONAL_LOCKS": "0", + "GIT_TERMINAL_PROMPT": "0", + "HOME": os.environ.get("HOME", str(Path.home())), + "LANG": "C.UTF-8", + "LC_ALL": "C.UTF-8", + "PATH": safe_engine_path(repo), + }) + return env + + +def global_excludes_file(repo: Path) -> Path | None: + env = safe_git_env(repo) + env.pop("GIT_CONFIG_GLOBAL", None) + home = Path(env["HOME"]).expanduser() + if not external_env_path(repo, str(home)): + return None + result = run( + [ + resolve_command("git", repo), + "--no-optional-locks", + *SAFE_GIT_CONFIG_ARGS, + "config", + "--global", + "--path", + "--get", + "core.excludesFile", + ], + repo, + check=False, + env=env, + ) + if result.returncode != 0: + return None + raw_path = result.stdout.strip() + if not raw_path: + return None + candidate = Path(raw_path).expanduser() + if not candidate.is_absolute(): + candidate = home / candidate + try: + resolved = candidate.resolve(strict=True) + except OSError: + return None + if is_within(resolved, repo.resolve()) or not resolved.is_file(): + return None + return resolved + + +def global_excludes_git_args(repo: Path) -> list[str]: + if excludes_file := global_excludes_file(repo): + return ["-c", f"core.excludesFile={excludes_file}"] + return [] + + +def safe_engine_path(repo: Path, extra_paths: list[Path] | None = None) -> str: + entries: list[str] = [] + resolved_repo = repo.resolve() + + def add(path: str | Path) -> None: + candidate = Path(path).expanduser() + try: + if not candidate.is_absolute() or not candidate.exists(): + return + resolved = candidate.resolve() + except OSError: + return + if is_within(resolved, resolved_repo): + return + value = str(resolved) + if value not in entries: + entries.append(value) + + for path in extra_paths or []: + add(path) + for part in os.environ.get("PATH", "").split(os.pathsep): + if part: + add(part) + for path in DEFAULT_ENGINE_PATHS: + add(path) + return os.pathsep.join(entries) + + +def codex_tool_git_env() -> dict[str, str]: + env = {"GIT_CONFIG_COUNT": str(len(ENGINE_GIT_CONFIG_OVERRIDES))} + for index, (key, value) in enumerate(ENGINE_GIT_CONFIG_OVERRIDES): + env[f"GIT_CONFIG_KEY_{index}"] = key + env[f"GIT_CONFIG_VALUE_{index}"] = value + return env + + +def external_env_path(repo: Path, value: str) -> bool: + try: + resolved = Path(value).expanduser().resolve() + except OSError: + return False + return not is_within(resolved, repo.resolve()) + + +def external_env_path_value(repo: Path, key: str, value: str) -> bool: + return normalize_external_env_path_value(repo, key, value) is not None + + +def normalize_external_env_path_value( + repo: Path, + key: str, + value: str, +) -> str | None: + values = value.split(os.pathsep) if key == "SSL_CERT_DIR" else [value] + normalized: list[str] = [] + for item in values: + if not item: + return None + try: + resolved = Path(item).expanduser().resolve() + except OSError: + return None + if is_within(resolved, repo.resolve()): + return None + normalized.append(str(resolved)) + return os.pathsep.join(normalized) if normalized else None + + +def safe_dbus_session_address(repo: Path, value: str) -> bool: + match = re.fullmatch( + r"unix:path=(?P[^,;%]+)(?:,guid=[0-9a-fA-F]+)?", + value, + ) + if not match: + return False + path = match.group("path") + return Path(path).is_absolute() and external_env_path(repo, path) + + +def safe_temp_root(repo: Path) -> Path: + try: + root = Path(tempfile.gettempdir()).resolve(strict=True) + except OSError as exc: + raise SystemExit(f"unable to resolve temporary directory: {exc}") from exc + if is_within(root, repo.resolve()): + raise SystemExit( + "temporary directory must be outside the reviewed repository; " + "unset or relocate TMPDIR/TMP/TEMP" + ) + return root + + +def parallel_test_temp_root(repo: Path) -> Path: + root = safe_temp_root(repo) + if os.name == "nt" or not env_truthy("OPENCLAW_TESTBOX"): + return root + + # Blacksmith stores its SSH control socket below HOME. macOS temp roots can + # already consume the Unix-socket path budget before the socket name. + try: + short_root = Path("/tmp").resolve(strict=True) + short_root_stat = short_root.stat() + except OSError as exc: + raise SystemExit(f"unable to resolve short Testbox temp root: {exc}") from exc + if not stat.S_ISDIR(short_root_stat.st_mode): + raise SystemExit("short Testbox temp root must be a directory") + if is_within(short_root, repo.resolve()): + raise SystemExit("short Testbox temp root must be outside the reviewed repository") + if short_root_stat.st_mode & stat.S_IWOTH and not ( + short_root_stat.st_mode & stat.S_ISVTX + ): + raise SystemExit("world-writable Testbox temp root must have the sticky bit set") + return short_root + + +def safe_proxy_url(value: str) -> bool: + try: + candidate = value if "://" in value else f"http://{value}" + parsed = urllib.parse.urlsplit(candidate) + _ = parsed.port + except ValueError: + return False + return ( + parsed.scheme.lower() + in {"http", "https", "socks", "socks4", "socks4a", "socks5", "socks5h"} + and bool(parsed.hostname) + and parsed.username is None + and parsed.password is None + and parsed.path in {"", "/"} + and not parsed.query + and not parsed.fragment + ) + + +def safe_engine_env( + repo: Path, + extra_paths: list[Path] | None = None, + extra: dict[str, str] | None = None, + *, + engine: str | None = None, +) -> dict[str, str]: + common_allowed_exact = { + "ALL_PROXY", + "COMSPEC", + "DISABLE_AUTOUPDATER", + "DISABLE_ERROR_REPORTING", + "DISABLE_TELEMETRY", + "DO_NOT_TRACK", + "HTTP_PROXY", + "HTTPS_PROXY", + "LANG", + "LC_ALL", + "LOGNAME", + "NO_PROXY", + "PATHEXT", + "SHELL", + "SYSTEMROOT", + "TEMP", + "TMP", + "TMPDIR", + "USER", + "WINDIR", + "all_proxy", + "http_proxy", + "https_proxy", + "no_proxy", + } + codex_allowed_exact = { + "AZURE_OPENAI_API_KEY", + "AZURE_OPENAI_ENDPOINT", + "CODEX_API_KEY", + "OPENAI_API_KEY", + "OPENAI_BASE_URL", + "OPENAI_ORGANIZATION", + "OPENAI_PROJECT", + } + claude_allowed_exact = { + "ANTHROPIC_API_KEY", + "ANTHROPIC_AUTH_TOKEN", + "ANTHROPIC_AWS_API_KEY", + "ANTHROPIC_AWS_BASE_URL", + "ANTHROPIC_AWS_WORKSPACE_ID", + "ANTHROPIC_BASE_URL", + "ANTHROPIC_BEDROCK_BASE_URL", + "ANTHROPIC_BEDROCK_MANTLE_BASE_URL", + "ANTHROPIC_BEDROCK_SERVICE_TIER", + "ANTHROPIC_CUSTOM_HEADERS", + "ANTHROPIC_FOUNDRY_API_KEY", + "ANTHROPIC_FOUNDRY_AUTH_TOKEN", + "ANTHROPIC_FOUNDRY_BASE_URL", + "ANTHROPIC_FOUNDRY_RESOURCE", + "ANTHROPIC_SMALL_FAST_MODEL_AWS_REGION", + "ANTHROPIC_VERTEX_BASE_URL", + "ANTHROPIC_VERTEX_PROJECT_ID", + "ANTHROPIC_WORKSPACE_ID", + "AWS_ACCESS_KEY_ID", + "AWS_BEARER_TOKEN_BEDROCK", + "AWS_DEFAULT_REGION", + "AWS_PROFILE", + "AWS_REGION", + "AWS_SECRET_ACCESS_KEY", + "AWS_SESSION_TOKEN", + "CLAUDE_CODE_API_KEY_HELPER_TTL_MS", + "CLAUDE_CODE_CERT_STORE", + "CLAUDE_CODE_CLIENT_CERT", + "CLAUDE_CODE_CLIENT_KEY", + "CLAUDE_CODE_CLIENT_KEY_PASSPHRASE", + "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC", + "CLAUDE_CODE_OAUTH_REFRESH_TOKEN", + "CLAUDE_CODE_OAUTH_SCOPES", + "CLAUDE_CODE_OAUTH_TOKEN", + "CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST", + "CLAUDE_CODE_SKIP_ANTHROPIC_AWS_AUTH", + "CLAUDE_CODE_SKIP_BEDROCK_AUTH", + "CLAUDE_CODE_SKIP_FOUNDRY_AUTH", + "CLAUDE_CODE_SKIP_MANTLE_AUTH", + "CLAUDE_CODE_SKIP_VERTEX_AUTH", + "CLAUDE_CODE_USE_ANTHROPIC_AWS", + "CLAUDE_CODE_USE_BEDROCK", + "CLAUDE_CODE_USE_FOUNDRY", + "CLAUDE_CODE_USE_MANTLE", + "CLAUDE_CODE_USE_VERTEX", + "CLOUD_ML_REGION", + } | CLAUDE_CLOUD_CREDENTIAL_ENV_KEYS + multi_provider_allowed_exact = { + "ANTHROPIC_AWS_BASE_URL", + "ANTHROPIC_AWS_WORKSPACE_ID", + "ANTHROPIC_BASE_URL", + "ANTHROPIC_BEDROCK_BASE_URL", + "ANTHROPIC_BEDROCK_MANTLE_BASE_URL", + "ANTHROPIC_BEDROCK_SERVICE_TIER", + "ANTHROPIC_CUSTOM_HEADERS", + "ANTHROPIC_FOUNDRY_BASE_URL", + "ANTHROPIC_FOUNDRY_RESOURCE", + "ANTHROPIC_SMALL_FAST_MODEL_AWS_REGION", + "ANTHROPIC_VERTEX_BASE_URL", + "ANTHROPIC_VERTEX_PROJECT_ID", + "ANTHROPIC_WORKSPACE_ID", + "AWS_ACCESS_KEY_ID", + "AWS_BEARER_TOKEN_BEDROCK", + "AWS_DEFAULT_REGION", + "AWS_PROFILE", + "AWS_REGION", + "AWS_SECRET_ACCESS_KEY", + "AWS_SESSION_TOKEN", + "AZURE_OPENAI_ENDPOINT", + "CLOUD_ML_REGION", + "COPILOT_GITHUB_TOKEN", + "GITHUB_TOKEN", + "GH_TOKEN", + "OPENAI_BASE_URL", + "OPENAI_ORGANIZATION", + "OPENAI_PROJECT", + } | MULTI_PROVIDER_ENV_KEYS + pi_allowed_exact = multi_provider_allowed_exact | { + "PI_OFFLINE", + "PI_SKIP_VERSION_CHECK", + "PI_TELEMETRY", + } + engine_allowed_exact = { + "claude": claude_allowed_exact, + "codex": codex_allowed_exact, + "opencode": multi_provider_allowed_exact, + "pi": pi_allowed_exact, + }.get(engine or "", set()) + allowed_prefixes = ("AUTOREVIEW_FAKE_",) + custom_provider_env_keys: set[str] = set() + if engine in {"opencode", "pi"}: + for raw_key in os.environ.get("AUTOREVIEW_PROVIDER_ENV_ALLOW", "").split(","): + key = raw_key.strip() + if not key: + continue + if not CUSTOM_PROVIDER_ENV_NAME_PATTERN.fullmatch(key): + raise SystemExit( + "invalid AUTOREVIEW_PROVIDER_ENV_ALLOW entry; use comma-separated " + "credential variable names such as CORP_LLM_API_KEY" + ) + custom_provider_env_keys.add(key) + env = { + key: value + for key, value in os.environ.items() + if ( + key in common_allowed_exact + or key in engine_allowed_exact + or any(key.startswith(prefix) for prefix in allowed_prefixes) + or ( + engine == "pi" + and ( + key in MULTI_PROVIDER_CREDENTIAL_ENV_KEYS + or key in MULTI_PROVIDER_ENV_KEYS + or key in custom_provider_env_keys + ) + ) + or ( + engine == "opencode" + and ( + key in OPENCODE_PROVIDER_ENV_KEYS + or key in MULTI_PROVIDER_CREDENTIAL_ENV_KEYS + or key in MULTI_PROVIDER_ENV_KEYS + or key in custom_provider_env_keys + ) + ) + ) + } + for key in ( + "ALL_PROXY", + "HTTP_PROXY", + "HTTPS_PROXY", + "all_proxy", + "http_proxy", + "https_proxy", + ): + value = env.get(key) + if value and not safe_proxy_url(value): + raise SystemExit( + f"unsafe credentialed or malformed proxy URL in {key}; " + "configure a credential-free proxy URL before running autoreview" + ) + env["PATH"] = safe_engine_path(repo, extra_paths) + for key in ("HOME", "USERPROFILE"): + value = os.environ.get(key) + if value and external_env_path(repo, value): + env[key] = value + engine_config_paths = { + "claude": ("CLAUDE_CONFIG_DIR",), + "codex": ("CODEX_HOME",), + "pi": ("PI_CODING_AGENT_DIR",), + } + for key in engine_config_paths.get(engine or "", ()): + value = os.environ.get(key) + if value and external_env_path(repo, value): + env[key] = value + if engine == "codex": + dbus_address = os.environ.get("DBUS_SESSION_BUS_ADDRESS") + if dbus_address and safe_dbus_session_address(repo, dbus_address): + env["DBUS_SESSION_BUS_ADDRESS"] = dbus_address + xdg_runtime_dir = os.environ.get("XDG_RUNTIME_DIR") + if xdg_runtime_dir and external_env_path(repo, xdg_runtime_dir): + env["XDG_RUNTIME_DIR"] = xdg_runtime_dir + for key in CODEX_TRUST_PATH_ENV_KEYS: + value = os.environ.get(key) + env.pop(key, None) + normalized = ( + normalize_external_env_path_value(repo, key, value) + if value + else None + ) + if normalized: + env[key] = normalized + if engine in {"claude", "opencode", "pi"}: + for key in PROVIDER_CREDENTIAL_PATH_ENV_KEYS: + value = os.environ.get(key) + env.pop(key, None) + normalized = ( + normalize_external_env_path_value(repo, key, value) + if value + else None + ) + if normalized: + env[key] = normalized + if engine == "opencode" and (xdg_data_home := os.environ.get("XDG_DATA_HOME")): + if external_env_path(repo, xdg_data_home): + env["XDG_DATA_HOME"] = xdg_data_home + env.update(codex_tool_git_env()) + env.update(extra or {}) + if engine == "claude": + env["CLAUDE_CODE_DISABLE_AUTO_MEMORY"] = "1" + return env + + +def quote_java_tool_option(option: str) -> str: + if any(char in option for char in ("\0", "\r", "\n")): + raise SystemExit("parallel test home contains unsupported control characters") + return "'" + option.replace("'", "'\"'\"'") + "'" + + +def copy_blacksmith_testbox_credentials( + repo: Path, + source_home: str | None, + isolated_home: Path, +) -> None: + if not source_home: + return + source = Path(source_home).expanduser() / ".blacksmith" / "credentials" + try: + resolved = source.resolve() + source_stat = source.lstat() + except OSError: + return + if ( + not stat.S_ISREG(source_stat.st_mode) + or source_stat.st_size > 64 * 1024 + or not external_env_path(repo, str(resolved)) + ): + return + try: + data = source.read_bytes() + except OSError: + return + target_dir = isolated_home / ".blacksmith" + target_dir.mkdir(parents=True, exist_ok=True) + target = target_dir / "credentials" + target.write_bytes(data) + target.chmod(0o600) + + +def safe_test_env(repo: Path, isolated_home: Path) -> dict[str, str]: + resolved_home = isolated_home.resolve() + if is_within(resolved_home, repo.resolve()): + raise SystemExit("parallel test home must be outside the reviewed repository") + env = { + key: value + for key, value in os.environ.items() + if key in TEST_ENV_ALLOWED_EXACT + or any(key.startswith(prefix) for prefix in TEST_ENV_ALLOWED_PREFIXES) + } + for key in ( + "ALL_PROXY", + "HTTP_PROXY", + "HTTPS_PROXY", + "all_proxy", + "http_proxy", + "https_proxy", + ): + value = env.get(key) + if value and not safe_proxy_url(value): + raise SystemExit( + f"unsafe credentialed or malformed proxy URL in {key}; " + "configure a credential-free proxy URL before running autoreview" + ) + config_home = resolved_home / ".config" + data_home = resolved_home / ".local" / "share" + state_home = resolved_home / ".local" / "state" + cache_home = resolved_home / ".cache" + temp_home = resolved_home / "tmp" + gradle_home = resolved_home / ".gradle" + app_data = resolved_home / "AppData" / "Roaming" + local_app_data = resolved_home / "AppData" / "Local" + for path in ( + config_home, + data_home, + state_home, + cache_home, + temp_home, + gradle_home, + app_data, + local_app_data, + ): + path.mkdir(parents=True, exist_ok=True) + java_tool_options = quote_java_tool_option(f"-Duser.home={resolved_home}") + env.update( + { + "APPDATA": str(app_data), + "GRADLE_USER_HOME": str(gradle_home), + "HOME": str(resolved_home), + # JVM launchers do not derive user.home from HOME/USERPROFILE. + # This also reaches build-tool daemons that bypass PATH wrappers. + "JAVA_TOOL_OPTIONS": java_tool_options, + "LOCALAPPDATA": str(local_app_data), + "TEMP": str(temp_home), + "TMP": str(temp_home), + "TMPDIR": str(temp_home), + "USERPROFILE": str(resolved_home), + "XDG_CACHE_HOME": str(cache_home), + "XDG_CONFIG_HOME": str(config_home), + "XDG_DATA_HOME": str(data_home), + "XDG_STATE_HOME": str(state_home), + } + ) + original_home = os.environ.get("HOME") or os.environ.get("USERPROFILE") + if env.get("OPENCLAW_TESTBOX", "").strip().lower() in { + "1", + "true", + "yes", + "on", + }: + copy_blacksmith_testbox_credentials(repo, original_home, resolved_home) + rustup_home = os.environ.get("RUSTUP_HOME") + if not rustup_home and original_home: + default_rustup_home = Path(original_home).expanduser() / ".rustup" + if default_rustup_home.is_dir(): + rustup_home = str(default_rustup_home) + if rustup_home and external_env_path(repo, rustup_home): + env["RUSTUP_HOME"] = str(Path(rustup_home).expanduser().resolve()) + return env + + +def parse_ps_time(value: str) -> float: + try: + day_text, clock = value.strip().split("-", 1) if "-" in value else ("0", value.strip()) + parts = clock.split(":") + if not parts or len(parts) > 3: + return 0.0 + total = float(parts[-1]) + if len(parts) >= 2: + total += int(parts[-2]) * 60 + if len(parts) == 3: + total += int(parts[-3]) * 3600 + return int(day_text) * 86400 + total + except (IndexError, ValueError): + return 0.0 + + +def process_pids(repo: Path, pid: int) -> list[str]: + ps = find_command("ps", repo) + if not ps: + return [str(pid)] + result = subprocess.run( + [ps, "-A", "-o", "pid=", "-o", "ppid="], + text=True, + encoding=SUBPROCESS_TEXT_ENCODING, + errors=SUBPROCESS_TEXT_ERRORS, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + check=False, + ) + if result.returncode != 0: + return [str(pid)] + pids = [str(pid)] + for line in result.stdout.splitlines(): + fields = line.split() + if len(fields) == 2 and fields[1] == str(pid): + pids.append(fields[0]) + return pids + + +def sample_process_metrics(repo: Path, pid: int) -> tuple[float, float, int, str] | None: + ps = find_command("ps", repo) + if not ps: + return None + try: + result = subprocess.run( + [ + ps, + "-o", + "state=", + "-o", + "rss=", + "-o", + "time=", + "-p", + ",".join(process_pids(repo, pid)), + ], + text=True, + encoding=SUBPROCESS_TEXT_ENCODING, + errors=SUBPROCESS_TEXT_ERRORS, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + check=False, + ) + except OSError: + return None + if result.returncode != 0: + return None + + cpu_seconds = 0.0 + rss_kb = 0 + states: list[str] = [] + for line in result.stdout.splitlines(): + fields = line.split() + if len(fields) < 3: + continue + states.append(fields[0][:1] or "?") + try: + rss_kb += int(fields[1]) + except ValueError: + pass + cpu_seconds += parse_ps_time(fields[2]) + if not states: + return None + return (time.monotonic(), cpu_seconds, rss_kb, ",".join(sorted(set(states)))) + + +def format_process_metrics( + previous: tuple[float, float, int, str] | None, + current: tuple[float, float, int, str] | None, +) -> str: + if current is None: + return "" + interval = max(0.0, current[0] - previous[0]) if previous else 0.0 + cpu_delta = max(0.0, current[1] - previous[1]) if previous else 0.0 + cpu_percent = (cpu_delta / interval * 100) if interval > 0 else 0.0 + rss_mb = int(round(current[2] / 1024)) + interval_seconds = int(interval) if interval else 0 + return ( + f" cpu={cpu_delta:.1f}s/{interval_seconds}s({cpu_percent:.0f}%)" + f" rss={rss_mb}M state={current[3]}" + ) + + +def emit_heartbeat( + label: str, + repo: Path, + started: float, + proc: subprocess.Popen, + metrics: tuple[float, float, int, str] | None, +) -> tuple[float, float, int, str] | None: + elapsed = int(time.monotonic() - started) + next_metrics = sample_process_metrics(repo, proc.pid) + metrics_text = format_process_metrics(metrics, next_metrics) + print(f"review still running: {label} elapsed={elapsed}s pid={proc.pid}{metrics_text}", file=sys.stderr, flush=True) + return next_metrics or metrics + + def opencode_review_config(web_search: bool = True) -> dict[str, Any]: permission: dict[str, Any] = { "*": "deny", - "read": { - "*": "allow", - "*.env": "ask", - "*.env.*": "ask", - "*.env.example": "allow", - }, - "grep": "allow", - "glob": "allow", + "read": "deny", + "grep": "deny", + "glob": "deny", } if web_search: permission["websearch"] = "allow" - permission["webfetch"] = "allow" else: permission["websearch"] = "deny" - permission["webfetch"] = "deny" + # Generic fetches can reach loopback, private, link-local, or metadata endpoints. + permission["webfetch"] = "deny" return { "$schema": "https://opencode.ai/config.json", "autoupdate": False, @@ -156,13 +1597,16 @@ def opencode_review_config(web_search: bool = True) -> dict[str, Any]: def opencode_review_env(web_search: bool = True) -> dict[str, str]: - return { + env = { "OPENCODE_DISABLE_PROJECT_CONFIG": "1", "OPENCODE_CONFIG_CONTENT": json.dumps(opencode_review_config(web_search), separators=(",", ":")), "OPENCODE_DISABLE_AUTOUPDATE": "1", "OPENCODE_DISABLE_AUTOCOMPACT": "1", "OPENCODE_DISABLE_MODELS_FETCH": "1", } + if web_search and (enable_exa := os.environ.get("OPENCODE_ENABLE_EXA")): + env["OPENCODE_ENABLE_EXA"] = enable_exa + return env def run_with_heartbeat( @@ -175,7 +1619,9 @@ def run_with_heartbeat( stream_output: bool = False, stream_display: Callable[[str, str], str | None] | None = None, env: dict[str, str] | None = None, + resolve_root: Path | None = None, ) -> subprocess.CompletedProcess[str]: + resolve_root = resolve_root or cwd if stream_output: return run_with_stream( args, @@ -185,6 +1631,7 @@ def run_with_heartbeat( heartbeat_seconds=heartbeat_seconds, stream_display=stream_display, env=env, + resolve_root=resolve_root, ) started = time.monotonic() proc = subprocess.Popen( @@ -194,9 +1641,12 @@ def run_with_heartbeat( stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, - env=subprocess_env(env), + encoding=SUBPROCESS_TEXT_ENCODING, + errors=SUBPROCESS_TEXT_ERRORS, + env=env, ) first_communicate = True + metrics = sample_process_metrics(resolve_root, proc.pid) while True: try: stdout, stderr = proc.communicate( @@ -206,8 +1656,7 @@ def run_with_heartbeat( return subprocess.CompletedProcess(args, int(proc.returncode or 0), stdout, stderr) except subprocess.TimeoutExpired: first_communicate = False - elapsed = int(time.monotonic() - started) - print(f"review still running: {label} elapsed={elapsed}s pid={proc.pid}", file=sys.stderr, flush=True) + metrics = emit_heartbeat(label, resolve_root, started, proc, metrics) def run_with_stream( @@ -219,6 +1668,7 @@ def run_with_stream( heartbeat_seconds: int, stream_display: Callable[[str, str], str | None] | None, env: dict[str, str] | None = None, + resolve_root: Path, ) -> subprocess.CompletedProcess[str]: started = time.monotonic() proc = subprocess.Popen( @@ -228,8 +1678,10 @@ def run_with_stream( stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, + encoding=SUBPROCESS_TEXT_ENCODING, + errors=SUBPROCESS_TEXT_ERRORS, bufsize=1, - env=subprocess_env(env), + env=env, ) events: queue.Queue[tuple[str, str | None]] = queue.Queue() stdout_parts: list[str] = [] @@ -259,14 +1711,14 @@ def run_with_stream( thread.start() stdin_thread = threading.Thread(target=write_stdin, daemon=True) stdin_thread.start() + metrics = sample_process_metrics(resolve_root, proc.pid) open_streams = 2 while open_streams: try: name, line = events.get(timeout=heartbeat_seconds) except queue.Empty: - elapsed = int(time.monotonic() - started) - print(f"review still running: {label} elapsed={elapsed}s pid={proc.pid}", file=sys.stderr, flush=True) + metrics = emit_heartbeat(label, resolve_root, started, proc, metrics) continue if line is None: open_streams -= 1 @@ -278,7 +1730,7 @@ def run_with_stream( display = stream_display(name, line) if stream_display else line if display: target = sys.stdout if name == "stdout" else sys.stderr - target.write(display) + target.write(stream_display_escape(display)) target.flush() for thread in threads: @@ -288,33 +1740,32 @@ def run_with_stream( return subprocess.CompletedProcess(args, returncode, "".join(stdout_parts), "".join(stderr_parts)) +def git_result( + repo: Path, + *args: str, + check: bool = True, +) -> subprocess.CompletedProcess[str]: + try: + return run( + [resolve_command("git", repo), "--no-optional-locks", *SAFE_GIT_CONFIG_ARGS, *args], + repo, + check=check, + env=safe_git_env(repo), + text_errors="strict", + ) + except UnicodeDecodeError as exc: + raise SystemExit( + "refusing non-UTF-8 Git output because paths and diff content " + "cannot be validated without loss" + ) from exc + + def git(repo: Path, *args: str, check: bool = True) -> str: - return run([resolve_command("git", repo), *args], repo, check=check).stdout + return git_result(repo, *args, check=check).stdout -def git_limited(repo: Path, *args: str, limit: int = BUNDLE_LIMIT_BYTES, check: bool = True) -> str: - proc = subprocess.Popen( - [resolve_command("git", repo), *args], - cwd=repo, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - ) - assert proc.stdout is not None - assert proc.stderr is not None - data = proc.stdout.read(limit + 1) - truncated = len(data) > limit - if truncated: - proc.kill() - stderr = proc.stderr.read() - returncode = proc.wait() - if check and returncode != 0 and not truncated: - cmd = " ".join(["git", *args]) - raise SystemExit(f"command failed ({returncode}): {cmd}\n{stderr.decode('utf-8', errors='replace')}") - if truncated: - cmd = " ".join(["git", *args]) - raise SystemExit(f"review bundle too large; command output exceeded {limit} bytes: {cmd}. Narrow the review target.") - text = data[:limit].decode("utf-8", errors="replace") - return text +def git_path_list(repo: Path, *args: str, check: bool = True) -> list[str]: + return [path for path in git(repo, *args, check=check).split("\0") if path] def repo_root() -> Path: @@ -323,12 +1774,18 @@ def repo_root() -> Path: git_bin = find_command("git", unsafe_root) if not git_bin: raise SystemExit("git executable not found. Install Git or add it to PATH.") - result = subprocess.run( - [git_bin, "rev-parse", "--show-toplevel"], - text=True, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - ) + try: + result = subprocess.run( + [git_bin, "--no-optional-locks", *SAFE_GIT_CONFIG_ARGS, "rev-parse", "--show-toplevel"], + text=True, + encoding=SUBPROCESS_TEXT_ENCODING, + errors="strict", + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + env=safe_git_env(unsafe_root), + ) + except UnicodeDecodeError as exc: + raise SystemExit("repository root is not valid UTF-8") from exc if result.returncode != 0: raise SystemExit("autoreview must run inside a git repository") return Path(result.stdout.strip()).resolve() @@ -349,7 +1806,14 @@ def current_branch(repo: Path) -> str: def is_dirty(repo: Path) -> bool: - return bool(git(repo, "status", "--porcelain").strip()) + return bool( + git( + repo, + *global_excludes_git_args(repo), + "status", + "--porcelain", + ).strip() + ) def choose_target(repo: Path, mode: str, base_ref: str | None) -> tuple[str, str | None]: @@ -384,7 +1848,12 @@ def find_command(name: str, repo: Path) -> str | None: command = Path(name) if has_directory_component(name, command): base = command if command.is_absolute() else repo / command - return first_executable_candidate(base) + if is_within( + Path(os.path.abspath(base)), + Path(os.path.abspath(repo)), + ): + return None + return first_executable_candidate(base, reject_root=repo.resolve()) for part in os.environ.get("PATH", "").split(os.pathsep): if not part or part == ".": continue @@ -423,206 +1892,6776 @@ def first_executable_candidate(path: Path, *, reject_root: Path | None = None) - candidates = [path] for candidate in candidates: if candidate.is_file() and os.access(candidate, os.X_OK): - if reject_root is not None: - try: - if is_within(candidate.resolve(), reject_root): - continue - except OSError: - continue - return str(candidate) + try: + lexical_candidate = Path(os.path.abspath(candidate)) + resolved_candidate = candidate.resolve(strict=True) + except OSError: + continue + if reject_root is not None and ( + is_within(lexical_candidate, reject_root) + or is_within(resolved_candidate, reject_root) + ): + continue + return str(lexical_candidate) return None -def bounded(text: str, limit: int = 180_000) -> str: - if len(text) <= limit: - return text - return text[:limit] + f"\n\n[truncated at {limit} characters]\n" +def validate_git_ref(repo: Path, ref: str, label: str) -> str: + if not ref or ref.startswith("-") or ":" in ref or "\0" in ref or any(char.isspace() for char in ref): + raise SystemExit(f"unsafe {label} ref: {ref}") + result = git( + repo, + "rev-parse", + "--verify", + "--quiet", + "--end-of-options", + f"{ref}^{{commit}}", + check=False, + ) + if not result: + raise SystemExit(f"unknown {label} ref: {ref}") + return ref -def bounded_field(text: str, limit: int) -> str: - if len(text) <= limit: - return text - suffix = "\n\n[truncated]" - return text[: max(0, limit - len(suffix))] + suffix +TRUFFLEHOG_INSTALL_URL = "https://github.com/trufflesecurity/trufflehog#installation" +TRUFFLEHOG_FINDINGS_EXIT_CODE = 183 -def read_text(path: Path, limit: int = 40_000, *, fail_on_truncate: bool = False) -> str: - if path.is_symlink(): - return "[symlink omitted]" +def git_bytes( + repo: Path, + *args: str, + check: bool = True, +) -> subprocess.CompletedProcess[bytes]: + result = subprocess.run( + [ + resolve_command("git", repo), + "--no-optional-locks", + *SAFE_GIT_CONFIG_ARGS, + *args, + ], + cwd=repo, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + env=safe_git_env(repo), + ) + if check and result.returncode != 0: + detail = (result.stderr or result.stdout).decode( + SUBPROCESS_TEXT_ENCODING, + errors=SUBPROCESS_TEXT_ERRORS, + ) + raise SystemExit( + f"Git failed while preparing the TruffleHog scan ({result.returncode}): " + f"{display_escape(detail, 1000, multiline=True)}" + ) + return result + + +def snapshot_destination(root: Path, rel: str) -> Path: + path = PurePosixPath(rel) + if ( + path.is_absolute() + or not path.parts + or path.parts[0].casefold() == ".git" + or any(part in {"", ".", ".."} for part in path.parts) + ): + raise SystemExit("Git returned an unsafe path while preparing the TruffleHog scan") + return root.joinpath(*path.parts) + + +def write_snapshot_blob(root: Path, rel: str, content: bytes) -> None: + destination = snapshot_destination(root, rel) try: - with path.open("rb") as handle: - data = handle.read(limit + 1) + destination.parent.mkdir(parents=True, exist_ok=True) + destination.write_bytes(content) except OSError as exc: - return f"[unreadable: {exc}]" - if b"\0" in data: - return "[binary file omitted]" - truncated = len(data) > limit - if truncated and fail_on_truncate: - raise SystemExit(f"review bundle too large; untracked file exceeded {limit} bytes: {path}") - text = data[:limit].decode("utf-8", errors="replace") - if truncated: - return text + f"\n\n[truncated at {limit} bytes]\n" - return text + raise SystemExit( + "unable to prepare changed content for the TruffleHog scan: " + f"{display_escape(exc, 500)}" + ) from exc -def join_bundle(parts: list[str], *, limit: int = BUNDLE_LIMIT_BYTES) -> str: - bundle = "\n\n".join(parts) - if len(bundle.encode("utf-8")) > limit: - raise SystemExit(f"review bundle too large; assembled bundle exceeded {limit} bytes. Narrow the review target.") - return bundle - - -def risky_untracked_path(rel: str, *, include_fragments: bool = True) -> bool: - path = rel.lower() - name = Path(path).name - parts = set(Path(path).parts) - risky_names = { - ".env", - ".env.local", - ".envrc", - ".git-credentials", - ".netrc", - ".npmrc", - ".pypirc", - ".terraformrc", - "auth.toml", - "config.json", - "id_rsa", - "id_dsa", - "id_ecdsa", - "id_ed25519", - "pip.conf", - "credentials", - "credentials.toml", - "credentials.json", - "secrets.json", - } - risky_suffixes = ( - ".key", - ".pem", - ".p12", - ".pfx", - ".mobileprovision", - ".log", - ) - risky_dirs = {".aws", ".azure", ".docker", ".gem", ".gnupg", ".kube", ".ssh"} - risky_fragments = ("apikey", "api-key", "auth", "secret", "token", "password", "credential", "private-key") - return ( - name.startswith(".env") - or name in risky_names - or path.endswith(risky_suffixes) - or any(part in risky_dirs for part in parts) - or (include_fragments and any(fragment in path for fragment in risky_fragments)) +def materialize_index_snapshot(repo: Path, root: Path, paths: list[str]) -> int: + if not paths: + return 0 + records = git_bytes( + repo, + "--literal-pathspecs", + "ls-files", + "--stage", + "-z", + "--", + *paths, + ).stdout + count = 0 + for record in records.split(b"\0"): + if not record: + continue + metadata, raw_path = record.split(b"\t", 1) + mode, object_id, stage = metadata.split() + if stage != b"0": + raise SystemExit( + "cannot run TruffleHog with unmerged index entries; resolve the conflict and rerun autoreview" + ) + if mode == b"160000": + continue + rel = raw_path.decode(SUBPROCESS_TEXT_ENCODING, errors="strict") + content = git_bytes( + repo, + "cat-file", + "blob", + object_id.decode("ascii"), + ).stdout + write_snapshot_blob(root, rel, content) + count += 1 + return count + + +def materialize_tree_snapshot( + repo: Path, + root: Path, + ref: str, + paths: list[str], +) -> int: + if not paths: + return 0 + records = git_bytes( + repo, + "--literal-pathspecs", + "ls-tree", + "-rz", + ref, + "--", + *paths, + ).stdout + count = 0 + for record in records.split(b"\0"): + if not record: + continue + metadata, raw_path = record.split(b"\t", 1) + mode, object_type, object_id = metadata.split() + if object_type != b"blob" or mode == b"160000": + continue + rel = raw_path.decode(SUBPROCESS_TEXT_ENCODING, errors="strict") + content = git_bytes( + repo, + "cat-file", + "blob", + object_id.decode("ascii"), + ).stdout + write_snapshot_blob(root, rel, content) + count += 1 + return count + + +def open_worktree_parent(repo: Path, rel_path: Path) -> int | bool | None: + required_dir_fd = {os.open, os.stat, os.readlink} + if not required_dir_fd <= os.supports_dir_fd: + return None + flags = ( + os.O_RDONLY + | getattr(os, "O_BINARY", 0) + | getattr(os, "O_CLOEXEC", 0) + | getattr(os, "O_DIRECTORY", 0) + | getattr(os, "O_NOFOLLOW", 0) ) + descriptor = os.open(repo.resolve(), flags) + try: + for part in rel_path.parts[:-1]: + try: + component = os.stat( + part, + dir_fd=descriptor, + follow_symlinks=False, + ) + except (FileNotFoundError, NotADirectoryError): + os.close(descriptor) + return False + if stat.S_ISLNK(component.st_mode): + raise OSError("symlinked parent directory") + if not stat.S_ISDIR(component.st_mode): + os.close(descriptor) + return False + next_descriptor = os.open(part, flags, dir_fd=descriptor) + os.close(descriptor) + descriptor = next_descriptor + return descriptor + except OSError as exc: + os.close(descriptor) + raise SystemExit( + "refusing to snapshot changed content through a symlinked or unstable parent directory: " + f"{display_escape(exc, 500)}" + ) from exc -def ensure_no_risky_review_paths(paths: list[str], *, include_risky_paths: bool, context: str) -> None: - risky = [path for path in paths if risky_untracked_path(path, include_fragments=False)] - if risky and not include_risky_paths: - joined = ", ".join(risky[:10]) - suffix = "" if len(risky) <= 10 else f", +{len(risky) - 10} more" +def copy_worktree_file(repo: Path, root: Path, rel: str) -> bool: + rel_path = Path(*PurePosixPath(rel).parts) + parent_descriptor = open_worktree_parent(repo, rel_path) + if parent_descriptor is False: + return False + if parent_descriptor is not None: + try: + source_stat = os.stat( + rel_path.name, + dir_fd=parent_descriptor, + follow_symlinks=False, + ) + except (FileNotFoundError, NotADirectoryError): + os.close(parent_descriptor) + return False + except OSError as exc: + os.close(parent_descriptor) + raise SystemExit( + "unable to read changed content for the TruffleHog scan: " + f"{display_escape(exc, 500)}" + ) from exc + if stat.S_ISLNK(source_stat.st_mode): + try: + content = os.fsencode( + os.readlink(rel_path.name, dir_fd=parent_descriptor) + ) + except OSError as exc: + raise SystemExit( + "unable to read changed symlink for the TruffleHog scan: " + f"{display_escape(exc, 500)}" + ) from exc + finally: + os.close(parent_descriptor) + write_snapshot_blob(root, rel, content) + return True + if stat.S_ISDIR(source_stat.st_mode): + os.close(parent_descriptor) + return False + if not stat.S_ISREG(source_stat.st_mode): + os.close(parent_descriptor) + raise SystemExit( + "changed content is not a regular file; remove it or resolve the repository state before rerunning autoreview" + ) + flags = ( + os.O_RDONLY + | getattr(os, "O_BINARY", 0) + | getattr(os, "O_CLOEXEC", 0) + | getattr(os, "O_NOFOLLOW", 0) + ) + try: + descriptor = os.open( + rel_path.name, + flags, + dir_fd=parent_descriptor, + ) + except OSError as exc: + raise SystemExit( + "unable to snapshot changed content for the TruffleHog scan: " + f"{display_escape(exc, 500)}" + ) from exc + finally: + os.close(parent_descriptor) + opened = os.fstat(descriptor) + if not stat.S_ISREG(opened.st_mode) or ( + source_stat.st_dev, + source_stat.st_ino, + ) != ( + opened.st_dev, + opened.st_ino, + ): + os.close(descriptor) + raise SystemExit( + "unable to snapshot changed content for the TruffleHog scan: file changed while opening" + ) + destination = snapshot_destination(root, rel) + destination.parent.mkdir(parents=True, exist_ok=True) + try: + with destination.open("wb") as output: + while chunk := os.read(descriptor, 1024 * 1024): + output.write(chunk) + after = os.fstat(descriptor) + if ( + opened.st_mode, + opened.st_size, + opened.st_mtime_ns, + ) != ( + after.st_mode, + after.st_size, + after.st_mtime_ns, + ): + raise OSError("file changed while reading") + except OSError as exc: + destination.unlink(missing_ok=True) + raise SystemExit( + "unable to snapshot changed content for the TruffleHog scan: " + f"{display_escape(exc, 500)}" + ) from exc + finally: + os.close(descriptor) + return True + + source = repo / rel_path + if rel_path.parent != Path(".") and raw_repo_path_has_symlink_component( + repo, + rel_path.parent, + ): + raise SystemExit( + "refusing to snapshot changed content through a symlinked parent directory" + ) + try: + source_stat = source.lstat() + except (FileNotFoundError, NotADirectoryError): + return False + if stat.S_ISLNK(source_stat.st_mode): + write_snapshot_blob(root, rel, os.fsencode(os.readlink(source))) + return True + if stat.S_ISDIR(source_stat.st_mode): + return False + if not stat.S_ISREG(source_stat.st_mode): + raise SystemExit( + "changed content is not a regular file; remove it or resolve the repository state before rerunning autoreview" + ) + content = source.read_bytes() + if source.lstat() != source_stat: raise SystemExit( - f"risky {context} path omitted from review bundle: {joined}{suffix}. " - "Pass --include-risky-paths only when intentionally reviewing secret-like files." + "unable to snapshot changed content for the TruffleHog scan: file changed while reading" ) + write_snapshot_blob(root, rel, content) + return True -def local_bundle(repo: Path, *, include_risky_paths: bool = False) -> str: - staged_paths = [line for line in git(repo, "diff", "--name-only", "--cached").splitlines() if line] - unstaged_paths = [line for line in git(repo, "diff", "--name-only").splitlines() if line] - ensure_no_risky_review_paths(staged_paths, include_risky_paths=include_risky_paths, context="staged") - ensure_no_risky_review_paths(unstaged_paths, include_risky_paths=include_risky_paths, context="unstaged") - parts = [ - "# Git Status", - git(repo, "status", "--short"), - "# Staged Diff", - git(repo, "diff", "--cached", "--stat"), - git_limited(repo, "diff", "--cached", "--patch", "--find-renames"), - "# Unstaged Diff", - git(repo, "diff", "--stat"), - git_limited(repo, "diff", "--patch", "--find-renames"), - ] - untracked = [line for line in git(repo, "ls-files", "--others", "--exclude-standard").splitlines() if line] - if untracked: - parts.append("# Untracked Files") - for rel in untracked: - if risky_untracked_path(rel) and not include_risky_paths: - parts.append(f"## {rel}\n[omitted: risky untracked path; pass --include-risky-paths to review intentionally]") - continue - path = repo / rel - parts.append(f"## {rel}\n{read_text(path, limit=180_000, fail_on_truncate=True)}") - return join_bundle(parts) +def materialize_worktree_snapshot(repo: Path, root: Path, paths: list[str]) -> int: + return sum(copy_worktree_file(repo, root, rel) for rel in paths) -def branch_bundle(repo: Path, base_ref: str, *, include_risky_paths: bool = False) -> str: - paths = [line for line in git(repo, "diff", "--name-only", f"{base_ref}...HEAD").splitlines() if line] - ensure_no_risky_review_paths(paths, include_risky_paths=include_risky_paths, context="branch") - return join_bundle( - [ - "# Branch Diff", - f"base: {base_ref}", - git(repo, "diff", "--stat", f"{base_ref}...HEAD"), - git_limited(repo, "diff", "--patch", "--find-renames", f"{base_ref}...HEAD"), - ] - ) +def clear_snapshot_paths(root: Path, paths: list[str]) -> None: + for rel in paths: + destination = snapshot_destination(root, rel) + if destination.is_symlink() or destination.is_file(): + destination.unlink() + elif destination.exists(): + shutil.rmtree(destination) -def commit_bundle(repo: Path, commit_ref: str, *, include_risky_paths: bool = False) -> str: - paths = [line for line in git(repo, "show", "--name-only", "--format=", commit_ref).splitlines() if line] - ensure_no_risky_review_paths(paths, include_risky_paths=include_risky_paths, context="commit") - return join_bundle( - [ - "# Commit Diff", - f"commit: {commit_ref}", - git(repo, "show", "--stat", "--format=fuller", commit_ref), - git_limited(repo, "show", "--patch", "--find-renames", "--format=fuller", commit_ref), - ] +def commit_snapshot(repo: Path, label: str) -> str: + git(repo, "add", "-A", "-f") + git( + repo, + "-c", + "user.name=Autoreview", + "-c", + "user.email=autoreview@example.invalid", + "commit", + "-q", + "--allow-empty", + "-m", + label, ) + return git(repo, "rev-parse", "HEAD").strip() -def review_paths(repo: Path, target: str, target_ref: str | None, commit_ref: str) -> set[str]: - names: set[str] = set() - if target == "local": - sources = [ - git(repo, "diff", "--name-only", "--cached"), - git(repo, "diff", "--name-only"), - git(repo, "ls-files", "--others", "--exclude-standard"), +def safe_trufflehog_env(repo: Path) -> dict[str, str]: + env = safe_git_env(repo) + for key in ( + "ALL_PROXY", + "HTTP_PROXY", + "HTTPS_PROXY", + "NO_PROXY", + "all_proxy", + "http_proxy", + "https_proxy", + "no_proxy", + ): + value = os.environ.get(key) + if not value: + continue + if key.casefold() != "no_proxy" and not safe_proxy_url(value): + raise SystemExit( + f"unsafe credentialed or malformed proxy URL in {key}; " + "configure a credential-free proxy URL before running autoreview" + ) + env[key] = value + return env + + +def prepare_trufflehog_history( + repo: Path, + target: str, + target_ref: str | None, + commit_ref: str, + root: Path, +) -> str: + git(root, "init", "-q") + if target == "local": + staged_paths = git_path_list( + repo, + "diff", + *SAFE_DIFF_FLAGS, + "--name-only", + "--cached", + "-z", + ) + unstaged_paths = git_path_list( + repo, + "diff", + *SAFE_DIFF_FLAGS, + "--name-only", + "-z", + ) + unstaged_paths.extend( + git_path_list( + repo, + *global_excludes_git_args(repo), + "ls-files", + "--others", + "--exclude-standard", + "-z", + ) + ) + paths = sorted(set(staged_paths + unstaged_paths)) + head = git(repo, "rev-parse", "--verify", "HEAD", check=False).strip() + if head: + materialize_tree_snapshot(repo, root, head, paths) + base_commit = commit_snapshot(root, "baseline") + clear_snapshot_paths(root, paths) + materialize_index_snapshot(repo, root, paths) + commit_snapshot(root, "staged") + clear_snapshot_paths(root, paths) + materialize_worktree_snapshot(repo, root, paths) + commit_snapshot(root, "working") + # TruffleHog's Git parser scans additions only. Reverse commits make + # deleted bytes additions without exposing unchanged baseline content. + clear_snapshot_paths(root, paths) + materialize_index_snapshot(repo, root, paths) + commit_snapshot(root, "reverse staged") + clear_snapshot_paths(root, paths) + if head: + materialize_tree_snapshot(repo, root, head, paths) + commit_snapshot(root, "reverse baseline") + return base_commit + + if target == "branch": + assert target_ref + target_ref = validate_git_ref(repo, target_ref, "base") + paths = git_path_list( + repo, + "diff", + *SAFE_DIFF_FLAGS, + "--name-only", + "-z", + "--end-of-options", + f"{target_ref}...HEAD", + ) + base_ref = git(repo, "merge-base", target_ref, "HEAD").strip() + reviewed_ref = "HEAD" + else: + reviewed_ref = validate_git_ref(repo, commit_ref, "commit") + parents = git(repo, "rev-list", "--parents", "-n", "1", reviewed_ref).split() + if len(parents) > 2: + raise SystemExit( + "commit review does not accept merge commits; review the branch diff " + "or an individual parent-relative commit instead" + ) + base_ref = parents[1] if len(parents) == 2 else "" + paths = git_path_list( + repo, + "show", + *SAFE_DIFF_FLAGS, + "--name-only", + "--format=", + "-z", + "--end-of-options", + reviewed_ref, + ) + if base_ref: + materialize_tree_snapshot(repo, root, base_ref, paths) + base_commit = commit_snapshot(root, "baseline") + clear_snapshot_paths(root, paths) + materialize_tree_snapshot(repo, root, reviewed_ref, paths) + commit_snapshot(root, "reviewed") + # Scan the reverse diff too so deleted bytes become additions for + # TruffleHog without scanning unchanged content from the baseline. + clear_snapshot_paths(root, paths) + if base_ref: + materialize_tree_snapshot(repo, root, base_ref, paths) + commit_snapshot(root, "reverse baseline") + return base_commit + + +def run_trufflehog_preflight( + repo: Path, + target: str, + target_ref: str | None, + commit_ref: str, +) -> None: + trufflehog_bin = find_command("trufflehog", repo) + if not trufflehog_bin: + raise SystemExit( + "TruffleHog is required but was not found. Install it using the official " + f"instructions, then rerun autoreview: {TRUFFLEHOG_INSTALL_URL}" + ) + started = time.monotonic() + with tempfile.TemporaryDirectory( + prefix="autoreview-trufflehog.", + dir=safe_temp_root(repo), + ) as tempdir: + scan_repo = Path(tempdir) + base_commit = prepare_trufflehog_history( + repo, + target, + target_ref, + commit_ref, + scan_repo, + ) + result = run( + [ + trufflehog_bin, + "git", + scan_repo.resolve().as_uri(), + "--since-commit", + base_commit, + "--branch", + "HEAD", + "--no-update", + "--no-color", + # Match TruffleHog's pre-commit mode. Unverified heuristic + # candidates are excluded to avoid restoring false positives. + "--results=verified,unknown", + "--fail", + "--fail-on-scan-errors", + ], + repo, + check=False, + env=safe_trufflehog_env(repo), + ) + if result.returncode == TRUFFLEHOG_FINDINGS_EXIT_CODE: + raise SystemExit( + "TruffleHog found verified or unknown credentials in the reviewed changes. " + "Remove or rotate them, then rerun autoreview." + ) + if result.returncode != 0: + raise SystemExit( + "TruffleHog could not complete the credential scan. Run TruffleHog directly " + "to diagnose the scanner error, then rerun autoreview." + ) + print(f"trufflehog: clean ({time.monotonic() - started:.1f}s)") + + +def bounded(text: str, limit: int = 180_000) -> str: + if len(text) <= limit: + return text + return text[:limit] + f"\n\n[truncated at {limit} characters]\n" + + +def ensure_reviewer_input_complete(reviewer: argparse.Namespace, input_truncated: bool) -> None: + if input_truncated: + raise SystemExit( + f"{reviewer.engine} engine refused truncated review input because it cannot recover omitted diff hunks; " + "reduce the change/input size" + ) + + +def bounded_field(text: str, limit: int) -> str: + if len(text) <= limit: + return text + suffix = "\n\n[truncated]" + return text[: max(0, limit - len(suffix))] + suffix + + +def display_escape(text: object, limit: int, *, multiline: bool = False) -> str: + parts: list[str] = [] + for char in str(text): + codepoint = ord(char) + if multiline and char == "\n": + parts.append(char) + elif codepoint < 32 or 127 <= codepoint <= 159: + parts.append(f"\\x{codepoint:02x}") + elif unicodedata.category(char) in {"Cf", "Cs"}: + parts.append( + f"\\u{codepoint:04x}" + if codepoint <= 0xFFFF + else f"\\U{codepoint:08x}" + ) + else: + parts.append(char) + rendered = "".join(parts) + if len(rendered) <= limit: + return rendered + suffix = "...[truncated]" + return rendered[: max(0, limit - len(suffix))] + suffix[:limit] + + +def stream_display_escape(text: str) -> str: + return display_escape( + text, + max(1000, len(text) * 10), + multiline=True, + ) + + +def read_prefix(path: Path, limit: int) -> tuple[bytes, bool]: + descriptor: int | None = None + try: + # os.stat, not Path.stat: the follow_symlinks kwarg on pathlib needs + # Python 3.10+, and macOS system python3 is still 3.9. + before = os.stat(path, follow_symlinks=False) + if not stat.S_ISREG(before.st_mode): + raise OSError("not a regular file") + flags = ( + os.O_RDONLY + | getattr(os, "O_BINARY", 0) + | getattr(os, "O_CLOEXEC", 0) + | getattr(os, "O_NOFOLLOW", 0) + ) + descriptor = os.open(path, flags) + opened = os.fstat(descriptor) + if ( + not stat.S_ISREG(opened.st_mode) + or (before.st_dev, before.st_ino) != (opened.st_dev, opened.st_ino) + ): + raise OSError("file changed while opening") + chunks: list[bytes] = [] + remaining = limit + 1 + while remaining: + chunk = os.read(descriptor, remaining) + if not chunk: + break + chunks.append(chunk) + remaining -= len(chunk) + after = os.fstat(descriptor) + if ( + (opened.st_dev, opened.st_ino, opened.st_size, opened.st_mtime_ns) + != (after.st_dev, after.st_ino, after.st_size, after.st_mtime_ns) + ): + raise OSError("file changed while reading") + data = b"".join(chunks) + except OSError as exc: + raise SystemExit( + f"unreadable file: {display_escape(path, 500)}: " + f"{display_escape(exc, 500)}" + ) from exc + finally: + if descriptor is not None: + os.close(descriptor) + return data[:limit], len(data) > limit + + +def read_text_with_status(path: Path, limit: int = MAX_BUNDLE_TEXT_BYTES) -> tuple[str, bool]: + try: + data, truncated = read_prefix(path, limit) + except SystemExit as exc: + return f"[unreadable: {exc}]", True + if b"\0" in data: + return "[binary file omitted]", True + try: + text = data.decode("utf-8") + except UnicodeDecodeError: + return "[non-UTF-8 file omitted]", True + if len(text) > limit: + text = text[:limit] + truncated = True + if truncated: + return text + f"\n\n[truncated at {limit} characters]\n", True + return text, False + + +def read_text(path: Path, limit: int = MAX_BUNDLE_TEXT_BYTES) -> str: + return read_text_with_status(path, limit)[0] + + +def path_has_sensitive_part(rel: str | Path) -> bool: + normalized = Path(rel).as_posix().lower() + if "/.config/gcloud/" in f"/{normalized}/": + return True + return any(part.lower() in SENSITIVE_PATH_PARTS for part in Path(rel).parts) + + +def raw_repo_path_has_symlink_component(repo: Path, rel_path: Path) -> bool: + current = repo.resolve() + for part in rel_path.parts: + current = current / part + if current.is_symlink(): + return True + if not current.exists(): + break + return False + + +def fallback_expression(text: str, *, typescript: bool = False) -> str: + operator_keywords = ( + r"|as(?![\w$])|satisfies(?![\w$])" if typescript else "" + ) + operator = re.match( + r"\s*(?:\|\||&&|\?\?|[=<>!~:+\-*/%&|^?.,]" + r"|and\b|else\b|if\b|in(?![\w$])|instanceof(?![\w$])" + r"|is\b|not\b|or\b" + r"|unless\b" + + operator_keywords + + r")", + text, + ) + cursor = operator.end() if operator is not None else 0 + stack: list[str] = [] + operand_started = False + quote: str | None = None + escaped = False + line_comment = False + block_comment = False + pairs = {"(": ")", "[": "]", "{": "}"} + while cursor < len(text): + char = text[cursor] + next_char = text[cursor + 1] if cursor + 1 < len(text) else "" + if line_comment: + if char in "\r\n\u2028\u2029": + line_comment = False + if ( + operand_started + and not stack + and not javascript_continues_after_line_terminator( + text[cursor + 1 :], + typescript=typescript, + ) + ): + break + cursor += 1 + continue + if block_comment: + if char == "*" and next_char == "/": + block_comment = False + cursor += 2 + else: + cursor += 1 + continue + if quote is not None: + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == quote: + quote = None + cursor += 1 + continue + regex_end = javascript_regex_literal_end(text, cursor) + if regex_end is not None: + cursor = regex_end + continue + if char == "/" and next_char == "/": + line_comment = True + cursor += 2 + continue + if char == "/" and next_char == "*": + block_comment = True + cursor += 2 + continue + if char == "#": + line_comment = True + cursor += 1 + continue + if char in {'"', "'", "`"}: + quote = char + operand_started = True + cursor += 1 + continue + if char in pairs: + stack.append(pairs[char]) + operand_started = True + cursor += 1 + continue + if stack and char == stack[-1]: + stack.pop() + cursor += 1 + continue + if not stack and char in ",;)]}": + break + if char in "\r\n\u2028\u2029" and operand_started and not stack: + if not javascript_continues_after_line_terminator( + text[cursor + 1 :], + typescript=typescript, + ): + break + if not stack and (char.isalpha() or char in "_$"): + word_end = cursor + 1 + while word_end < len(text) and ( + text[word_end].isalnum() or text[word_end] in "_$" + ): + word_end += 1 + word = text[cursor:word_end] + word_operators = {"in", "instanceof"} + if typescript: + word_operators.update({"as", "satisfies"}) + operand_started = word not in word_operators + cursor = word_end + continue + if not stack and char in "=<>!~:+-*/%&|^?.": + operand_started = False + elif not char.isspace(): + operand_started = True + cursor += 1 + return text[:cursor] + + +def javascript_continues_after_line_terminator( + text: str, + *, + typescript: bool = False, +) -> bool: + suffix = text.lstrip() + if suffix.startswith(("++", "--", "~")) or ( + suffix.startswith("!") and not suffix.startswith("!=") + ): + return False + type_operators = ( + r"|as(?![\w$])|satisfies(?![\w$])" if typescript else "" + ) + return re.match( + r"(?:[=<>!:+\-*/%&|^?.,([`]|in(?![\w$])|instanceof(?![\w$])" + + type_operators + + r")", + suffix, + ) is not None + + +def top_level_fallback_suffix( + text: str, + *, + allow_chained_assignment: bool = False, +) -> str | None: + stack: list[tuple[str, bool]] = [] + pairs = {"(": ")", "[": "]", "{": "}"} + outer_group_openers: set[int] = set() + probe = 0 + while probe < len(text) and text[probe].isspace(): + probe += 1 + while probe < len(text) and text[probe] == "(": + outer_group_openers.add(probe) + probe += 1 + while probe < len(text) and text[probe].isspace(): + probe += 1 + quote: str | None = None + escaped = False + line_comment = False + block_comment = False + cursor = 0 + while cursor < len(text): + char = text[cursor] + next_char = text[cursor + 1] if cursor + 1 < len(text) else "" + if line_comment: + if char == "\n": + line_comment = False + object_member_context = any( + closer == "}" + for closer, _is_outer in stack + ) + remaining = text[cursor + 1 :] + top_level_statement = ( + not stack + and re.match( + r"\s*(?:\|\||&&|\?\?|\+|\?(?!\.)|or\b)", + remaining, + ) + is None + ) + object_sibling = ( + object_member_context + and starts_sibling_assignment(remaining) + ) + if top_level_statement or object_sibling: + return None + cursor += 1 + continue + if block_comment: + if char == "*" and next_char == "/": + block_comment = False + cursor += 2 + else: + cursor += 1 + continue + if quote is not None: + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == quote: + quote = None + cursor += 1 + continue + if char == "\\" and next_char: + cursor += 2 + continue + regex_end = javascript_regex_literal_end(text, cursor) + if regex_end is not None: + cursor = regex_end + continue + if char == "/" and next_char == "/": + line_comment = True + cursor += 2 + continue + if char == "/" and next_char == "*": + block_comment = True + cursor += 2 + continue + if char in {'"', "'", "`"}: + quote = char + cursor += 1 + continue + if char in pairs: + stack.append((pairs[char], cursor in outer_group_openers)) + cursor += 1 + continue + if stack and char == stack[-1][0]: + stack.pop() + cursor += 1 + continue + fallback_depth = not stack or all(is_outer for _closer, is_outer in stack) + if fallback_depth: + if char == "," and not stack: + sibling = sibling_assignment_match(text[cursor + 1 :]) + if sibling is not None: + if not allow_chained_assignment: + return None + value_start = cursor + 1 + sibling.end() + value = fallback_expression(text[value_start:]) + if fallback_secret_risk(value): + return value + cursor = value_start + len(value) + continue + if allow_chained_assignment: + value_start = cursor + 1 + value = fallback_expression(text[value_start:]) + if fallback_secret_risk(value): + return value + cursor = value_start + len(value) + continue + if text.startswith(("||", "&&", "??"), cursor): + return text[cursor:] + if char in {"+", "?"} and not text.startswith("?.", cursor): + return text[cursor:] + left_boundary = cursor == 0 or not ( + text[cursor - 1].isalnum() or text[cursor - 1] == "_" + ) + word = ( + re.match(r"(?:or|and|if|unless)\b", text[cursor:]) + if left_boundary + else None + ) + if word is not None: + return text[cursor:] + if char in "\n;" and not stack: + return None + cursor += 1 + return None + + +def starts_sibling_assignment(text: str) -> bool: + return sibling_assignment_match(text) is not None + + +def sibling_assignment_match(text: str) -> re.Match[str] | None: + return re.match( + r"\s*(?:" + r"\.\.\.[^,\r\n]+(?:,|$)" + r"|(?:[A-Za-z_$][A-Za-z0-9_$]*" + r"|[0-9]+(?:\.[0-9]+)?" + r"|[\"'][^\"'\r\n]+[\"']" + r"|\[[^\]\r\n]+\]" + r"|\{[^}\r\n]+\})\s*" + r"(?::(?![:=])|=(?!=|>)))", + text, + ) + + +def top_level_line_assignment_positions( + text: str, + positions: set[int], +) -> set[int]: + top_level: set[int] = set() + stack: list[str] = [] + quote: str | None = None + escaped = False + line_comment = False + block_comment = False + line_start = 0 + cursor = 0 + while cursor < len(text): + if ( + cursor in positions + and not stack + and not text[line_start:cursor].strip() + ): + top_level.add(cursor) + char = text[cursor] + next_char = text[cursor + 1] if cursor + 1 < len(text) else "" + if line_comment: + if char == "\n": + line_comment = False + line_start = cursor + 1 + cursor += 1 + continue + if block_comment: + if char == "*" and next_char == "/": + block_comment = False + cursor += 2 + else: + if char == "\n": + line_start = cursor + 1 + cursor += 1 + continue + if quote is not None: + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == quote: + quote = None + if char == "\n": + line_start = cursor + 1 + cursor += 1 + continue + regex_end = javascript_regex_literal_end(text, cursor) + if regex_end is not None: + cursor = regex_end + continue + if char == "/" and next_char == "/": + line_comment = True + cursor += 2 + continue + if char == "/" and next_char == "*": + block_comment = True + cursor += 2 + continue + if char == "#" and not javascript_private_member_marker(text, cursor): + line_comment = True + cursor += 1 + continue + if char in {'"', "'", "`"}: + quote = char + elif char == "(": + stack.append(")") + elif char == "[": + stack.append("]") + elif stack and char == stack[-1]: + stack.pop() + if char == "\n": + line_start = cursor + 1 + cursor += 1 + return top_level + + +def raw_double_quote_start( + text: str, + start: int, +) -> tuple[str, int] | None: + if ( + not text.startswith('"""', start) + or "@" in text[max(0, start - 2) : start] + ): + return None + width = 3 + while start + width < len(text) and text[start + width] == '"': + width += 1 + after = start + width + delimiter = '"' * width + return delimiter, after + + +def raw_double_quote_end( + text: str, + start: int, + width: int, +) -> int | None: + cursor = start + while cursor < len(text): + run_start = text.find('"', cursor) + if run_start < 0: + return None + run_end = run_start + 1 + while run_end < len(text) and text[run_end] == '"': + run_end += 1 + if run_end - run_start >= width: + return run_end + cursor = run_end + return None + + +def csharp_quoted_literal_end( + text: str, + quote_start: int, + *, + verbatim: bool, + interpolated: bool, + nesting: int = 0, +) -> int | None: + if nesting > 64: + return None + quote = text[quote_start] + cursor = quote_start + 1 + interpolation_depth = 0 + while cursor < len(text): + char = text[cursor] + next_char = text[cursor + 1] if cursor + 1 < len(text) else "" + if interpolation_depth: + if char == "/" and next_char == "/": + line_end = text.find("\n", cursor + 2) + cursor = len(text) if line_end < 0 else line_end + continue + if char == "/" and next_char == "*": + comment_end = text.find("*/", cursor + 2) + cursor = len(text) if comment_end < 0 else comment_end + 2 + continue + if char == '"': + raw_start = raw_double_quote_start(text, cursor) + if raw_start is not None: + delimiter, content_start = raw_start + raw_end = raw_double_quote_end( + text, + content_start, + len(delimiter), + ) + if raw_end is None: + return None + cursor = raw_end + continue + if char in {'"', "'"}: + marker = text[max(0, cursor - 2) : cursor] + nested_end = csharp_quoted_literal_end( + text, + cursor, + verbatim=char == '"' and "@" in marker, + interpolated=char == '"' and "$" in marker, + nesting=nesting + 1, + ) + if nested_end is None: + return None + cursor = nested_end + continue + if char == "{": + interpolation_depth += 1 + elif char == "}": + interpolation_depth -= 1 + cursor += 1 + continue + if interpolated and char == "{": + if next_char == "{": + cursor += 2 + continue + interpolation_depth = 1 + cursor += 1 + continue + if interpolated and char == "}" and next_char == "}": + cursor += 2 + continue + if verbatim and char == '"' and next_char == '"': + cursor += 2 + continue + if not verbatim and char == "\\": + cursor += 2 + continue + if char == quote: + return cursor + 1 + cursor += 1 + return None + + +@functools.lru_cache(maxsize=8) +def mask_csharp_evidence_prefix(text: str) -> str: + masked = list(text) + + def mask_span(start: int, end: int) -> None: + for index in range(start, end): + if masked[index] not in "\r\n": + masked[index] = " " + + cursor = 0 + line_has_content = False + while cursor < len(text): + char = text[cursor] + next_char = text[cursor + 1] if cursor + 1 < len(text) else "" + line_leading = not line_has_content + if char == "\n": + line_has_content = False + cursor += 1 + continue + if line_leading and char == "#": + line_end = text.find("\n", cursor) + line_end = len(text) if line_end < 0 else line_end + mask_span(cursor, line_end) + line_has_content = True + cursor = line_end + continue + if ( + line_leading + and char == "[" + and re.match(r"\[(?:assembly|module)\s*:", text[cursor:]) + ): + depth = 0 + end = cursor + quote: str | None = None + verbatim_quote = False + escaped = False + while end < len(text): + current = text[end] + if quote is not None: + if escaped: + escaped = False + elif ( + verbatim_quote + and quote == '"' + and text.startswith('""', end) + ): + end += 2 + continue + elif current == "\\" and not verbatim_quote: + escaped = True + elif current == quote: + quote = None + verbatim_quote = False + elif current in {'"', "'"}: + quote = current + verbatim_quote = ( + current == '"' + and text[max(cursor, end - 1) : end] == "@" + ) + elif current == "[": + depth += 1 + elif current == "]": + depth -= 1 + if depth == 0: + end += 1 + break + end += 1 + mask_span(cursor, end) + line_has_content = True + cursor = end + continue + if char == "/" and next_char == "/": + line_end = text.find("\n", cursor) + line_end = len(text) if line_end < 0 else line_end + mask_span(cursor, line_end) + line_has_content = True + cursor = line_end + continue + if char == "/" and next_char == "*": + comment_end = text.find("*/", cursor + 2) + comment_end = len(text) if comment_end < 0 else comment_end + 2 + mask_span(cursor, comment_end) + line_has_content = True + cursor = comment_end + continue + if char in {'"', "'"}: + raw_start = raw_double_quote_start(text, cursor) + if raw_start is not None: + delimiter, content_start = raw_start + end = raw_double_quote_end( + text, + content_start, + len(delimiter), + ) + end = len(text) if end is None else end + mask_span(cursor, end) + line_has_content = True + cursor = end + continue + marker = text[max(0, cursor - 2) : cursor] + end = csharp_quoted_literal_end( + text, + cursor, + verbatim=char == '"' and "@" in marker, + interpolated=char == '"' and "$" in marker, + ) + end = len(text) if end is None else end + mask_span(cursor, min(end, len(text))) + line_has_content = True + cursor = end + continue + if not char.isspace(): + line_has_content = True + cursor += 1 + return "".join(masked) + + +def csharp_verbatim_string_content(text: str, quote_start: int) -> str: + quote_end = csharp_quoted_literal_end( + text, + quote_start, + verbatim=True, + interpolated=True, + ) + end = len(text) if quote_end is None else quote_end - 1 + return text[quote_start + 1 : end] + + +@functools.lru_cache(maxsize=8) +def mask_shell_heredoc_bodies(text: str) -> str: + masked = list(text) + pending: list[tuple[str, bool]] = [] + offset = 0 + code_keywords = { + "class", + "const", + "for", + "foreach", + "if", + "interface", + "namespace", + "new", + "record", + "return", + "struct", + "switch", + "using", + "var", + "while", + } + for line in text.splitlines(keepends=True): + content = line.rstrip("\r\n") + if pending: + delimiter, strip_tabs = pending[0] + comparison = content.lstrip("\t") if strip_tabs else content + for index in range(offset, offset + len(content)): + masked[index] = " " + if comparison == delimiter: + pending.pop(0) + offset += len(line) + continue + for match in re.finditer( + r"<<(?P-)?[ \t]*(?P['\"]?)" + r"(?P[A-Za-z_][A-Za-z0-9_]*)" + r"(?P=quote)", + content, + ): + quote = match.group("quote") + prefix = content[: match.start()] + shell_segment = re.split(r"[;|&]", prefix)[-1].strip() + first_word = ( + re.match(r"[A-Za-z_][A-Za-z0-9_.-]*", shell_segment) + if shell_segment + else None + ) + shell_like = ( + first_word is not None + and first_word.group(0) not in code_keywords + and re.fullmatch( + r"[A-Za-z_][A-Za-z0-9_.-]*" + r"(?:[ \t]+[^=(){}\[\];|&]+)*[ \t]*", + shell_segment, + ) + is not None + ) + if shell_like: + for index in range( + offset + match.start(), + offset + match.end(), + ): + masked[index] = " " + pending.append( + (match.group("delimiter"), match.group("strip") is not None) + ) + offset += len(line) + return "".join(masked) + + +@functools.lru_cache(maxsize=8) +def csharp_recognized_scope_intervals( + text: str, +) -> tuple[tuple[int, ...], tuple[int, ...]]: + masked = mask_csharp_evidence_prefix(mask_shell_heredoc_bodies(text)) + starts: list[int] = [] + ends: list[int] = [] + stack: list[bool] = [] + recognized_start: int | None = None + for cursor, char in enumerate(masked): + if char == "{": + recognized = False + if recognized_start is None: + quick_prefix = masked[max(0, cursor - 256) : cursor] + scope_candidate = ")" in quick_prefix or re.search( + r"\b(?:class|interface|namespace|record|struct)\b", + quick_prefix, + ) + if scope_candidate: + prefix = masked[max(0, cursor - 4096) : cursor + 1] + recognized = ( + re.search( + rf"(?:^|[;}}])\s*" + rf"{CSHARP_TYPE_PREFIX_PATTERN}\s*$", + prefix, + re.DOTALL, + ) + is not None + or re.search( + rf"(?:^|[;{{}}])\s*" + rf"{CSHARP_METHOD_PREFIX_PATTERN}\s*$", + prefix, + re.DOTALL, + ) + is not None + ) + stack.append(recognized) + if recognized: + recognized_start = cursor + elif char == "}" and stack: + recognized = stack.pop() + if recognized: + assert recognized_start is not None + starts.append(recognized_start) + ends.append(cursor + 1) + recognized_start = None + if recognized_start is not None: + starts.append(recognized_start) + ends.append(len(text)) + return tuple(starts), tuple(ends) + + +def csharp_recognized_scope_at(text: str, position: int) -> bool: + starts, ends = csharp_recognized_scope_intervals(text) + index = bisect.bisect_right(starts, position) - 1 + return index >= 0 and position < ends[index] + + +def csharp_interpolated_string_context( + text: str, + quote_start: int, +) -> bool: + masked = mask_csharp_evidence_prefix(mask_shell_heredoc_bodies(text)) + prefix = masked[ + max(0, quote_start - CSHARP_EVIDENCE_WINDOW) : quote_start + ] + statement_start = max( + prefix.rfind(";"), + prefix.rfind("}"), + ) + statement = prefix[statement_start + 1 :] + marker = re.search(r"(?:\$@|@\$)$", statement) + if marker is None: + return False + quote_end = csharp_quoted_literal_end( + text, + quote_start, + verbatim=True, + interpolated=True, + ) + if quote_end is None: + return False + terminator = re.match( + r"\s*(?:[,;)}:\[\].!?]|==|!=|<=|>=|>>>|>>|<<|&&|\|\||\?\?" + r"|\+\+|--|[+\-*/%&|^<>]|\b(?:as|is)\b)", + text[quote_end:], + ) + if terminator is None: + return False + expression_prefix = statement[: marker.start()] + typed_declaration = re.search( + r"\b(?:bool|byte|char|decimal|double|dynamic|float|int|long|object" + r"|sbyte|short|string|uint|ulong|ushort|var|" + r"[A-Z][A-Za-z0-9_.<>,?\[\]]*)\s+" + r"[A-Za-z_][A-Za-z0-9_]*\s*(?])=(?!=)\s*[^;]*$", + expression_prefix, + ) + csharp_statement = ( + re.search( + r"(?:^|[;{}])\s*(?:return\s+|new\s+" + r"[A-Za-z_][A-Za-z0-9_.<>,?\[\]]*\b)[^;]*$", + expression_prefix, + re.DOTALL, + ) + is not None + or re.search( + rf"(?:^|[;}}])\s*{CSHARP_TYPE_PREFIX_PATTERN}.*$", + expression_prefix, + re.DOTALL, + ) + is not None + or re.search( + rf"(?:^|[;{{}}])\s*{CSHARP_METHOD_PREFIX_PATTERN}.*$", + expression_prefix, + re.DOTALL, + ) + is not None + ) + assignment_operator = re.search( + r"(?])(?:=|[+\-*/%&|^]=|\?\?=|<<=|>>=|>>>=)\s*$", + expression_prefix, + ) + surrounding_prefix = prefix[max(0, statement_start - 4096) : statement_start + 1] + surrounding_csharp = ( + re.search( + r"(?:^|[;}\n])\s*using\s+(?:static\s+)?" + r"[A-Za-z_][A-Za-z0-9_.]*\s*;\s*$", + surrounding_prefix, + ) + is not None + or re.search( + rf"(?:^|[;}}])\s*{CSHARP_TYPE_PREFIX_PATTERN}.*$", + surrounding_prefix, + re.DOTALL, + ) + is not None + or re.search( + r"\b[A-Za-z_][A-Za-z0-9_.]*\([^;\r\n]*\)\s*;\s*$", + surrounding_prefix, + ) + is not None + or re.search( + rf"(?:^|[;{{}}])\s*{CSHARP_METHOD_PREFIX_PATTERN}.*$", + surrounding_prefix, + re.DOTALL, + ) + is not None + or re.search( + r"(?:^|[;}\n])\s*(?:bool|byte|char|decimal|double|dynamic|float" + r"|int|long|object|sbyte|short|string|uint|ulong|ushort|var|" + r"[A-Z][A-Za-z0-9_.<>,?\[\]]*)\s+" + r"[A-Za-z_][A-Za-z0-9_]*\s*(?])=(?!=)[^;]*;\s*$", + surrounding_prefix, + ) + is not None + ) + surrounding_csharp = surrounding_csharp or csharp_recognized_scope_at( + text, + quote_start, + ) + csharp_control_context = ( + re.search( + r"\b(?:catch|for|foreach|if|lock|switch|while)\s*" + r"\([^)]*\)\s*\{[^{}]*$", + expression_prefix, + re.DOTALL, + ) + is not None + or re.search( + r"\bif\s*\([^)]*(?:==|!=|<=|>=|&&|\|\||\bis\b)[^)]*$", + expression_prefix, + re.DOTALL, + ) + is not None + or re.search( + r"\b(?:do|else|finally|try)\s*\{[^{}]*$", + expression_prefix, + re.DOTALL, + ) + is not None + or ( + surrounding_csharp + and re.search( + r"\bif\s*\([^)]*$", + expression_prefix, + re.DOTALL, + ) + is not None + ) + ) + unmatched_parenthesis = ( + expression_prefix.count("(") > expression_prefix.count(")") + ) + open_parenthesized_call = ( + unmatched_parenthesis + and re.search( + r"\b[A-Za-z_][A-Za-z0-9_.]*\s*\([^()]*$", + expression_prefix, + re.DOTALL, + ) + is not None + ) + spaced_assignment = ( + re.search( + r"\b[A-Za-z_][A-Za-z0-9_]*[ \t]+" + r"(?])=(?!=)[ \t]*$", + expression_prefix, + ) + is not None + ) + standalone_content = csharp_verbatim_string_content( + text, + quote_start, + ) + standalone_fields = re.findall(r"\{([^{}]*)\}", standalone_content) + standalone_reference_assignment = ( + spaced_assignment + and bool(standalone_fields) + and standalone_content.count("{") == len(standalone_fields) + and standalone_content.count("}") == len(standalone_fields) + and all( + CSHARP_STANDALONE_REFERENCE_PATTERN.fullmatch(field) + is not None + for field in standalone_fields + ) + ) + expression_evidence = ( + csharp_statement + or csharp_control_context + or typed_declaration is not None + or "=>" in expression_prefix + or open_parenthesized_call + # Standalone spaced assignments are ambiguous with shell commands, so + # recover only ordinary credential references in this C#-only shape. + or standalone_reference_assignment + or (assignment_operator is not None and surrounding_csharp) + ) + return expression_evidence + + +def quote_prefix_matches( + text: str, + quote_start: int, + pattern: str, + *, + limit: int = 4, +) -> bool: + prefix_tail = text[max(0, quote_start - limit) : quote_start] + return re.search(pattern, prefix_tail) is not None + + +def csharp_interpolated_marker(text: str, quote_start: int) -> bool: + return text[max(0, quote_start - 2) : quote_start] in {"$@", "@$"} + + +@functools.lru_cache(maxsize=8) +def csharp_interpolated_verbatim_spans( + text: str, +) -> tuple[tuple[int, ...], tuple[int, ...]]: + masked = mask_csharp_evidence_prefix(mask_shell_heredoc_bodies(text)) + starts: list[int] = [] + ends: list[int] = [] + for marker in re.finditer(r"(?:\$@|@\$)(?=\")", text): + quote_start = marker.end() + if masked[marker.start() : quote_start] != text[marker.start() : quote_start]: + continue + quote_end = csharp_quoted_literal_end( + text, + quote_start, + verbatim=True, + interpolated=True, + ) + if quote_end is None: + continue + starts.append(quote_start) + ends.append(quote_end) + return tuple(starts), tuple(ends) + + +def explicit_csharp_interpolated_context( + text: str, + position: int, +) -> tuple[str, int] | None: + starts, ends = csharp_interpolated_verbatim_spans(text) + index = bisect.bisect_right(starts, position) - 1 + if index < 0 or position >= ends[index]: + return None + quote_start = starts[index] + if not csharp_interpolated_string_context(text, quote_start): + return None + return '"', quote_start + + +def bounded_line_start( + text: str, + position: int, + *, + limit: int = 4096, +) -> int: + search_start = max(0, position - limit) + found = max( + text.rfind("\n", search_start, position), + text.rfind("\r", search_start, position), + ) + return found if found >= 0 else search_start - 1 + + +def uri_authority_end( + text: str, + start: int, + context: tuple[str, int] | None, +) -> int: + outer_quote = context[0] if context is not None else None + quote_start = context[1] if context is not None else -1 + brace_interpolation = ( + outer_quote in {'"', "'", '"""', "'''"} + and ( + quote_prefix_matches( + text, + quote_start, + r"(?i)(?:^|[^A-Za-z0-9_])(?:f|fr|rf|(? tuple[tuple[int, int, int, tuple[str, int] | None], ...]: + matches = list(URI_SCHEME_PATTERN.finditer(text)) + contexts = string_contexts_at( + text, + {match.start() for match in matches}, + ) + return tuple( + ( + match.start(), + match.end(), + uri_authority_end( + text, + match.end(), + contexts.get(match.start()), + ), + contexts.get(match.start()), + ) + for match in matches + ) + + +def credentialed_uri_risk( + text: str, + authorities: tuple[ + tuple[int, int, int, tuple[str, int] | None], + ..., + ] | None = None, +) -> bool: + for authority_range in ( + authorities if authorities is not None else uri_authority_ranges(text) + ): + credential = uri_authority_credential(text, authority_range) + if credential is None: + continue + if ( + credential.has_password + and uri_userinfo_literal_risk( + credential.username, + allow_plus_address=True, + ) + and not uri_password_is_interpolated( + text, + credential.scheme_start, + credential.username, + credential.host, + credential.context, + ) + ): + return True + if uri_credential_value_risk(text, credential): + return True + return False + + +class UriAuthorityCredential(NamedTuple): + username: str + value: str + host: str + has_password: bool + empty_password: bool + scheme_start: int + context: tuple[str, int] | None + value_start: int + value_end: int + + +def uri_authority_credential( + text: str, + authority_range: tuple[ + int, + int, + int, + tuple[str, int] | None, + ], +) -> UriAuthorityCredential | None: + scheme_start, authority_start, authority_end, context = authority_range + authority = text[authority_start:authority_end] + userinfo, authority_separator, host = authority.rpartition("@") + if not authority_separator: + return None + username, password_separator, password = userinfo.partition(":") + has_password = bool(password_separator and password) + empty_password = bool(password_separator and not password) + value = password if has_password else (userinfo if not password_separator else username) + value_start = ( + authority_start + len(username) + 1 + if has_password + else authority_start + ) + return UriAuthorityCredential( + username, + value, + host, + has_password, + empty_password, + scheme_start, + context, + value_start, + value_start + len(value), + ) + + +def interpolated_empty_password_uri_ranges( + text: str, + authorities: tuple[ + tuple[int, int, int, tuple[str, int] | None], + ..., + ], +) -> tuple[tuple[int, int], ...]: + safe: list[tuple[int, int]] = [] + for authority_range in authorities: + credential = uri_authority_credential(text, authority_range) + if credential is None or not credential.empty_password: + continue + if uri_password_is_interpolated( + text, + credential.scheme_start, + credential.value, + credential.host, + credential.context, + ): + safe.append( + (credential.value_start, credential.value_end) + ) + return tuple(safe) + + +def mask_ranges( + text: str, + ranges: tuple[tuple[int, int], ...], +) -> str: + masked = list(text) + for start, end in ranges: + masked[start:end] = " " * (end - start) + return "".join(masked) + + +def position_in_ranges( + position: int, + ranges: tuple[tuple[int, int], ...], +) -> bool: + return any( + start <= position < end + for start, end in ranges + ) + + +def secret_assignment_matches( + pattern: re.Pattern[str], + text: str, + masked_text: str, + safe_ranges: tuple[tuple[int, int], ...], +) -> tuple[re.Match[str], ...]: + matches: list[re.Match[str]] = [] + spans: set[tuple[int, int]] = set() + for match in pattern.finditer(text): + if position_in_ranges(match.start(), safe_ranges): + continue + matches.append(match) + spans.add(match.span()) + for match in pattern.finditer(masked_text): + if match.span() not in spans: + matches.append(match) + return tuple(matches) + + +def string_contexts_at( + text: str, + positions: set[int], +) -> dict[int, tuple[str, int] | None]: + contexts: dict[int, tuple[str, int] | None] = {} + quote: str | None = None + quote_start = -1 + verbatim_quote = False + escaped = False + line_comment = False + block_comment = False + brace_depth = 0 + class_depths: list[int] = [] + pending_class = False + cursor = 0 + while cursor < len(text) and len(contexts) < len(positions): + if cursor in positions: + contexts[cursor] = explicit_csharp_interpolated_context( + text, + cursor, + ) or ( + None if quote is None else (quote, quote_start) + ) + char = text[cursor] + next_char = text[cursor + 1] if cursor + 1 < len(text) else "" + if line_comment: + if char == "\n": + line_comment = False + cursor += 1 + continue + if block_comment: + if char == "*" and next_char == "/": + block_comment = False + cursor += 2 + else: + cursor += 1 + continue + if quote is not None: + if escaped: + escaped = False + elif ( + verbatim_quote + and quote == '"' + and text.startswith('""', cursor) + ): + cursor += 2 + continue + elif char == "\\" and not verbatim_quote: + escaped = True + elif text.startswith(quote, cursor): + quote_length = len(quote) + quote = None + quote_start = -1 + verbatim_quote = False + cursor += quote_length + continue + elif ( + regex_end := javascript_regex_literal_end(text, cursor) + ) is not None: + cursor = regex_end + continue + elif text.startswith('"""', cursor): + quote = '"""' + quote_start = cursor + cursor += 3 + continue + elif text.startswith("'''", cursor): + quote = "'''" + quote_start = cursor + cursor += 3 + continue + elif char == "/" and next_char == "/": + line_comment = True + cursor += 2 + continue + elif char == "/" and next_char == "*": + block_comment = True + cursor += 2 + continue + elif char == "#" and not javascript_private_member_marker( + text, + cursor, + allow_bare=bool(class_depths), + ): + line_comment = True + elif char in {'"', "'", "`"}: + quote = char + quote_start = cursor + verbatim_quote = ( + char == '"' + and text[max(0, cursor - 2) : cursor] in {"$@", "@$"} + ) + elif char.isalpha() or char in "_$": + word_end = cursor + 1 + while word_end < len(text) and ( + text[word_end].isalnum() or text[word_end] in "_$" + ): + word_end += 1 + if text[cursor:word_end] == "class": + pending_class = True + cursor = word_end + continue + elif char == "{": + brace_depth += 1 + if pending_class: + class_depths.append(brace_depth) + pending_class = False + elif char == "}": + if class_depths and class_depths[-1] == brace_depth: + class_depths.pop() + brace_depth = max(0, brace_depth - 1) + elif char == ";": + pending_class = False + cursor += 1 + for position in positions - contexts.keys(): + contexts[position] = None if quote is None else (quote, quote_start) + return contexts + + +def uri_password_is_interpolated( + text: str, + scheme_start: int, + password: str, + host: str, + context: tuple[str, int] | None, +) -> bool: + if uri_placeholder_password_is_safe(password, host): + return True + if context is not None: + quote, quote_start = context + if quote == "`": + if any( + pattern.fullmatch(password) + for pattern in URI_PASSWORD_REFERENCE_PATTERNS[1:2] + + URI_PASSWORD_REFERENCE_PATTERNS[3:4] + ): + return True + return dynamic_uri_expression(password, "${", "}") + if quote == '"' and ( + quote_prefix_matches(text, quote_start, r"(? bool: + normalized_password = password.lower() + if normalized_password in URI_PASSWORD_PLACEHOLDER_VALUES: + return True + normalized_host = host.lower() + if normalized_host.startswith("[") and "]" in normalized_host: + normalized_host = normalized_host[1 : normalized_host.index("]")] + elif normalized_host.count(":") == 1: + normalized_host = normalized_host.split(":", 1)[0] + localhost = ( + normalized_host in {"127.0.0.1", "::1", "localhost"} + or normalized_host.endswith(".localhost") + ) + return localhost and normalized_password in { + *SECRET_PLACEHOLDER_VALUES, + "password", + } + + +def uri_userinfo_literal_risk( + value: str, + *, + allow_plus_address: bool = False, +) -> bool: + if value.lower() in URI_PASSWORD_PLACEHOLDER_VALUES: + return False + if value.startswith(("$", "{")): + return True + credential_name = URI_CREDENTIAL_NAME_PATTERN.search(value) is not None + structured_username = ( + re.fullmatch( + r"(?=[^\r\n]*[._-])" + r"[A-Za-z][A-Za-z0-9]*(?:[._-][A-Za-z0-9]+)+", + value, + ) + is not None + ) + character_classes = sum( + ( + any(char.islower() for char in value), + any(char.isupper() for char in value), + any(char.isdigit() for char in value), + any(not char.isalnum() for char in value), + ) + ) + opaque_alphanumeric = ( + re.fullmatch(r"[A-Za-z0-9]{20,}", value) is not None + and character_classes >= 3 + ) + opaque_hex = ( + re.fullmatch(r"[0-9A-Fa-f]{32,}", value) is not None + or re.fullmatch( + r"[0-9A-Fa-f]{8}-" + r"(?:[0-9A-Fa-f]{4}-){3}" + r"[0-9A-Fa-f]{12}", + value, + ) + is not None + ) + plus_local, plus_separator, plus_tag = value.rpartition("+") + local_case_transitions = sum( + left.islower() != right.islower() + for left, right in zip(plus_local, plus_local[1:]) + if left.isalpha() and right.isalpha() + ) + tag_case_transitions = sum( + left.islower() != right.islower() + for left, right in zip(plus_tag, plus_tag[1:]) + if left.isalpha() and right.isalpha() + ) + plus_address_username = ( + bool(plus_separator) + and ( + plus_local == plus_local.lower() + or re.search(r"[._-]", plus_local) is not None + ) + and re.fullmatch( + r"[A-Za-z]+[0-9]*(?:[._-][A-Za-z]+[0-9]*)*", + plus_local, + ) + is not None + and ( + re.fullmatch(r"[0-9]{1,4}", plus_tag) is not None + or re.fullmatch( + r"[A-Za-z]+[0-9]{0,4}" + r"(?:[._-](?:[A-Za-z]+[0-9]{0,4}|[0-9]{1,4}))*", + plus_tag, + ) + is not None + ) + and local_case_transitions <= ( + 8 if re.search(r"[._-]", plus_local) else 4 + ) + and tag_case_transitions <= 4 + ) + opaque_plus_tag = ( + bool(plus_separator) + and len(plus_local) >= 16 + and len(plus_tag) >= 16 + and re.fullmatch(r"[A-Za-z0-9]+", plus_tag) is not None + and any(char.isdigit() for char in plus_tag) + and local_case_transitions >= 6 + and tag_case_transitions >= 6 + ) + return len(value) >= 20 and ( + credential_name + or opaque_alphanumeric + or opaque_hex + or opaque_plus_tag + or ( + character_classes >= 4 + and not structured_username + and not (allow_plus_address and plus_address_username) + ) + ) + + +def uri_named_credential_reference(password: str) -> bool: + if not any( + pattern.fullmatch(password) + for pattern in URI_PASSWORD_REFERENCE_PATTERNS[:3] + ): + return False + name = password + if name.startswith("${") and name.endswith("}"): + name = name[2:-1] + elif name.startswith(("$", "{")): + name = name[1:-1] if name.startswith("{") else name[1:] + return ( + re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", name) is not None + and URI_CREDENTIAL_NAME_PATTERN.search(name) is not None + ) + + +def dynamic_uri_expression( + password: str, + prefix: str, + suffix: str, +) -> bool: + if not password.startswith(prefix) or not password.endswith(suffix): + return False + expression = password[len(prefix) : -len(suffix)] + return ( + URI_CREDENTIAL_REFERENCE_PATTERN.fullmatch(expression) is not None + or URI_COMPUTED_REFERENCE_PATTERN.fullmatch(expression) is not None + ) + + +@functools.lru_cache(maxsize=64) +def quoted_string_end( + text: str, + quote: str, + quote_start: int, + *, + doubled_quote_escape: bool = False, +) -> int | None: + quote_end = quote_start + len(quote) + if doubled_quote_escape: + doubled_quote = quote + quote + while quote_end < len(text): + if text.startswith(doubled_quote, quote_end): + quote_end += len(doubled_quote) + elif text.startswith(quote, quote_end): + return quote_end + len(quote) + else: + quote_end += 1 + return None + escaped = False + while quote_end < len(text): + char = text[quote_end] + if escaped: + escaped = False + quote_end += 1 + elif char == "\\": + escaped = True + quote_end += 1 + elif text.startswith(quote, quote_end): + quote_end += len(quote) + break + else: + quote_end += 1 + else: + return None + return quote_end + + +def uri_password_is_format_placeholder( + text: str, + password: str, + quote: str, + quote_start: int, +) -> bool: + quote_end = quoted_string_end(text, quote, quote_start) + if quote_end is None: + return False + prefix_tail = text[max(0, quote_start - 32) : quote_start] + suffix = text[quote_end : quote_end + 8192] + formatter = re.search( + r"(?:\bfmt\.Sprintf|\bformat!)\(\s*$", + prefix_tail, + ) + if formatter is not None: + arguments = re.match(r"\s*,\s*(?P.*?)\s*\)", suffix, re.DOTALL) + if arguments is not None and format_arguments_are_references( + arguments.group("args") + ): + return password == "%s" or re.fullmatch( + r"\{(?:[A-Za-z_][A-Za-z0-9_]*|[0-9]*)\}", + password, + ) is not None + if password == "%s": + python_percent_format = re.match( + rf"\s*%\s*(?:" + rf"(?P{URI_CREDENTIAL_REFERENCE_TEXT})\b" + rf"|\((?P[^()]*)\)" + rf")", + suffix, + ) + if python_percent_format is not None: + arguments = ( + [python_percent_format.group("single")] + if python_percent_format.group("single") is not None + else split_top_level_call_arguments( + python_percent_format.group("tuple") or "" + ) + ) + return all( + argument is not None + and URI_CREDENTIAL_REFERENCE_PATTERN.fullmatch(argument.strip()) + for argument in arguments + ) + return False + field_match = re.fullmatch( + r"\{(?P[A-Za-z_][A-Za-z0-9_]*|[0-9]*)\}", + password, + ) + if field_match is not None: + field = field_match.group("field") + format_call = re.match( + r"\s*\.format\s*\((?P.*?)\)", + suffix, + re.DOTALL, + ) + if format_call is not None and format_arguments_are_references( + format_call.group("args") + ): + return True + return False + + +def uri_credential_value_risk( + text: str, + credential: UriAuthorityCredential, +) -> bool: + if uri_password_is_interpolated( + text, + credential.scheme_start, + credential.value, + credential.host, + credential.context, + ): + return False + return credential.has_password or uri_userinfo_literal_risk( + credential.value, + allow_plus_address=True, + ) + + +def format_arguments_are_references(arguments: str) -> bool: + values = split_top_level_call_arguments(arguments) + if not values or any(not value.strip() for value in values): + return False + for value in values: + expression = value.strip() + named = re.fullmatch( + rf"[A-Za-z_][A-Za-z0-9_]*\s*=\s*" + rf"(?P{URI_CREDENTIAL_REFERENCE_TEXT})", + expression, + ) + if named is not None: + expression = named.group("value") + if URI_CREDENTIAL_REFERENCE_PATTERN.fullmatch(expression) is None: + return False + return True + + +def config_assignment_context( + text: str, + position: int, +) -> tuple[str, str] | None: + line_start = bounded_line_start(text, position) + prefix = text[line_start + 1 : position] + match = re.fullmatch( + r"\s*(?P#\s*)?(?:-\s+)?[\"']?" + r"(?P(?:[A-Za-z_][A-Za-z0-9_.-]*)?(?:dsn|uri|url))" + r"[\"']?\s*(?P[:=])\s*[\"']?", + prefix, + re.IGNORECASE, + ) + if match is None or ( + match.group("separator") != ":" and match.group("comment") is None + ): + return None + return match.group("key"), match.group("separator") + + +def config_assignment_prefix(text: str, position: int) -> bool: + return config_assignment_context(text, position) is not None + + +def config_uri_reference_is_safe( + text: str, + position: int, + password: str, + *, + allow_lowercase_key: bool, +) -> bool: + context = config_assignment_context(text, position) + if context is None: + return False + key, separator = context + syntactic_reference = any( + pattern.fullmatch(password) + for pattern in URI_PASSWORD_REFERENCE_PATTERNS[:2] + ) + return syntactic_reference and ( + uri_named_credential_reference(password) + or key == key.upper() + or (allow_lowercase_key and separator == ":") + ) + + +def shell_assignment_prefix(text: str) -> bool: + match = re.fullmatch( + r"\s*(?:-\s+)?(?Pexport\s+)?(?P[A-Za-z_][A-Za-z0-9_]*)=", + text, + ) + return match is not None and ( + match.group("export") is not None + or match.group("name") == match.group("name").upper() + ) + + +def powershell_assignment_prefix(text: str) -> bool: + return ( + re.match( + r"(?i)\s*(?:" + r"\[[^\]\r\n]+\]\s*\$[A-Za-z_][A-Za-z0-9_]*" + r"|\$env:[A-Za-z_][A-Za-z0-9_]*)\s*=", + text, + ) + is not None + ) + + +def shell_command_prefix(text: str, position: int) -> bool: + line_start = bounded_line_start(text, position) + prefix = text[line_start + 1 : position] + match = re.fullmatch( + r"\s*(?P[A-Za-z0-9_./-]+)" + r"(?:[ \t]+[^ \t\"'`]+)*[ \t]+", + prefix, + ) + if match is None: + return False + tokens = prefix.split() + while tokens and tokens[0].rsplit("/", 1)[-1] in SHELL_COMMAND_WRAPPERS: + wrapper = tokens.pop(0).rsplit("/", 1)[-1] + while tokens and ( + tokens[0].startswith("-") + or (wrapper == "env" and "=" in tokens[0]) + ): + tokens.pop(0) + if not tokens: + return False + command = tokens[0].rsplit("/", 1)[-1] + return ( + command == command.lower() + and command not in NON_SHELL_COMMAND_WORDS + and re.fullmatch(r"[a-z0-9][a-z0-9._+-]*", command) is not None + and not any( + token in {"=", "=>", ":", "::"} or token.endswith(("=", "=>")) + for token in tokens[1:] + ) + ) + + +def secret_literal_risk( + expression: str, + minimum_length: int = 12, + *, + javascript_dialect: str | None = None, +) -> bool: + if credentialed_uri_risk(expression) or basic_authorization_risk(expression) or any( + pattern.search(expression) for pattern in SECRET_VALUE_PATTERNS + ): + return True + value_pattern = re.compile( + rf'"(?P[^"\r\n]{{{minimum_length},}})"' + rf"|'(?P[^'\r\n]{{{minimum_length},}})'" + rf"|`(?P[^`\r\n]{{{minimum_length},}})`" + rf"|(?P[A-Za-z0-9_./+=:@#$%&*!?-]{{{max(20, minimum_length)},}})" + ) + for match in value_pattern.finditer(expression): + value = next(group for group in match.groups() if group is not None) + if value.lower() in SECRET_PLACEHOLDER_VALUES: + continue + if match.group("backtick") is not None and any( + pattern.fullmatch(value) + for pattern in BACKTICK_SECRET_REFERENCE_PATTERNS + ): + continue + reference_patterns = ( + UNQUOTED_SECRET_REFERENCE_PATTERNS + if match.group("bare") is not None + else QUOTED_SECRET_REFERENCE_PATTERNS + ) + if any(pattern.fullmatch(value) for pattern in reference_patterns): + continue + if match.group("bare") is not None: + if re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", value): + continue + if ( + javascript_dialect is not None + and re.fullmatch(r"[A-Za-z_$][A-Za-z0-9_$]*:", value) + ): + # Object/type property names are syntax, not credential content. + # Any literal value after the colon is scanned independently. + continue + if ( + javascript_dialect is not None + and SOURCE_CODE_REFERENCE_ROOT_PATTERN.fullmatch(value) + ): + continue + suffix = expression[match.end() :].lstrip() + if suffix.startswith("("): + continue + return True + return False + + +def fallback_secret_risk( + text: str, + minimum_length: int = 8, + *, + javascript_dialect: str | None = None, +) -> bool: + return secret_literal_risk( + fallback_expression( + text, + typescript=javascript_dialect == "typescript", + ), + minimum_length=minimum_length, + javascript_dialect=javascript_dialect, + ) + + +def synthetic_secret_fixture(value: str, key: str) -> bool: + normalized = value.casefold() + if normalized in SECRET_PLACEHOLDER_VALUES: + return True + if len(normalized) > 80: + return False + camel_split_key = re.sub( + r"(?<=[a-z0-9])(?=[A-Z])", + "-", + key.strip("\"'"), + ) + normalized_key = re.sub( + r"[^a-z0-9]+", + "-", + camel_split_key.casefold(), + ).strip("-") + return any( + normalized == f"{prefix}-{normalized_key}" + for prefix in SYNTHETIC_SECRET_PREFIXES + ) + + +def safe_secret_assignment_suffix( + text: str, + end: int, + *, + javascript_dialect: str | None = None, +) -> bool: + cursor = end + raw_diff = text.startswith("diff --git ") + while cursor < len(text): + while cursor < len(text) and text[cursor] in " \t\r": + cursor += 1 + if cursor >= len(text): + return True + if cursor < len(text) and text[cursor] == "\n": + cursor += 1 + while cursor < len(text): + if raw_diff and text[cursor : cursor + 1] in {"+", "-", " "}: + cursor += 1 + while cursor < len(text) and text[cursor] in " \t\r": + cursor += 1 + if text.startswith("//", cursor) or text.startswith("#", cursor): + newline = text.find("\n", cursor) + if newline < 0: + return True + cursor = newline + 1 + continue + if text.startswith("/*", cursor): + comment_end = text.find("*/", cursor + 2) + if comment_end < 0: + return False + cursor = comment_end + 2 + continue + break + suffix = text[cursor:] + if ( + suffix.startswith(("||", "&&", "??", "+")) + or (suffix.startswith("?") and not suffix.startswith("?.")) + or re.match(r"(?:or|and)\b", suffix) is not None + ): + return not fallback_secret_risk( + suffix, + javascript_dialect=javascript_dialect, + ) + return True + if text.startswith("//", cursor) or text.startswith("#", cursor): + cursor = text.find("\n", cursor) + if cursor < 0: + return True + continue + if text.startswith("/*", cursor): + comment_end = text.find("*/", cursor + 2) + if comment_end < 0: + return False + cursor = comment_end + 2 + continue + suffix = text[cursor:] + if ( + suffix.startswith(("||", "&&", "??", "+")) + or (suffix.startswith("?") and not suffix.startswith("?.")) + or re.match(r"(?:or|and|if|unless)\b", suffix) is not None + ): + return not fallback_secret_risk( + suffix, + javascript_dialect=javascript_dialect, + ) + if text[cursor] in ",;)]}": + return True + if text[cursor] in {'"', "'", "`"}: + after_quote = cursor + 1 + while after_quote < len(text) and text[after_quote] in " \t\r": + after_quote += 1 + if after_quote >= len(text) or text[after_quote] in ",;)]}": + return True + quote = text[cursor] + cursor += 1 + escaped = False + while cursor < len(text): + char = text[cursor] + cursor += 1 + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == quote: + break + continue + cursor += 1 + return True + + +def split_top_level_call_arguments(text: str) -> list[str]: + arguments: list[str] = [] + start = 0 + stack: list[str] = [] + pairs = {"(": ")", "[": "]", "{": "}"} + quote: str | None = None + escaped = False + line_comment = False + block_comment = False + index = 0 + while index < len(text): + char = text[index] + next_char = text[index + 1] if index + 1 < len(text) else "" + if line_comment: + if char == "\n": + line_comment = False + index += 1 + continue + if block_comment: + if char == "*" and next_char == "/": + block_comment = False + index += 2 + else: + index += 1 + continue + if quote is not None: + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == quote: + quote = None + index += 1 + continue + regex_end = javascript_regex_literal_end(text, index) + if regex_end is not None: + index = regex_end + elif char == "/" and next_char == "/": + line_comment = True + index += 2 + elif char == "/" and next_char == "*": + block_comment = True + index += 2 + elif char in {'"', "'", "`"}: + quote = char + index += 1 + elif char in pairs: + stack.append(pairs[char]) + index += 1 + elif stack and char == stack[-1]: + stack.pop() + index += 1 + elif char == "," and not stack: + arguments.append(text[start:index]) + start = index + 1 + index += 1 + else: + index += 1 + arguments.append(text[start:]) + return arguments + + +def javascript_private_member_marker( + text: str, + index: int, + *, + allow_bare: bool = False, +) -> bool: + next_char = text[index + 1] if index + 1 < len(text) else "" + return ( + text[index : index + 1] == "#" + and bool(next_char) + and (next_char.isalpha() or next_char in "_$") + and ( + allow_bare + or (index > 0 and text[index - 1] == ".") + ) + ) + + +@functools.lru_cache(maxsize=16) +def javascript_control_contexts(text: str) -> frozenset[int]: + closes: set[int] = set() + stack: list[str | None] = [] + quote: str | None = None + escaped = False + line_comment = False + block_comment = False + last_word: str | None = None + prior_word: str | None = None + last_word_is_member = False + after_dot = False + cursor = 0 + while cursor < len(text): + char = text[cursor] + next_char = text[cursor + 1] if cursor + 1 < len(text) else "" + if line_comment: + if char == "\n": + line_comment = False + cursor += 1 + continue + if block_comment: + if char == "*" and next_char == "/": + block_comment = False + cursor += 2 + else: + cursor += 1 + continue + if quote is not None: + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == quote: + quote = None + cursor += 1 + continue + regex_end = javascript_regex_literal_end( + text, + cursor, + control_conditions=False, + known_control_closes=closes, + ) + if regex_end is not None: + cursor = regex_end + last_word = None + prior_word = None + last_word_is_member = False + after_dot = False + elif char == "/" and next_char == "/": + line_comment = True + cursor += 2 + elif char == "/" and next_char == "*": + block_comment = True + cursor += 2 + elif char in {'"', "'", "`"}: + quote = char + cursor += 1 + last_word = None + prior_word = None + last_word_is_member = False + after_dot = False + elif char.isalpha() or char in "_$": + word_end = cursor + 1 + while word_end < len(text) and ( + text[word_end].isalnum() or text[word_end] in "_$" + ): + word_end += 1 + word = text[cursor:word_end] + prior_word = last_word if not after_dot else None + last_word = word + last_word_is_member = after_dot + after_dot = False + cursor = word_end + elif char == "(": + control_kind: str | None = None + if not last_word_is_member: + if last_word in {"if", "while", "with"}: + control_kind = "control" + elif last_word == "for" or ( + prior_word == "for" and last_word == "await" + ): + control_kind = "for" + stack.append(control_kind) + last_word = None + prior_word = None + last_word_is_member = False + after_dot = False + cursor += 1 + elif char == ")": + if not stack: + cursor += 1 + continue + if stack.pop() is not None: + closes.add(cursor) + last_word = None + prior_word = None + last_word_is_member = False + after_dot = False + cursor += 1 + elif char == ".": + last_word = None + prior_word = None + last_word_is_member = False + after_dot = True + cursor += 1 + elif javascript_private_member_marker( + text, + cursor, + allow_bare=True, + ): + last_word = None + prior_word = None + last_word_is_member = False + after_dot = True + cursor += 1 + elif char == ";": + last_word = None + prior_word = None + last_word_is_member = False + after_dot = False + cursor += 1 + elif char.isspace(): + cursor += 1 + else: + last_word = None + prior_word = None + last_word_is_member = False + after_dot = False + cursor += 1 + return frozenset(closes) + + +@functools.lru_cache(maxsize=16) +def javascript_control_condition_closes(text: str) -> frozenset[int]: + return javascript_control_contexts(text) + + +def javascript_regex_literal_end( + text: str, + start: int, + *, + control_conditions: bool = True, + known_control_closes: set[int] | frozenset[int] | None = None, +) -> int | None: + if text[start : start + 1] != "/" or text[start + 1 : start + 2] in { + "/", + "*", + }: + return None + previous = start - 1 + while previous >= 0 and text[previous].isspace(): + previous -= 1 + if ( + previous >= 2 + and text[previous - 2 : previous + 1] == "..." + and (previous == 2 or text[previous - 3] != ".") + ): + previous = -1 + if previous >= 0 and text[previous] == ")": + closes_control_condition = ( + previous in known_control_closes + if known_control_closes is not None + else ( + control_conditions + and previous in javascript_control_condition_closes(text) + ) + ) + if closes_control_condition: + previous = -1 + if previous >= 0 and text[previous] not in "([{:;,=!?&|+-*%^~<>": + word_start = previous + while word_start >= 0 and ( + text[word_start].isalnum() or text[word_start] in "_$" + ): + word_start -= 1 + keyword = text[word_start + 1 : previous + 1] + expression_keyword = keyword in { + "case", + "default", + "delete", + "do", + "else", + "extends", + "in", + "instanceof", + "new", + "return", + "throw", + "typeof", + "void", + } + if ( + not expression_keyword + or (word_start >= 0 and text[word_start] == ".") + ): + return None + if ( + previous > 0 + and text[previous] in "+-" + and text[previous - 1] == text[previous] + ): + return None + if text[previous : previous + 1] == "!": + before = previous - 1 + while before >= 0 and text[before].isspace(): + before -= 1 + if before >= 0 and ( + text[before].isalnum() or text[before] in "_$)]}" + ): + return None + escaped = False + character_class = False + cursor = start + 1 + while cursor < len(text): + char = text[cursor] + if char in "\r\n": + return None + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == "[": + character_class = True + elif char == "]" and character_class: + character_class = False + elif char == "/" and not character_class: + cursor += 1 + while cursor < len(text) and text[cursor].isalpha(): + cursor += 1 + return cursor + cursor += 1 + return None + + +def regex_tail_end(text: str, start: int, limit: int) -> int | None: + escaped = False + character_class = False + cursor = start + while cursor < limit: + char = text[cursor] + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == "[": + character_class = True + elif char == "]" and character_class: + character_class = False + elif char == "/" and not character_class: + return cursor + 1 + cursor += 1 + return None + + +def previous_regex_delimiter(text: str, start: int, lower: int) -> int | None: + character_class = False + cursor = start - 1 + while cursor >= lower: + char = text[cursor] + backslashes = 0 + previous = cursor - 1 + while previous >= lower and text[previous] == "\\": + backslashes += 1 + previous -= 1 + escaped = backslashes % 2 == 1 + if not escaped: + if char == "]": + character_class = True + elif char == "[" and character_class: + character_class = False + elif char == "/" and not character_class: + return cursor + cursor -= 1 + return None + + +def text_without_ranges( + text: str, + start: int, + end: int, + ranges: list[tuple[int, int]], +) -> str: + parts: list[str] = [] + cursor = start + for range_start, range_end in ranges: + if range_end <= cursor or range_start >= end: + continue + if cursor < range_start: + parts.append(text[cursor:range_start]) + parts.append(" ") + cursor = max(cursor, range_end) + if cursor < end: + parts.append(text[cursor:end]) + return "".join(parts) + + +def premature_regex_call_tail( + text: str, + call_start: int, + cursor: int, +) -> tuple[str, int] | None: + line_end = len(text) + for delimiter in ("\n", "\r"): + found = text.find(delimiter, cursor) + if found >= 0: + line_end = min(line_end, found) + line_start = max( + text.rfind("\n", 0, cursor), + text.rfind("\r", 0, cursor), + ) + 1 + search_start = max(call_start, line_start) + nearest = previous_regex_delimiter(text, cursor, search_start) + candidates = [] + if nearest is not None: + candidates.append(nearest) + previous = previous_regex_delimiter(text, nearest, search_start) + if previous is not None: + candidates.append(previous) + for regex_start in candidates: + regex_end = regex_tail_end(text, regex_start + 1, line_end) + if regex_end is None or ")" not in text[regex_start + 1 : regex_end - 1]: + continue + depth = 0 + quote: str | None = None + escaped = False + line_comment = False + block_comment = False + regex_ranges = [(regex_start, regex_end)] + index = call_start + while index < len(text): + char = text[index] + next_char = text[index + 1] if index + 1 < len(text) else "" + if line_comment: + if char == "\n": + line_comment = False + index += 1 + continue + if block_comment: + if char == "*" and next_char == "/": + block_comment = False + index += 2 + else: + index += 1 + continue + if quote is not None: + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == quote: + quote = None + index += 1 + continue + if index == regex_start: + index = regex_end + elif ( + later_regex_end := javascript_regex_literal_end(text, index) + ) is not None: + regex_ranges.append((index, later_regex_end)) + index = later_regex_end + elif char == "/" and next_char == "/": + line_comment = True + index += 2 + elif char == "/" and next_char == "*": + block_comment = True + index += 2 + # This recovery scans JavaScript; `#name` is a private identifier, + # so only JavaScript's slash-delimited comment forms apply here. + elif char in {'"', "'", "`"}: + quote = char + index += 1 + elif char == "(": + depth += 1 + index += 1 + elif char == ")": + depth -= 1 + index += 1 + if depth == 0: + return ( + ( + text_without_ranges( + text, + cursor, + index, + regex_ranges, + ), + index, + ) + if index > cursor + else None + ) + else: + index += 1 + return None + + +def safe_credential_lookup_argument( + call_target: str, + argument: str, + argument_index: int, +) -> bool: + if argument_index != 0: + return False + normalized_target = call_target.replace("?.", ".") + result_lookup = normalized_target in { + "response.json().get", + "response.get", + "result.get", + } + if ( + not result_lookup + and normalized_target not in {"os.getenv", "os.environ.get"} + and normalized_target != "headers.get" + and not normalized_target.endswith(".headers.get") + ): + return False + match = re.fullmatch(r"\s*([\"'])([^\"'\r\n]+)\1\s*", argument) + if match is None: + return False + key = match.group(2) + if result_lookup and any( + pattern.search(key) for pattern in SECRET_VALUE_PATTERNS + ): + return False + return ( + ( + result_lookup + and key.casefold() + in { + "access_token", + "api_key", + "auth_token", + "client_secret", + "credential", + "credentials", + "id_token", + "password", + "refresh_token", + "secret", + "token", + } + ) + or ( + not result_lookup + and re.fullmatch(r"[A-Z][A-Z0-9_]{2,}", key) is not None + ) + or key.casefold() in {"authorization", "proxy-authorization"} + ) + + +def mask_javascript_comments(text: str) -> str: + masked = list(text) + cursor = 0 + quote: str | None = None + escaped = False + while cursor < len(text): + char = text[cursor] + next_char = text[cursor + 1] if cursor + 1 < len(text) else "" + if quote is not None: + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == quote: + quote = None + cursor += 1 + continue + regex_end = javascript_regex_literal_end(text, cursor) + if regex_end is not None: + cursor = regex_end + continue + if char == "/" and next_char == "/": + comment_end = text.find("\n", cursor + 2) + comment_end = len(text) if comment_end < 0 else comment_end + for index in range(cursor, comment_end): + masked[index] = " " + cursor = comment_end + continue + if char == "/" and next_char == "*": + comment_end = text.find("*/", cursor + 2) + comment_end = len(text) if comment_end < 0 else comment_end + 2 + for index in range(cursor, comment_end): + if masked[index] not in "\r\n": + masked[index] = " " + cursor = comment_end + continue + if char in {'"', "'", "`"}: + quote = char + cursor += 1 + return "".join(masked) + + +def safe_javascript_config_path_literal( + text: str, + literal: re.Match[str], + *, + call_target: str, + context_start: int, + context_end: int, + argument_index: int, + javascript_dialect: str | None = None, +) -> bool: + if javascript_dialect is None: + return False + value = literal.group("value") + if CANONICAL_CONFIG_PATH_REFERENCE_PATTERN.fullmatch(value) is None: + return False + final_segment = value.rsplit(".", 1)[-1] + if CONFIG_PATH_CREDENTIAL_FIELD_PATTERN.search(final_segment) is None: + return False + normalized_target = call_target.replace("?.", ".").rsplit(".", 1)[-1] + secret_file_reader = normalized_target in { + "readCredentialFile", + "readCredentialFileSync", + "readSecretFile", + "readSecretFileSync", + "tryReadCredentialFile", + "tryReadCredentialFileSync", + "tryReadSecretFile", + "tryReadSecretFileSync", + } + quote_start = literal.start("value") - 1 + quote_end = literal.end("value") + 1 + argument_prefix = mask_javascript_comments(text[context_start:quote_start]) + property_match = re.search( + r"(?:^|[^A-Za-z0-9_$])(?PconfigPath|path)\s*:\s*$", + argument_prefix, + ) + if property_match is not None: + return ( + property_match.group("key") == "configPath" and secret_file_reader + ) or ( + property_match.group("key") == "path" + and normalized_target == "normalizeResolvedSecretInputString" + ) + return ( + secret_file_reader + and argument_index == 1 + and not text[context_start:quote_start].strip() + and not text[quote_end:context_end].strip() + ) + + +def mask_safe_javascript_config_path_literals( + argument: str, + call_target: str, + *, + argument_index: int, + javascript_dialect: str | None = None, +) -> str: + spans = [ + literal.span("value") + for pattern in SECRET_FALLBACK_LITERAL_PATTERNS + for literal in pattern.finditer(argument) + if safe_javascript_config_path_literal( + argument, + literal, + call_target=call_target, + context_start=0, + context_end=len(argument), + argument_index=argument_index, + javascript_dialect=javascript_dialect, + ) + ] + return redact_review_spans(argument, spans) + + +def prompt_service_segment_is_secret_like(segment: str) -> bool: + suffix = re.search(r"\d{4,}$", segment) + if suffix is None: + return False + prefix = segment[: suffix.start()] + components = re.findall( + r"[A-Z]+(?=[A-Z][a-z]|$)|[A-Z]?[a-z]+", + prefix, + ) + theme_phrase = bool(components) and all( + component.casefold() in PROMPT_SECRET_THEME_WORDS + for component in components + ) + sequential_letters = len(prefix) >= 8 and all( + ord(right.casefold()) == ord(left.casefold()) + 1 + for left, right in zip(prefix, prefix[1:]) + ) + return theme_phrase or sequential_letters + + +def generic_credential_prompt_is_safe(value: str) -> bool: + match = GENERIC_CREDENTIAL_PROMPT_PATTERN.fullmatch(value) + if match is None: + return False + service = match.group("service") + if service is None: + return True + service = service.strip() + segments = re.split(r"[ _-]+", service) + secret_like_version = any( + prompt_service_segment_is_secret_like(segment) + for segment in segments + ) + natural_service = bool(service) and len(segments) <= 5 and all( + re.fullmatch(r"[A-Za-z][A-Za-z0-9]{0,23}", segment) is not None + and sum( + left.islower() != right.islower() + for left, right in zip(segment, segment[1:]) + if left.isalpha() and right.isalpha() + ) + <= 4 + for segment in segments + ) + return natural_service and not secret_like_version and not any( + pattern.search(service) for pattern in SECRET_VALUE_PATTERNS + ) + + +def public_call_argument_risk( + call_target: str, + argument: str, + argument_index: int, +) -> bool | None: + normalized_target = call_target.replace("?.", ".") + target_parts = normalized_target.split(".") + credential_scope_call = ( + len(target_parts) >= 2 + and target_parts[-2].lstrip("_") in {"credential", "credentials"} + and target_parts[-1] == "get_token" + ) + if ( + not credential_scope_call + and normalized_target not in PUBLIC_PROMPT_TARGETS + ): + return None + if normalized_target == "prompt": + match = re.fullmatch(r"\s*([\"'])([^\"'\r\n]+)\1\s*", argument) + if ( + argument_index == 0 + and match is not None + and generic_credential_prompt_is_safe(match.group(2)) + ): + return False + return secret_literal_risk(argument, minimum_length=8) + if not credential_scope_call and argument_index != 0: + return None + literal_argument = argument + if normalized_target == "getpass.getpass": + literal_argument = re.sub( + r"^\s*prompt\s*=\s*", + "", + literal_argument, + count=1, + ) + match = re.fullmatch(r"\s*([\"'])([^\"'\r\n]+)\1\s*", literal_argument) + if match is None: + return None + value = match.group(2) + if credential_scope_call: + decoded_value = value + for _ in range(8): + next_value = urllib.parse.unquote(decoded_value) + if next_value == decoded_value: + break + decoded_value = next_value + else: + return True + if any(ord(char) < 32 or ord(char) == 127 for char in decoded_value): + return True + if secret_text_risk(decoded_value): + return True + if re.fullmatch( + r"[0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-" + r"[0-9A-Fa-f]{4}-[0-9A-Fa-f]{12}/\.default", + decoded_value, + ): + return False + if "://" not in decoded_value: + return None + try: + parsed = urllib.parse.urlsplit(decoded_value) + hostname = parsed.hostname + port = parsed.port + except ValueError: + return None + valid_authority = ( + parsed.username is None + and parsed.password is None + and hostname is not None + and re.fullmatch( + r"[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?" + r"(?:\.[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?)*", + hostname, + ) + is not None + and (port is None or 1 <= port <= 65535) + ) + valid_scope_uri = ( + parsed.scheme in {"api", "https"} + and valid_authority + and parsed.path == "/.default" + and not parsed.query + and not parsed.fragment + ) + return False if valid_scope_uri else None + return False if generic_credential_prompt_is_safe(value) else None + + +def call_arguments_risk( + arguments: str, + call_target: str, + *, + javascript_dialect: str | None = None, +) -> bool: + for index, argument in enumerate(split_top_level_call_arguments(arguments)): + public_risk = public_call_argument_risk(call_target, argument, index) + if public_risk is not None: + if public_risk: + return True + continue + scanned_argument = mask_safe_javascript_config_path_literals( + argument, + call_target, + argument_index=index, + javascript_dialect=javascript_dialect, + ) + if not safe_credential_lookup_argument( + call_target, scanned_argument, index + ) and fallback_secret_risk( + scanned_argument, + minimum_length=12, + javascript_dialect=javascript_dialect, + ): + return True + return False + + +def truncated_call_arguments_risk( + arguments: str, + call_target: str, + *, + javascript_dialect: str | None = None, +) -> bool: + if call_arguments_risk( + arguments, + call_target, + javascript_dialect=javascript_dialect, + ): + return True + for argument in split_top_level_call_arguments(arguments): + match = SECRET_FALLBACK_UNQUOTED_PATTERN.fullmatch(argument.strip()) + if match is None: + continue + value = match.group("value") + if value.casefold() in SECRET_PLACEHOLDER_VALUES or any( + pattern.fullmatch(value) + for pattern in UNQUOTED_SECRET_REFERENCE_PATTERNS + ): + continue + if len(value) >= 7 and ( + any(character.isdigit() for character in value) + or re.fullmatch(r"[A-Za-z_$][A-Za-z0-9_$]*", value) is None + ): + return True + return False + + +def balanced_delimiter_end( + text: str, + start: int, + opener: str, + closer: str, +) -> int | None: + depth = 0 + quote: str | None = None + escaped = False + line_comment = False + block_comment = False + index = start + while index < len(text): + char = text[index] + next_char = text[index + 1] if index + 1 < len(text) else "" + if line_comment: + if char == "\n": + line_comment = False + index += 1 + continue + if block_comment: + if char == "*" and next_char == "/": + block_comment = False + index += 2 + else: + index += 1 + continue + if quote is not None: + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == quote: + quote = None + index += 1 + continue + regex_end = javascript_regex_literal_end(text, index) + if regex_end is not None: + index = regex_end + elif char == "/" and next_char == "/": + line_comment = True + index += 2 + elif char == "/" and next_char == "*": + block_comment = True + index += 2 + elif char == "#" and not javascript_private_member_marker(text, index): + line_comment = True + index += 1 + elif char in {'"', "'", "`"}: + quote = char + index += 1 + elif char == opener: + depth += 1 + index += 1 + elif char == closer: + depth -= 1 + if depth == 0: + return index + 1 + index += 1 + elif depth == 0: + return None + else: + index += 1 + return None + + +def safe_secret_call_suffix( + text: str, + end: int, + call_target: str, + *, + javascript_dialect: str | None = None, +) -> bool: + if end >= len(text) or text[end] != "(": + return False + + def safe_call_end(start: int, target: str) -> int | None: + cursor = balanced_delimiter_end(text, start, "(", ")") + if cursor is None: + if javascript_dialect is None: + return None + boundary = re.search(r"(?m)^;\r?$", text[start + 1 :]) + visible_end = ( + start + 1 + boundary.start() + if boundary is not None + else len(text) + ) + visible_arguments = text[start + 1 : visible_end] + return ( + None + if any( + pattern.search(visible_arguments) + for pattern in SECRET_FALLBACK_LITERAL_PATTERNS + ) + or truncated_call_arguments_risk( + visible_arguments, + target, + javascript_dialect=javascript_dialect, + ) + else visible_end + ) + arguments = text[start + 1 : cursor - 1] + return ( + None + if call_arguments_risk( + arguments, + target, + javascript_dialect=javascript_dialect, + ) + else cursor + ) + + cursor = safe_call_end(end, call_target) + if cursor is None: + return False + regex_recovery = premature_regex_call_tail(text, end, cursor) + if regex_recovery is not None: + regex_tail, cursor = regex_recovery + if secret_literal_risk(regex_tail): + return False + chained_target = ( + "response.json()" + if call_target.replace("?.", ".") == "response.json" + else "" + ) + while True: + match = re.match(r"\s*(?:\?\.|\.)[A-Za-z_][A-Za-z0-9_]*", text[cursor:]) + if match is not None: + member = re.search(r"[A-Za-z_][A-Za-z0-9_]*$", match.group(0)) + assert member is not None + member_name = member.group(0) + if chained_target == "response.json()" and member_name == "get": + chained_target = "response.json().get" + elif chained_target == "" and member_name == "headers": + chained_target = ".headers" + elif ( + chained_target == ".headers" + and member_name == "get" + ): + chained_target = ".headers.get" + else: + chained_target = "" + cursor += match.end() + continue + whitespace = re.match(r"\s*", text[cursor:]) + assert whitespace is not None + call_start = cursor + whitespace.end() + if call_start < len(text) and text[call_start] == "(": + cursor = safe_call_end(call_start, chained_target) + if cursor is None: + return False + chained_target = "" + continue + if call_start < len(text) and text[call_start] == "[": + cursor = balanced_delimiter_end(text, call_start, "[", "]") + if cursor is None: + return False + chained_target = "" + continue + return safe_secret_assignment_suffix(text, cursor) + + +def safe_backtick_literal_segment(value: str) -> bool: + return bool( + BACKTICK_TEMPLATE_SAFE_LITERAL_PATTERN.fullmatch(value) + or value.casefold() in BACKTICK_TEMPLATE_FIXTURE_PREFIX_VALUES + ) + + +def safe_backtick_interpolation_expression(expression: str) -> bool: + quoted_reference = "${" + expression + "}" + if any( + pattern.fullmatch(quoted_reference) + for pattern in QUOTED_SECRET_REFERENCE_PATTERNS + ): + return True + return re.fullmatch(r"(?:crypto\.)?randomUUID\(\)", expression) is not None + + +def safe_backtick_secret_template(value: str) -> bool: + cursor = 0 + found_interpolation = False + for match in BACKTICK_TEMPLATE_INTERPOLATION_PATTERN.finditer(value): + literal = value[cursor : match.start()] + if not safe_backtick_literal_segment(literal): + return False + expression = match.group(1).strip() + if not safe_backtick_interpolation_expression(expression): + return False + found_interpolation = True + cursor = match.end() + literal = value[cursor:] + return found_interpolation and safe_backtick_literal_segment(literal) + + +def javascript_template_literal_end( + text: str, + start: int, + nesting: int = 0, +) -> int | None: + if nesting > 64: + return None + cursor = start + 1 + expression_depth = 0 + while cursor < len(text): + char = text[cursor] + next_char = text[cursor + 1] if cursor + 1 < len(text) else "" + if expression_depth: + if char == "/" and next_char == "/": + line_end = text.find("\n", cursor + 2) + cursor = len(text) if line_end < 0 else line_end + continue + if char == "/" and next_char == "*": + comment_end = text.find("*/", cursor + 2) + cursor = len(text) if comment_end < 0 else comment_end + 2 + continue + regex_end = javascript_regex_literal_end(text, cursor) + if regex_end is not None: + cursor = regex_end + continue + if char in {'"', "'"}: + string_end = csharp_quoted_literal_end( + text, + cursor, + verbatim=False, + interpolated=False, + ) + if string_end is None: + return None + cursor = string_end + continue + if char == "`": + nested_end = javascript_template_literal_end( + text, + cursor, + nesting + 1, + ) + if nested_end is None: + return None + cursor = nested_end + continue + if char == "{": + expression_depth += 1 + elif char == "}": + expression_depth -= 1 + cursor += 1 + continue + if char == "\\": + cursor += 2 + continue + if char == "`": + return cursor + 1 + if char == "$" and next_char == "{": + expression_depth = 1 + cursor += 2 + continue + cursor += 1 + return None + + +@functools.lru_cache(maxsize=8) +def mask_reference_declaration_evidence(text: str) -> str: + masked = list(text) + + def mask_span(start: int, end: int) -> None: + for index in range(start, end): + if masked[index] not in "\r\n": + masked[index] = " " + + cursor = 0 + while cursor < len(text): + char = text[cursor] + next_char = text[cursor + 1] if cursor + 1 < len(text) else "" + if char == "/" and next_char == "/": + line_end = text.find("\n", cursor + 2) + cursor = len(text) if line_end < 0 else line_end + continue + if char == "/" and next_char == "*": + comment_end = text.find("*/", cursor + 2) + cursor = len(text) if comment_end < 0 else comment_end + 2 + continue + if char == "#" and not javascript_private_member_marker(text, cursor): + line_end = text.find("\n", cursor + 1) + line_end = len(text) if line_end < 0 else line_end + mask_span(cursor, line_end) + cursor = line_end + continue + regex_end = javascript_regex_literal_end(text, cursor) + if regex_end is not None: + mask_span(cursor, regex_end) + cursor = regex_end + continue + if char in {'"', "'"}: + raw_start = ( + raw_double_quote_start(text, cursor) + if char == '"' + else None + ) + if raw_start is not None: + delimiter, content_start = raw_start + raw_end = raw_double_quote_end( + text, + content_start, + len(delimiter), + ) + cursor = len(text) if raw_end is None else raw_end + continue + marker = text[max(0, cursor - 2) : cursor] + string_end = csharp_quoted_literal_end( + text, + cursor, + verbatim=char == '"' and "@" in marker, + interpolated=char == '"' and "$" in marker, + ) + cursor = len(text) if string_end is None else string_end + continue + if char != "`": + cursor += 1 + continue + template_end = javascript_template_literal_end(text, cursor) + template_end = len(text) if template_end is None else template_end + mask_span(cursor, min(template_end, len(text))) + cursor = template_end + return mask_csharp_evidence_prefix("".join(masked)) + + +@functools.lru_cache(maxsize=8) +def javascript_reference_spans(text: str) -> frozenset[tuple[int, int]]: + return frozenset( + (match.start(), match.end()) + for pattern in ( + SOURCE_CODE_REFERENCE_PATTERN, + SOURCE_CODE_REFERENCE_ROOT_PATTERN, + SOURCE_CODE_LIFECYCLE_REFERENCE_PATTERN, + ) + for match in pattern.finditer(text) + ) + + +def javascript_source_whitespace(char: str) -> bool: + return ( + char in "\t\v\f \r\n\u2028\u2029\ufeff" + or unicodedata.category(char) == "Zs" + ) + + +def safe_javascript_reference_suffix( + text: str, + end: int, + *, + typescript: bool, +) -> bool: + line_terminators = "\r\n\u2028\u2029" + cursor = end + saw_newline = False + while cursor < len(text): + while cursor < len(text) and javascript_source_whitespace(text[cursor]): + saw_newline = saw_newline or text[cursor] in line_terminators + cursor += 1 + if text.startswith("//", cursor): + newline = re.search(r"[\r\n\u2028\u2029]", text[cursor + 2 :]) + if newline is None: + return True + cursor += 2 + newline.start() + saw_newline = True + continue + if text.startswith("/*", cursor): + comment_end = text.find("*/", cursor + 2) + if comment_end < 0: + return True + comment = text[cursor : comment_end + 2] + saw_newline = saw_newline or any( + terminator in comment for terminator in line_terminators + ) + cursor = comment_end + 2 + continue + break + if cursor >= len(text): + return True + if text[cursor] in ";}])" or text[cursor] == ",": + return True + if saw_newline and ( + text.startswith(("++", "--"), cursor) + or text[cursor] == "~" + or (text[cursor] == "!" and not text.startswith("!=", cursor)) + ): + return True + continuation_keyword = re.match( + ( + r"(?:as|in|instanceof|satisfies)(?![\w$])" + if typescript + else r"(?:in|instanceof)(?![\w$])" + ), + text[cursor:], + ) + continuation = ( + text[cursor] in "=<>!~:+-*/%&|^?.,([`" + or continuation_keyword is not None + ) + if continuation: + return not fallback_secret_risk( + text[cursor:], + javascript_dialect="typescript" if typescript else "javascript", + ) + return saw_newline + + +def bare_code_reference( + text: str, + start: int, + end: int, + separator: str, + value: str, +) -> bool: + camel_reference = re.fullmatch( + r"[a-z][A-Za-z0-9]*[A-Z][A-Za-z0-9]*", + value, + ) + snake_reference = re.fullmatch( + r"[a-z][a-z0-9]*(?:_[a-z0-9]+)+", + value, + ) + line_start = max(text.rfind("\n", 0, start), text.rfind("\r", 0, start)) + masked_text = mask_reference_declaration_evidence(text) + declaration = masked_text[line_start + 1 : start] + pascal_type_reference = re.fullmatch( + r"[A-Z][A-Za-z]*(?:Credential|Credentials|Options|Config|Type|Enum)", + value, + ) + if separator == ":" and pascal_type_reference is not None: + type_prefix = masked_text[ + max(0, start - 2048) : start + ] + if re.search( + r"\b(?:class|interface|record|struct|type)\b" + r"[^{};\r\n]*\{[^}]*$", + type_prefix, + re.DOTALL, + ): + return True + # TS/JS named-function parameter annotations use PascalCase type names. + function_prefix = re.search( + r"\bfunction\b[^()\r\n]*\((?P[^()]*)$", + declaration, + ) + annotation_suffix = re.match( + r"[ \t\r\n]*(?P[,)=])", + text[end:], + ) + if ( + function_prefix is not None + and re.fullmatch( + r"\s*(?:\.\.\.\s*)?", + split_top_level_call_arguments( + function_prefix.group("parameters") + )[-1], + ) + and annotation_suffix is not None + ): + if annotation_suffix.group("terminator") != "=": + return True + return not fallback_secret_risk( + text[end + annotation_suffix.end() :], + minimum_length=8, + ) + if camel_reference is None and snake_reference is None: + return False + return bool( + re.search(r"\b(?:const|let|var)\s+$", declaration) + or re.search( + r"\b(?:const|let|var)\s+[A-Za-z_$][A-Za-z0-9_$]*" + r"\s*=\s*\{[^{}]*$", + declaration, + re.DOTALL, + ) + ) + + +def basic_authorization_risk(text: str) -> bool: + for match in BASIC_AUTHORIZATION_PATTERN.finditer(text): + encoded = match.group("credential") + padded = encoded + "=" * (-len(encoded) % 4) + try: + decoded = base64.b64decode(padded, validate=True) + except (binascii.Error, ValueError): + continue + if b":" in decoded: + return True + return False + + +def safe_self_reference_assignment( + text: str, + match: re.Match[str], + *, + javascript_dialect: str | None = None, +) -> bool: + if javascript_dialect is None: + return False + value = ( + match.group("reference_value") + or match.group("call_value") + or match.group("bare_value") + ) + if value is None: + return False + key = re.split(r"\s*[:=]\s*", match.group(0), maxsplit=1)[0].strip("\"'") + if value != key: + return False + separator = re.search(r"[:=]", match.group(0)) + if separator is None or separator.group(0) != "=": + return False + line_end_candidates = [ + position + for position in ( + text.find("\n", match.end()), + text.find("\r", match.end()), + ) + if position >= 0 + ] + line_end = min(line_end_candidates, default=len(text)) + if not safe_secret_assignment_suffix( + text[:line_end], + match.end(), + javascript_dialect=javascript_dialect, + ): + return False + return safe_javascript_reference_suffix( + text, + line_end, + typescript=javascript_dialect == "typescript", + ) + + +def unsafe_multiline_self_reference_assignment( + text: str, + match: re.Match[str], + *, + javascript_dialect: str | None = None, +) -> bool: + if javascript_dialect is None: + return False + value = ( + match.group("reference_value") + or match.group("call_value") + or match.group("bare_value") + ) + if value is None: + return False + key = re.split(r"\s*[:=]\s*", match.group(0), maxsplit=1)[0].strip("\"'") + separator = re.search(r"[:=]", match.group(0)) + if value != key or separator is None or separator.group(0) != "=": + return False + line_end_candidates = [ + position + for position in ( + text.find("\n", match.end()), + text.find("\r", match.end()), + ) + if position >= 0 + ] + line_end = min(line_end_candidates, default=len(text)) + return line_end < len(text) and not safe_javascript_reference_suffix( + text, + line_end, + typescript=javascript_dialect == "typescript", + ) + + +def boolean_declaration_initializer_range( + text: str, + match: re.Match[str], + *, + javascript_dialect: str | None = None, +) -> tuple[int, int] | None: + initializer = re.match(r"\s*=\s*", text[match.end() :]) + if initializer is None: + return None + start = match.end() + initializer.end() + expression = fallback_expression( + text[start:], + typescript=javascript_dialect == "typescript", + ) + return start, start + len(expression) + + +def boolean_declaration_initializer_risk( + text: str, + match: re.Match[str], + *, + javascript_dialect: str | None = None, +) -> bool: + initializer_range = boolean_declaration_initializer_range( + text, + match, + javascript_dialect=javascript_dialect, + ) + if initializer_range is None: + return False + start, end = initializer_range + return secret_literal_risk( + text[start:end], + minimum_length=8, + javascript_dialect=javascript_dialect, + ) + + +def boolean_declaration_initializer_spans( + text: str, + match: re.Match[str], + *, + javascript_dialect: str | None = None, +) -> list[tuple[int, int]]: + initializer_range = boolean_declaration_initializer_range( + text, + match, + javascript_dialect=javascript_dialect, + ) + if initializer_range is None: + return [] + start, end = initializer_range + if not secret_literal_risk( + text[start:end], + minimum_length=8, + javascript_dialect=javascript_dialect, + ): + return [] + return top_level_fallback_value_spans(text, start, end) + + +def secret_text_risk( + text: str, + *, + javascript_dialect: str | None = None, +) -> bool: + if DIFF_HUNK_CONTENT_BOUNDARY in text: + return any( + secret_text_risk(segment, javascript_dialect=javascript_dialect) + for segment in text.split(DIFF_HUNK_CONTENT_BOUNDARY) + ) + uri_authorities = uri_authority_ranges(text) + if credentialed_uri_risk(text, uri_authorities) or basic_authorization_risk(text) or any( + pattern.search(text) for pattern in SECRET_VALUE_PATTERNS + ): + return True + boolean_declarations = tuple(SECRET_BOOLEAN_DECLARATION_PATTERN.finditer(text)) + boolean_declaration_positions = { + match.start("boolean_key") for match in boolean_declarations + } + if any( + boolean_declaration_initializer_risk( + text, + match, + javascript_dialect=javascript_dialect, + ) + for match in boolean_declarations + ): + return True + safe_uri_credentials = interpolated_empty_password_uri_ranges( + text, + uri_authorities, + ) + assignment_scan_text = mask_ranges(text, safe_uri_credentials) + assignment_prefixes = secret_assignment_matches( + SECRET_ASSIGNMENT_PREFIX_PATTERN, + text, + assignment_scan_text, + safe_uri_credentials, + ) + chained_assignment_positions = top_level_line_assignment_positions( + text, + {prefix.start() for prefix in assignment_prefixes}, + ) + source_reference_spans = ( + javascript_reference_spans(text) + if javascript_dialect is not None + else frozenset() + ) + for prefix in assignment_prefixes: + if prefix.start() in boolean_declaration_positions: + continue + assignment = SECRET_ASSIGNMENT_PATTERN.match(text, prefix.start()) + if assignment is not None and safe_self_reference_assignment( + text, + assignment, + javascript_dialect=javascript_dialect, + ): + continue + if assignment is not None and unsafe_multiline_self_reference_assignment( + text, + assignment, + javascript_dialect=javascript_dialect, + ): + return True + fallback = top_level_fallback_suffix( + text[prefix.end() :], + allow_chained_assignment=( + re.search(r"=(?!=|>)\s*$", prefix.group(0)) is not None + and prefix.start() in chained_assignment_positions + ), + ) + if fallback is not None and fallback_secret_risk( + fallback, + javascript_dialect=javascript_dialect, + ): + return True + for match in secret_assignment_matches( + SECRET_ASSIGNMENT_PATTERN, + text, + assignment_scan_text, + safe_uri_credentials, + ): + if match.start() in boolean_declaration_positions: + continue + quoted = any( + match.group(name) is not None + for name in ("double_value", "single_value", "backtick_value") + ) + value = ( + match.group("double_value") + or match.group("single_value") + or match.group("backtick_value") + or match.group("reference_value") + or match.group("call_value") + or match.group("bare_value") + ) + if value is None: + continue + key = re.split(r"\s*[:=]\s*", match.group(0), maxsplit=1)[0] + separator_match = re.search(r"[:=]", match.group(0)) + assert separator_match is not None + separator = separator_match.group(0) + if safe_self_reference_assignment( + text, + match, + javascript_dialect=javascript_dialect, + ): + continue + if ( + key.strip("\"'").lower() == "credentials" + and value.lower() in FETCH_CREDENTIAL_MODE_VALUES + and safe_secret_assignment_suffix( + text, + match.end(), + javascript_dialect=javascript_dialect, + ) + ): + continue + if ( + match.group("backtick_value") is not None + and ( + any( + pattern.fullmatch(value) + for pattern in BACKTICK_SECRET_REFERENCE_PATTERNS + ) + or safe_backtick_secret_template(value) + ) + and safe_secret_assignment_suffix( + text, + match.end(), + javascript_dialect=javascript_dialect, + ) + ): + continue + if synthetic_secret_fixture(value, key): + if safe_secret_assignment_suffix( + text, + match.end(), + javascript_dialect=javascript_dialect, + ): + continue + return True + if ( + match.group("bare_value") is not None + and len(value) < 12 + and re.fullmatch(r"[A-Za-z_$][A-Za-z0-9_$]*", value) + ): + if safe_secret_assignment_suffix( + text, + match.end(), + javascript_dialect=javascript_dialect, + ): + continue + return True + if ( + match.group("bare_value") is not None + and bare_code_reference( + text, + match.start(), + match.end(), + separator, + value, + ) + and safe_secret_assignment_suffix( + text, + match.end(), + javascript_dialect=javascript_dialect, + ) + ): + continue + if not quoted: + source_start = match.end() - len(value) + source_end = match.end() - (1 if value.endswith("!") else 0) + if ( + ( + (source_start, source_end) in source_reference_spans + or ( + javascript_dialect is not None + and SOURCE_CODE_REFERENCE_ROOT_PATTERN.fullmatch(value) + ) + ) + and safe_javascript_reference_suffix( + text, + match.end(), + typescript=javascript_dialect == "typescript", + ) + ): + continue + reference_patterns = ( + QUOTED_SECRET_REFERENCE_PATTERNS + if quoted + else UNQUOTED_SECRET_REFERENCE_PATTERNS + ) + call_target = re.fullmatch( + r"[A-Za-z_][A-Za-z0-9_]*(?:(?:\.|\?\.)[A-Za-z_][A-Za-z0-9_]*)*", + value, + ) + suffix = text[match.end() :] + # Crossing a newline can misread the next shell subshell as this value's call. + whitespace = re.match(r"[ \t]*", suffix) + assert whitespace is not None + call_start = match.end() + whitespace.end() + if ( + call_start > match.end() + and text[call_start : call_start + 1] == "(" + ): + if ( + call_target + and call_target.group(0).replace("?.", ".") + in PUBLIC_PROMPT_TARGETS + and safe_secret_call_suffix( + text, + call_start, + call_target.group(0), + javascript_dialect=javascript_dialect, + ) + ): + continue + return True + if ( + not quoted + and call_target + and text[call_start : call_start + 1] == "(" + ): + if safe_secret_call_suffix( + text, + call_start, + value, + javascript_dialect=javascript_dialect, + ): + continue + return True + if any(pattern.fullmatch(value) for pattern in reference_patterns): + if safe_secret_assignment_suffix( + text, + match.end(), + javascript_dialect=javascript_dialect, + ): + continue + return True + return True + return False + + +def require_no_secret_values( + label: str, + text: str, + *, + javascript_dialect: str | None = None, +) -> None: + if secret_text_risk(text, javascript_dialect=javascript_dialect): + raise SystemExit( + "refusing to include secret-like content in review bundle; " + f"clean or redact {label} before running autoreview" + ) + + +def assignment_fallback_literal_spans( + text: str, + match: re.Match[str], + *, + javascript_dialect: str | None = None, +) -> list[tuple[int, int]]: + line_end_candidates = [ + position + for position in ( + text.find("\n", match.end()), + text.find("\r", match.end()), + ) + if position >= 0 + ] + line_end = min(line_end_candidates, default=len(text)) + expression_end = line_end + if ( + javascript_dialect is not None + and line_end < len(text) + and not safe_javascript_reference_suffix( + text, + line_end, + typescript=javascript_dialect == "typescript", + ) + ): + continued = fallback_expression( + text[line_end:], + typescript=javascript_dialect == "typescript", + ) + expression_end = line_end + len(continued) + fallback_start = match.end() + if match.group("call_value") is not None: + whitespace = re.match(r"[ \t]*", text[fallback_start:expression_end]) + assert whitespace is not None + call_start = fallback_start + whitespace.end() + if call_start < line_end and text[call_start] == "(": + depth = 0 + quote: str | None = None + escaped = False + cursor = call_start + while cursor < expression_end: + char = text[cursor] + if quote is not None: + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == quote: + quote = None + elif char in {'"', "'", "`"}: + quote = char + elif char == "(": + depth += 1 + elif char == ")": + depth -= 1 + if depth == 0: + fallback_start = cursor + 1 + break + cursor += 1 + operator_span = top_level_fallback_operator_span( + text, + fallback_start, + expression_end, + ) + if operator_span is None: + return [] + expression_start = operator_span[1] + depth = 0 + quote: str | None = None + escaped = False + cursor = expression_start + while cursor < line_end: + char = text[cursor] + if quote is not None: + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == quote: + quote = None + elif char in {'"', "'", "`"}: + quote = char + elif char in "([{": + depth += 1 + elif char in ")]}": + if depth == 0: + expression_end = cursor + break + depth -= 1 + elif depth == 0 and char in ";,": + expression_end = cursor + break + cursor += 1 + return top_level_fallback_value_spans( + text, + expression_start, + expression_end, + include_nested=True, + ) + + +def top_level_fallback_operator_span( + text: str, + start: int, + end: int, +) -> tuple[int, int] | None: + stack: list[str] = [] + pairs = {"(": ")", "[": "]", "{": "}"} + quote: str | None = None + escaped = False + line_comment = False + block_comment = False + cursor = start + while cursor < end: + char = text[cursor] + next_char = text[cursor + 1] if cursor + 1 < end else "" + if line_comment: + if char in "\r\n": + line_comment = False + cursor += 1 + continue + if block_comment: + if char == "*" and next_char == "/": + block_comment = False + cursor += 2 + else: + cursor += 1 + continue + if quote is not None: + if escaped: + escaped = False + elif char == "\\": + escaped = True + elif char == quote: + quote = None + cursor += 1 + continue + if char == "/" and next_char == "/": + line_comment = True + cursor += 2 + continue + if char == "/" and next_char == "*": + block_comment = True + cursor += 2 + continue + if char == "#": + line_comment = True + cursor += 1 + continue + if char in {'"', "'", "`"}: + quote = char + cursor += 1 + continue + if char in pairs: + stack.append(pairs[char]) + cursor += 1 + continue + if stack and char == stack[-1]: + stack.pop() + cursor += 1 + continue + if not stack and text.startswith(("||", "??"), cursor): + return cursor, cursor + 2 + if not stack and text.startswith("or", cursor): + before = text[cursor - 1] if cursor > start else "" + after = text[cursor + 2] if cursor + 2 < end else "" + if not (before.isalnum() or before == "_") and not ( + after.isalnum() or after == "_" + ): + return cursor, cursor + 2 + cursor += 1 + return None + + +def top_level_fallback_value_spans( + text: str, + start: int, + end: int, + *, + include_nested: bool = False, +) -> list[tuple[int, int]]: + spans: list[tuple[int, int]] = [] + stack: list[str] = [] + pairs = {"(": ")", "[": "]", "{": "}"} + cursor = start + while cursor < end: + char = text[cursor] + next_char = text[cursor + 1] if cursor + 1 < end else "" + if char == "/" and next_char == "/": + newline = re.search(r"[\r\n]", text[cursor + 2 : end]) + if newline is None: + break + cursor += 2 + newline.end() + continue + if char == "/" and next_char == "*": + comment_end = text.find("*/", cursor + 2, end) + if comment_end < 0: + break + cursor = comment_end + 2 + continue + if char == "#": + newline = re.search(r"[\r\n]", text[cursor + 1 : end]) + if newline is None: + break + cursor += 1 + newline.end() + continue + if char in {'"', "'", "`"}: + quote = char + value_start = cursor + 1 + cursor = value_start + escaped = False + while cursor < end: + current = text[cursor] + if escaped: + escaped = False + elif current == "\\": + escaped = True + elif current == quote: + value = text[value_start:cursor] + repeatable_nested_value = ( + include_nested + and value.casefold() not in SECRET_PLACEHOLDER_VALUES + and len(value) >= 7 + and any(character.isdigit() for character in value) + and not any( + pattern.fullmatch(value) + for pattern in QUOTED_SECRET_REFERENCE_PATTERNS + ) + ) + if cursor > value_start and ( + not stack or repeatable_nested_value + ): + spans.append((value_start, cursor)) + cursor += 1 + break + cursor += 1 + continue + if char in pairs: + stack.append(pairs[char]) + cursor += 1 + continue + if stack and char == stack[-1]: + stack.pop() + cursor += 1 + continue + if not stack: + bare = SECRET_FALLBACK_UNQUOTED_PATTERN.match(text, cursor, end) + if bare is not None: + value = bare.group("value") + if len(value) >= 7 and ( + any(candidate.isdigit() for candidate in value) + or re.fullmatch(r"[A-Za-z_$][A-Za-z0-9_$]*", value) is None + ): + spans.append(bare.span("value")) + cursor = bare.end() + continue + cursor += 1 + return sorted(set(spans)) + + +def review_call_literal_spans( + text: str, + match: re.Match[str], + *, + javascript_dialect: str | None = None, +) -> list[tuple[int, int]]: + whitespace = re.match(r"[ \t]*", text[match.end() :]) + assert whitespace is not None + call_start = match.end() + whitespace.end() + if call_start >= len(text) or text[call_start] != "(": + return [] + call_end = balanced_delimiter_end(text, call_start, "(", ")") + if call_end is None or not secret_text_risk( + text[match.start() : call_end], + javascript_dialect=javascript_dialect, + ): + return [] + candidates = sorted( + ( + literal + for pattern in SECRET_FALLBACK_LITERAL_PATTERNS + for literal in pattern.finditer(text, call_start + 1, call_end - 1) + ), + key=lambda literal: literal.start("value"), + ) + argument_ranges: list[tuple[int, int]] = [] + argument_start = call_start + 1 + for argument in split_top_level_call_arguments( + text[call_start + 1 : call_end - 1] + ): + argument_end = argument_start + len(argument) + argument_ranges.append((argument_start, argument_end)) + argument_start = argument_end + 1 + candidates_with_argument_index: list[tuple[re.Match[str], int]] = [] + argument_index = 0 + for literal in candidates: + while ( + argument_index < len(argument_ranges) + and argument_ranges[argument_index][1] <= literal.start("value") + ): + argument_index += 1 + matched_index = ( + argument_index + if argument_index < len(argument_ranges) + and argument_ranges[argument_index][0] + <= literal.start("value") + < argument_ranges[argument_index][1] + else -1 + ) + candidates_with_argument_index.append((literal, matched_index)) + return [ + literal.span("value") + for literal, argument_index in candidates_with_argument_index + if len(literal.group("value")) >= 7 + and any(char.isalpha() for char in literal.group("value")) + and literal.group("value").casefold() not in SECRET_PLACEHOLDER_VALUES + and not any( + pattern.fullmatch(literal.group("value")) + for pattern in QUOTED_SECRET_REFERENCE_PATTERNS + ) + and not safe_javascript_config_path_literal( + text, + literal, + call_target=match.group("call_value") or "", + context_start=( + argument_ranges[argument_index][0] + if 0 <= argument_index < len(argument_ranges) + else call_start + 1 + ), + context_end=( + argument_ranges[argument_index][1] + if 0 <= argument_index < len(argument_ranges) + else call_end - 1 + ), + argument_index=argument_index, + javascript_dialect=javascript_dialect, + ) + ] + + +def review_repeatable_secret_spans( + text: str, + *, + javascript_dialect: str | None = None, +) -> list[tuple[int, int]]: + boolean_declarations = tuple(SECRET_BOOLEAN_DECLARATION_PATTERN.finditer(text)) + boolean_declaration_positions = { + match.start("boolean_key") for match in boolean_declarations + } + spans = [ + span + for match in boolean_declarations + for span in boolean_declaration_initializer_spans( + text, + match, + javascript_dialect=javascript_dialect, + ) + ] + for match in SECRET_ASSIGNMENT_PATTERN.finditer(text): + if match.start() in boolean_declaration_positions: + continue + selected_name = next( + ( + name + for name in ( + "double_value", + "single_value", + "backtick_value", + "reference_value", + "call_value", + "bare_value", + ) + if match.group(name) is not None + ), + None, + ) + if selected_name is None: + continue + if safe_self_reference_assignment( + text, + match, + javascript_dialect=javascript_dialect, + ): + continue + fallback_spans = ( + assignment_fallback_literal_spans( + text, + match, + javascript_dialect=javascript_dialect, + ) + if selected_name in {"reference_value", "call_value", "bare_value"} + else [] + ) + if fallback_spans: + spans.extend(fallback_spans) + continue + if selected_name == "call_value": + spans.extend( + review_call_literal_spans( + text, + match, + javascript_dialect=javascript_dialect, + ) + ) + continue + if secret_text_risk( + match.group(0), + javascript_dialect=javascript_dialect, + ): + spans.append(match.span(selected_name)) + for pattern in SECRET_VALUE_PATTERNS: + spans.extend( + match.span("credential") + if pattern is BEARER_CREDENTIAL_PATTERN + else match.span() + for match in pattern.finditer(text) + ) + spans.extend( + match.span("credential") + for match in BASIC_AUTHORIZATION_PATTERN.finditer(text) + ) + for authority_range in uri_authority_ranges(text): + credential = uri_authority_credential(text, authority_range) + if ( + credential is not None + and credential.value + and uri_credential_value_risk(text, credential) + ): + spans.append((credential.value_start, credential.value_end)) + return spans + + +def review_secret_value_spans( + text: str, + *, + javascript_dialect: str | None = None, +) -> list[tuple[int, int]]: + boolean_declarations = tuple(SECRET_BOOLEAN_DECLARATION_PATTERN.finditer(text)) + boolean_declaration_positions = { + match.start("boolean_key") for match in boolean_declarations + } + spans = [ + span + for match in boolean_declarations + for span in boolean_declaration_initializer_spans( + text, + match, + javascript_dialect=javascript_dialect, + ) + ] + for match in SECRET_ASSIGNMENT_PATTERN.finditer(text): + if match.start() in boolean_declaration_positions: + continue + for name in ( + "double_value", + "single_value", + "backtick_value", + "reference_value", + "call_value", + "bare_value", + ): + if match.group(name) is not None: + if safe_self_reference_assignment( + text, + match, + javascript_dialect=javascript_dialect, + ): + break + fallback_spans = ( + assignment_fallback_literal_spans( + text, + match, + javascript_dialect=javascript_dialect, + ) + if name in {"reference_value", "call_value", "bare_value"} + else [] + ) + if fallback_spans: + spans.extend(fallback_spans) + break + if name == "call_value": + spans.extend( + review_call_literal_spans( + text, + match, + javascript_dialect=javascript_dialect, + ) + ) + break + if secret_text_risk( + match.group(0), + javascript_dialect=javascript_dialect, + ): + spans.append(match.span(name)) + break + for pattern in SECRET_VALUE_PATTERNS: + spans.extend( + match.span("credential") + if pattern is BEARER_CREDENTIAL_PATTERN + else match.span() + for match in pattern.finditer(text) + ) + spans.extend( + match.span("credential") + for match in BASIC_AUTHORIZATION_PATTERN.finditer(text) + ) + for authority_range in uri_authority_ranges(text): + credential = uri_authority_credential(text, authority_range) + if ( + credential is not None + and credential.value + and uri_credential_value_risk(text, credential) + ): + spans.append((credential.value_start, credential.value_end)) + return spans + + +def redact_review_spans(text: str, spans: list[tuple[int, int]]) -> str: + merged: list[tuple[int, int]] = [] + for start, end in sorted(spans): + if start >= end: + continue + if merged and start <= merged[-1][1]: + merged[-1] = (merged[-1][0], max(end, merged[-1][1])) + else: + merged.append((start, end)) + if not merged: + return text + parts: list[str] = [] + cursor = 0 + for start, end in merged: + parts.extend((text[cursor:start], "redacted")) + cursor = end + parts.append(text[cursor:]) + return "".join(parts) + + +def known_secret_fragment_pattern(fragments: list[str]) -> re.Pattern[str] | None: + if not fragments: + return None + alternatives = "|".join(re.escape(fragment) for fragment in fragments) + return re.compile(rf"(?=(?P{alternatives}))") + + +def known_secret_fragment_spans( + text: str, + pattern: re.Pattern[str] | None, +) -> list[tuple[int, int]]: + if pattern is None: + return [] + return [match.span("fragment") for match in pattern.finditer(text)] + + +def javascript_regex_literal_ranges(text: str) -> list[tuple[int, int]]: + ranges: list[tuple[int, int]] = [] + cursor = 0 + while cursor < len(text): + end = javascript_regex_literal_end(text, cursor) + if end is None: + cursor += 1 + continue + ranges.append((cursor, end)) + cursor = end + return ranges + + +def repeated_secret_fragment_spans( + text: str, + pattern: re.Pattern[str] | None, + *, + javascript_dialect: str | None = None, +) -> list[tuple[int, int]]: + matches = known_secret_fragment_spans(text, pattern) + contexts = string_contexts_at(text, {start for start, _ in matches}) + regex_ranges = ( + javascript_regex_literal_ranges(text) + if javascript_dialect is not None + else [] + ) + regex_index = 0 + spans: list[tuple[int, int]] = [] + for start, end in matches: + while ( + regex_index < len(regex_ranges) + and regex_ranges[regex_index][1] <= start + ): + regex_index += 1 + if ( + regex_index < len(regex_ranges) + and regex_ranges[regex_index][0] <= start + and end <= regex_ranges[regex_index][1] + ): + spans.append((start, end)) + continue + token_start = start + while token_start > 0 and re.match(r"[A-Za-z0-9_$.-]", text[token_start - 1]): + token_start -= 1 + token_end = end + while token_end < len(text) and re.match(r"[A-Za-z0-9_$.-]", text[token_end]): + token_end += 1 + key_position = re.match( + r"\s*(?:=(?!=|>)|:(?![:=]))", + text[token_end:], + ) is not None + unquoted_token_substring = ( + contexts[start] is None + and (token_start < start or end < token_end) + ) + unquoted_identifier = ( + contexts[start] is None + and token_start == start + and token_end == end + and re.fullmatch(r"[A-Za-z_$][A-Za-z0-9_$.-]*", text[start:end]) + is not None + ) + if contexts[start] is None and ( + key_position or unquoted_token_substring or unquoted_identifier + ): + continue + spans.append((start, end)) + return spans + + +def private_key_body_spans(text: str) -> list[tuple[int, int]]: + matches = list(PRIVATE_KEY_BODY_PATTERN.finditer(text)) + contexts = string_contexts_at(text, {match.start() for match in matches}) + wrapper_parts: list[str] = [] + cursor = 0 + for match in matches: + wrapper_parts.append(text[cursor : match.start()]) + cursor = match.end() + wrapper_parts.append(text[cursor:]) + body_only_line = re.fullmatch( + r"(?:(?:#|//|/\*|\*)\s*)?[\s\"'`,;+\[\]]*", + "".join(wrapper_parts).strip(), + ) is not None + return [ + match.span() + for match in matches + if body_only_line or contexts[match.start()] is not None + ] + + +def private_key_token_spans(text: str) -> list[tuple[int, int]]: + escaped_spans = [ + match.span("body") + for match in ESCAPED_PRIVATE_KEY_BODY_PATTERN.finditer(text) + ] + ordinary_spans = [ + match.span() + for match in PRIVATE_KEY_BODY_PATTERN.finditer(text) + if not ( + match.start() > 0 + and text[match.start() - 1] == "\\" + and match.group(0)[:1] in {"n", "r"} + ) + and not any( + start < match.end() and match.start() < end + for start, end in escaped_spans + ) + ] + return sorted(escaped_spans + ordinary_spans) + + +def normalized_private_key_fragment(text: str, start: int, end: int) -> str: + fragment = text[start:end] + if start > 0 and text[start - 1] == "\\" and fragment[:1] in {"n", "r"}: + return fragment[1:] + return fragment + + +def likely_private_key_token(value: str) -> bool: + encoded = value.rstrip("=") + if not encoded: + return False + entropy = -sum( + (frequency := encoded.count(char) / len(encoded)) * math.log2(frequency) + for char in set(encoded) + ) + return ( + (len(encoded) >= 48 or encoded.startswith("MII")) + and any(char.islower() for char in encoded) + and any(char.isupper() for char in encoded) + and any(char.isdigit() or char in "+/=" for char in value) + and entropy >= 4.25 + ) + + +def wrapped_private_key_body_matches(text: str) -> list[re.Match[str]]: + matches = list(PRIVATE_KEY_BODY_PATTERN.finditer(text)) + if not matches: + return [] + wrapper_parts: list[str] = [] + cursor = 0 + for match in matches: + wrapper_parts.append(text[cursor : match.start()]) + cursor = match.end() + wrapper_parts.append(text[cursor:]) + if re.fullmatch( + r"(?:(?:#|//|/\*|\*)\s*)?[\s\\\"'`,;+\[\]]*" + r"(?:\*/[\s\\\"'`,;+\[\]]*)?", + "".join(wrapper_parts).strip(), + ) is None: + return [] + return matches + + +def orphan_private_key_body_spans(text: str) -> list[tuple[int, int]]: + matches = wrapped_private_key_body_matches(text) + if not matches: + return [] + values = [match.group(0) for match in matches] + raw_encoded = "".join(values) + encoded = raw_encoded.rstrip("=") + if not encoded: + return [] + entropy = -sum( + (frequency := encoded.count(char) / len(encoded)) * math.log2(frequency) + for char in set(encoded) + ) + mixed_short_chunks = ( + len(encoded) >= 20 + and len(matches) >= 2 + and all( + any(char.isdigit() for char in value) + and any(char.isupper() or char in "+/" for char in value) + for value in values + ) + and any(char.islower() for char in encoded) + and entropy >= 4.25 + ) + long_base64_line = len(matches) == 1 and likely_private_key_token(raw_encoded) + if not long_base64_line and not mixed_short_chunks: + return [] + return [match.span() for match in matches] + + +def bare_source_identifier_span(text: str, start: int, end: int) -> bool: + if re.fullmatch(r"[A-Za-z_$][A-Za-z0-9_$]*", text[start:end]) is None: + return False + before = text[:start].rstrip()[-1:] + after = text[end:].lstrip()[:1] + return before in {"=", "+", "(", "[", "{", ",", ":"} and after in { + "+", + ")", + "]", + "}", + ",", + ";", + ":", + } + + +def explicit_private_key_body_spans( + text: str, + *, + in_pem: bool, +) -> list[tuple[int, int]]: + if not in_pem: + return [] + candidates = private_key_token_spans(text) + if not candidates: + return [] + wrapper_parts: list[str] = [] + cursor = 0 + for start, end in candidates: + wrapper_parts.append(text[cursor:start]) + cursor = end + wrapper_parts.append(text[cursor:]) + wrapper_text = "".join(wrapper_parts).strip() + escaped_body_only = ("\\n" in wrapper_text or "\\r" in wrapper_text) and re.fullmatch( + r"(?:(?:\\[nr])|\s)*", + wrapper_text, + ) is not None + if escaped_body_only: + return candidates + contexts = string_contexts_at(text, {start for start, _ in candidates}) + wrapped = {match.span() for match in wrapped_private_key_body_matches(text)} + return [ + (start, end) + for start, end in candidates + if not bare_source_identifier_span(text, start, end) + and ( + (start, end) in wrapped + or contexts[start] is not None + or any(char.isdigit() or char in "+/=" for char in text[start:end]) + ) + ] + + +def markerless_private_key_body_spans(text: str) -> list[tuple[int, int]]: + spans = orphan_private_key_body_spans(text) + matches = list(PRIVATE_KEY_BODY_PATTERN.finditer(text)) + contexts = string_contexts_at(text, {match.start() for match in matches}) + spans.extend( + match.span() + for match in matches + if contexts[match.start()] is not None + and likely_private_key_token(match.group(0)) + ) + return sorted(set(spans)) + + +def pem_line_body_spans( + text: str, + in_pem: bool, +) -> tuple[list[tuple[int, int]], bool]: + markers = sorted( + [ + (match.start(), match.end(), True) + for match in PRIVATE_KEY_BEGIN_PATTERN.finditer(text) + ] + + [ + (match.start(), match.end(), False) + for match in PRIVATE_KEY_END_PATTERN.finditer(text) + ] + ) + if not markers and not in_pem: + return markerless_private_key_body_spans(text), False + spans: list[tuple[int, int]] = [] + cursor = 0 + for start, end, begins_pem in markers: + if in_pem: + spans.extend( + (cursor + body_start, cursor + body_end) + for body_start, body_end in explicit_private_key_body_spans( + text[cursor:start], + in_pem=in_pem, + ) + ) + else: + spans.extend( + (cursor + body_start, cursor + body_end) + for body_start, body_end in markerless_private_key_body_spans( + text[cursor:start] + ) + ) + if begins_pem: + if in_pem: + raise SystemExit("refusing review bundle with nested private-key BEGIN markers") + in_pem = True + else: + if not in_pem: + raise SystemExit( + "refusing review bundle with a private-key END marker but no visible BEGIN" + ) + in_pem = False + cursor = end + if in_pem: + spans.extend( + (cursor + body_start, cursor + body_end) + for body_start, body_end in explicit_private_key_body_spans( + text[cursor:], + in_pem=in_pem, + ) + ) + else: + spans.extend( + (cursor + body_start, cursor + body_end) + for body_start, body_end in markerless_private_key_body_spans(text[cursor:]) + ) + return spans, in_pem + + +def private_key_body_fragments(text: str) -> set[str]: + # Do not join short markerless chunks across lines. Without PEM boundaries, + # code and data are indistinguishable; explicit markers or long tokens redact. + fragments: set[str] = set() + in_private_key = False + for line in text.split("\n"): + spans, in_private_key = pem_line_body_spans(line, in_private_key) + fragments.update( + fragment + for start, end in spans + if (fragment := normalized_private_key_fragment(line, start, end)) + ) + if in_private_key: + raise SystemExit("refusing review bundle with an unterminated private-key block") + return fragments + + +def unified_diff_contents(patch: str) -> tuple[str, str]: + old_content: list[str] = [] + new_content: list[str] = [] + in_hunk = False + prefix_columns = 1 + for line in patch.split("\n"): + line = line.removesuffix("\r") + hunk_header = re.match(r"^(@{2,})", line) + if hunk_header: + old_content.append(DIFF_HUNK_CONTENT_BOUNDARY) + new_content.append(DIFF_HUNK_CONTENT_BOUNDARY) + in_hunk = True + prefix_columns = len(hunk_header.group(1)) - 1 + continue + if line.startswith("diff --"): + old_content.append(DIFF_HUNK_CONTENT_BOUNDARY) + new_content.append(DIFF_HUNK_CONTENT_BOUNDARY) + in_hunk = False + continue + prefix = line[:prefix_columns] + if in_hunk and len(prefix) == prefix_columns and set(prefix) <= {"+", "-", " "}: + content = line[prefix_columns:] + if set(prefix) == {" "}: + old_content.append(content) + new_content.append(content) + elif "+" in prefix and "-" not in prefix: + new_content.append(content) + elif "-" in prefix and "+" not in prefix: + old_content.append(content) + else: + old_content.append(content) + new_content.append(content) + return "\n".join(old_content), "\n".join(new_content) + + +REVIEW_SECURITY_REDACTION = ( + "[security-sensitive review material omitted before model review]" +) + + +def sensitive_repo_path_risk(rel: str) -> str | None: + normalized = rel.replace(os.sep, "/") + path = Path(normalized) + if secret_text_risk(normalized): + return "secret-like path" + credential_directory = any( + TRACKED_CREDENTIAL_DIR_PATTERN.fullmatch(part) + for part in path.parts[:-1] + ) + if ( + path_has_sensitive_part(normalized) + or credential_directory + or credential_store_path(normalized) + or token_credential_store_path(normalized) + ): + return "sensitive path" + if ( + any(pattern.search(normalized) for pattern in SENSITIVE_NAME_PATTERNS) + and not design_token_artifact_path(path, SENSITIVE_NAME_PATTERNS) + and not github_workflow_path(path) + ): + return "sensitive filename" + return None + + +def token_credential_store_path(normalized: str) -> bool: + path = Path(normalized) + parts = {part.lower() for part in path.parts} + return ( + bool(parts & {"token", "tokens"}) + and path.stem.lower() in TRACKED_TOKEN_CREDENTIAL_STEMS + and path.suffix.lower() in TRACKED_TOKEN_CREDENTIAL_EXTENSIONS + ) + + +def design_token_artifact_path( + path: Path, + sensitive_patterns: list[re.Pattern[str]], +) -> bool: + if re.fullmatch(r"design[-_]?tokens?\.json", path.name, re.IGNORECASE) is None: + return False + allowed_design_token_dirs = { + "design-token", + "design-tokens", + "design_token", + "design_tokens", + "token", + "tokens", + } + return not any( + part.lower() not in allowed_design_token_dirs + and any(pattern.search(part) for pattern in sensitive_patterns) + for part in path.parts[:-1] + ) + + +def github_workflow_path(path: Path) -> bool: + return ( + path.parts[:2] == (".github", "workflows") + and len(path.parts) == 3 + and path.suffix in {".yml", ".yaml"} + ) + + +def credential_store_path(normalized: str) -> bool: + path = Path(normalized) + credential_directory = any( + TRACKED_CREDENTIAL_DIR_PATTERN.fullmatch(part) + for part in path.parts[:-1] + ) + credential_data_file = path.suffix.lower() not in { + ".c", + ".cc", + ".cpp", + ".cs", + ".go", + ".h", + ".hpp", + ".java", + ".js", + ".jsx", + ".kt", + ".mjs", + ".php", + ".py", + ".rb", + ".rs", + ".sh", + ".swift", + ".ts", + ".tsx", + ".vue", + } + return credential_directory and credential_data_file and not skill_instruction_path(path) + + +def skill_instruction_path(path: Path) -> bool: + parts = tuple(part.lower() for part in path.parts) + skill_root = parts[:1] == ("skills",) or any( + parts[index : index + 2] in {(".agents", "skills"), (".claude", "skills")} + for index in range(len(parts) - 1) + ) + return skill_root and path.name.lower() in {"agents.md", "claude.md", "skill.md"} + + +def tracked_sensitive_repo_path_risk(rel: str) -> str | None: + normalized = rel.replace(os.sep, "/") + path = Path(normalized) + if secret_text_risk(normalized): + return "secret-like path" + parts = {part.lower() for part in path.parts} + if ( + "/.config/gcloud/" in f"/{normalized.lower()}/" + or f"/{normalized.lower()}".endswith("/.docker/config.json") + or parts & TRACKED_SENSITIVE_PATH_PARTS + or credential_store_path(normalized) + or token_credential_store_path(normalized) + ): + return "sensitive path" + if ( + any(pattern.search(normalized) for pattern in TRACKED_SENSITIVE_NAME_PATTERNS) + and not design_token_artifact_path(path, TRACKED_SENSITIVE_NAME_PATTERNS) + and not github_workflow_path(path) + ): + return "sensitive filename" + return None + + +def javascript_review_dialect(rel: str) -> str | None: + suffix = Path(rel).suffix.lower() + if suffix in {".cts", ".mts", ".ts", ".tsx"}: + return "typescript" + if suffix in {".cjs", ".js", ".jsx", ".mjs"}: + return "javascript" + return None + + +def git_c_unquote(value: str) -> str | None: + if len(value) < 2 or value[0] != '"' or value[-1] != '"': + return None + escapes = { + "a": 7, + "b": 8, + "f": 12, + "n": 10, + "r": 13, + "t": 9, + "v": 11, + "\\": 92, + '"': 34, + } + decoded = bytearray() + cursor = 1 + while cursor < len(value) - 1: + char = value[cursor] + if char != "\\": + decoded.extend(char.encode("utf-8")) + cursor += 1 + continue + cursor += 1 + if cursor >= len(value) - 1: + return None + escape = value[cursor] + if escape in escapes: + decoded.append(escapes[escape]) + cursor += 1 + continue + octal = re.match(r"[0-7]{1,3}", value[cursor:-1]) + if octal is None: + return None + decoded.append(int(octal.group(0), 8)) + cursor += octal.end() + try: + return decoded.decode("utf-8") + except UnicodeDecodeError: + return None + + +def diff_marker_path(value: str) -> str | None: + if value == "/dev/null": + return None + if value.startswith('"'): + decoded = git_c_unquote(value) + if decoded is None: + return None + value = decoded + if not value.startswith(("a/", "b/")): + return None + return value[2:] + + +def diff_section_paths(section: str) -> tuple[str | None, str | None]: + old_path: str | None = None + new_path: str | None = None + for line in section.splitlines(): + if line.startswith("@@"): + break + if line.startswith("--- "): + old_path = diff_marker_path(line[4:]) + elif line.startswith("+++ "): + new_path = diff_marker_path(line[4:]) + return old_path, new_path + + +def tracked_sensitive_paths(paths: list[str]) -> set[str]: + return { + rel + for rel in paths + if tracked_sensitive_repo_path_risk(rel) is not None + } + + +def omit_tracked_sensitive_diff_units( + patch: str, + paths: list[str], + blocked_paths: set[str], +) -> str: + if not blocked_paths: + return patch + units = review_bundle_units(patch) + diff_indexes = [ + index for index, unit in enumerate(units) if unit.startswith("diff --git ") + ] + if len(diff_indexes) != len(paths): + return REVIEW_SECURITY_REDACTION + "\n" + path_by_unit = dict(zip(diff_indexes, paths)) + retained = [ + unit + for index, unit in enumerate(units) + if path_by_unit.get(index) not in blocked_paths + ] + retained.insert(0, REVIEW_SECURITY_REDACTION + "\n") + return "".join(retained) + + +def redact_review_patch_metadata(patch: str) -> str: + redacted: list[str] = [] + metadata: list[str] = [] + in_hunk = False + prefix_columns = 1 + old_remaining: list[int] = [] + new_remaining = 0 + + def flush_metadata() -> None: + if not metadata: + return + text = "".join(metadata) + redacted.append( + REVIEW_SECURITY_REDACTION + "\n" + if secret_text_risk(text) + else text + ) + metadata.clear() + + def parse_range_count(token: str) -> int | None: + match = re.fullmatch(r"[+-]\d+(?:,(\d+))?", token) + if match is None: + return None + return int(match.group(1) or "1") + + for line in literal_lf_lines(patch): + body = line.rstrip("\r\n") + if body.startswith("diff --"): + flush_metadata() + metadata.append(line) + in_hunk = False + prefix_columns = 1 + old_remaining = [] + new_remaining = 0 + continue + hunk_header = re.match( + r"^(?P@{2,}) (?P.+?) (?P=marker)(?: .*)?$", + body, + ) + if hunk_header: + flush_metadata() + metadata.append(line) + marker = hunk_header.group("marker") + ranges = hunk_header.group("ranges").split() + old_counts = [ + count + for token in ranges + if token.startswith("-") + and (count := parse_range_count(token)) is not None + ] + new_counts = [ + count + for token in ranges + if token.startswith("+") + and (count := parse_range_count(token)) is not None + ] + prefix_columns = len(marker) - 1 + in_hunk = ( + len(old_counts) == prefix_columns + and len(new_counts) == 1 + ) + old_remaining = old_counts + new_remaining = new_counts[0] if new_counts else 0 + continue + prefix = body[:prefix_columns] + hunk_content = ( + in_hunk + and len(prefix) == prefix_columns + and set(prefix) <= {"+", "-", " "} + ) + if hunk_content: + flush_metadata() + redacted.append(line) + for index, marker in enumerate(prefix): + if marker != "+": + old_remaining[index] = max(0, old_remaining[index] - 1) + if any(marker != "-" for marker in prefix): + new_remaining = max(0, new_remaining - 1) + if new_remaining == 0 and not any(old_remaining): + in_hunk = False + else: + metadata.append(line) + flush_metadata() + return "".join(redacted) + + +def validate_review_patch( + label: str, + paths: list[str], + patch: str, + limit: int | None = None, +) -> str: + patch_bytes = len(patch.encode("utf-8")) + if limit is not None and patch_bytes > limit: + raise SystemExit( + f"{label} is too large to review safely " + f"({patch_bytes} bytes; limit {limit}); split the change into smaller review targets" + ) + blocked_paths = tracked_sensitive_paths(paths) + patch = omit_tracked_sensitive_diff_units(patch, paths, blocked_paths) + patch = redact_review_patch_metadata(patch) + patch_bytes = len(patch.encode("utf-8")) + if limit is not None and patch_bytes > limit: + raise SystemExit( + f"{label} is too large to review safely after metadata redaction " + f"({patch_bytes} bytes; limit {limit}); split the change into smaller review targets" + ) + return patch + + +def require_no_binary_diff(label: str, numstat: str) -> None: + binary_paths: list[str] = [] + for record in numstat.split("\0"): + if not record: + continue + fields = record.split("\t", 2) + if len(fields) == 3 and fields[0] == "-" and fields[1] == "-": + binary_paths.append(fields[2]) + if binary_paths: + details = "\n".join( + f"- {display_escape(path, 500)}" + for path in binary_paths[:20] + ) + more = f"\n... {len(binary_paths) - 20} more" if len(binary_paths) > 20 else "" + raise SystemExit( + f"refusing binary changes in {label} because their contents cannot be reviewed:\n" + f"{details}{more}" + ) + + +def require_no_gitlink_diff(label: str, raw_diff: str) -> None: + records = raw_diff.split("\0") + gitlink_paths: list[str] = [] + for index, record in enumerate(records): + if not record.startswith(":"): + continue + fields = record.split() + if len(fields) < 5: + continue + modes: list[str] = [] + for field_index, field in enumerate(fields): + candidate = field.lstrip(":") if field_index == 0 else field + if not re.fullmatch(r"[0-7]{6}", candidate): + break + modes.append(candidate) + if "160000" not in modes: + continue + path = records[index + 1] if index + 1 < len(records) else "" + gitlink_paths.append(path or "") + if gitlink_paths: + details = "\n".join( + f"- {display_escape(path, 500)}" + for path in gitlink_paths[:20] + ) + more = ( + f"\n... {len(gitlink_paths) - 20} more" + if len(gitlink_paths) > 20 + else "" + ) + raise SystemExit( + f"refusing gitlink/submodule changes in {label} because the referenced " + f"dependency contents are not present in the review bundle:\n{details}{more}" + ) + + +def file_bundle_risk( + repo: Path, + path: Path, + rel: str, + *, + allow_binary_omission: bool = False, +) -> str | None: + return file_bundle_snapshot( + repo, + path, + rel, + allow_binary_omission=allow_binary_omission, + )[2] + + +def file_bundle_snapshot( + repo: Path, + path: Path, + rel: str, + *, + allow_binary_omission: bool = False, +) -> tuple[str, bool, str | None]: + normalized = rel.replace(os.sep, "/") + path_risk = sensitive_repo_path_risk(normalized) + if path_risk: + return "", True, path_risk + if path.is_symlink(): + return "", True, "symlink" + try: + resolved = path.resolve(strict=True) + except OSError as exc: + return "", True, f"unreadable file: {exc}" + if not is_within(resolved, repo.resolve()): + return "", True, "path outside repository" + if not path.is_file(): + return "", True, "not a regular file" + try: + data, truncated = read_prefix(path, MAX_BUNDLE_TEXT_BYTES) + except SystemExit as exc: + return "", True, str(exc) + if b"\0" in data: + if allow_binary_omission: + return "[binary file omitted]", True, None + return "", True, "binary file" + if truncated: + return "", True, "file too large to scan safely" + try: + text = data.decode("utf-8") + except UnicodeDecodeError: + return "", True, "non-UTF-8 file" + return text, False, None + + +def collect_untracked_file_snapshots( + repo: Path, +) -> tuple[list[tuple[str, str, bool]], int]: + files = git_path_list( + repo, + *global_excludes_git_args(repo), + "ls-files", + "--others", + "--exclude-standard", + "-z", + ) + omitted = 0 + included: list[tuple[str, str, bool]] = [] + for rel in files: + content, truncated, risk = file_bundle_snapshot( + repo, + repo / rel, + rel, + allow_binary_omission=True, + ) + if risk: + if ( + sensitive_repo_path_risk(rel) is not None + or risk + in { + "secret-like content", + "symlink", + "path outside repository", + } + ): + omitted += 1 + else: + raise SystemExit( + "cannot safely include untracked file " + f"{display_escape(rel, 500)}: {risk}" + ) + else: + included.append((rel, content, truncated)) + return included, omitted + + +def safe_untracked_file_snapshots(repo: Path) -> list[tuple[str, str, bool]]: + snapshots, _omitted = collect_untracked_file_snapshots(repo) + return snapshots + + +def safe_untracked_files(repo: Path) -> list[str]: + return [rel for rel, _content, _truncated in safe_untracked_file_snapshots(repo)] + + +def local_status(repo: Path, untracked: list[str], *, redact: bool = False) -> str: + if redact: + return REVIEW_SECURITY_REDACTION + status = git(repo, "status", "--short", "--untracked-files=no").rstrip() + lines = [status] if status else [] + lines.extend(f"?? {rel}" for rel in untracked) + return "\n".join(lines) + + +def redact_secret_like_review_metadata(text: str) -> str: + return REVIEW_SECURITY_REDACTION if secret_text_risk(text) else text + + +def local_bundle(repo: Path) -> tuple[str, bool]: + staged_patch = git(repo, "diff", *SAFE_DIFF_FLAGS, "--cached", "--patch") + unstaged_patch = git(repo, "diff", *SAFE_DIFF_FLAGS, "--patch") + require_no_binary_diff( + "local staged diff", + git(repo, "diff", *SAFE_DIFF_FLAGS, "--cached", "--numstat", "-z"), + ) + require_no_binary_diff( + "local unstaged diff", + git(repo, "diff", *SAFE_DIFF_FLAGS, "--numstat", "-z"), + ) + require_no_gitlink_diff( + "local staged diff", + git(repo, "diff", *SAFE_DIFF_FLAGS, "--cached", "--raw", "-z"), + ) + require_no_gitlink_diff( + "local unstaged diff", + git(repo, "diff", *SAFE_DIFF_FLAGS, "--raw", "-z"), + ) + staged_paths = git_path_list( + repo, + "diff", + *SAFE_DIFF_FLAGS, + "--name-only", + "--cached", + "-z", + ) + unstaged_paths = git_path_list( + repo, + "diff", + *SAFE_DIFF_FLAGS, + "--name-only", + "-z", + ) + untracked_snapshots, omitted_untracked = collect_untracked_file_snapshots(repo) + untracked = [rel for rel, _content, _truncated in untracked_snapshots] + omitted_tracked = len( + tracked_sensitive_paths(staged_paths) | tracked_sensitive_paths(unstaged_paths) + ) + if ( + not staged_patch.strip() + and not unstaged_patch.strip() + and not untracked + and not omitted_untracked + ): + raise SystemExit("no local changes to review") + staged_patch = validate_review_patch("local staged diff", staged_paths, staged_patch) + unstaged_patch = validate_review_patch("local unstaged diff", unstaged_paths, unstaged_patch) + parts = [ + "# Git Status", + redact_secret_like_review_metadata( + local_status( + repo, + untracked, + redact=bool(omitted_tracked or omitted_untracked), + ) + ), + "# Staged Diff", + ( + REVIEW_SECURITY_REDACTION + if tracked_sensitive_paths(staged_paths) + else redact_secret_like_review_metadata( + git(repo, "diff", *SAFE_DIFF_FLAGS, "--cached", "--stat") + ) + ), + staged_patch, + "# Unstaged Diff", + ( + REVIEW_SECURITY_REDACTION + if tracked_sensitive_paths(unstaged_paths) + else redact_secret_like_review_metadata( + git(repo, "diff", *SAFE_DIFF_FLAGS, "--stat") + ) + ), + unstaged_patch, + ] + if omitted_tracked or omitted_untracked: + parts[0:0] = [ + "# Review Input Redactions", + REVIEW_SECURITY_REDACTION, + ( + f"Omitted tracked changes: {omitted_tracked}; " + f"omitted untracked files: {omitted_untracked}." + ), + ] + input_truncated = False + if untracked: + parts.append("# Untracked Files") + for rel, content, truncated in untracked_snapshots: + input_truncated = input_truncated or truncated + records = literal_lf_lines(content) or [""] + parts.append( + "# Untracked File\n" + f"path: {json.dumps(rel)}\n" + + "\n".join( + f"source-line {line_number}: {json.dumps(record)}" + for line_number, record in enumerate(records, start=1) + ) + ) + return "\n\n".join(parts), input_truncated + + +def source_file_fingerprint(path: Path) -> tuple[str, int, int, str]: + try: + before = os.stat(path, follow_symlinks=False) + except FileNotFoundError: + return "missing", 0, 0, "" + file_mode = stat.S_IMODE(before.st_mode) + if stat.S_ISLNK(before.st_mode): + try: + target = os.readlink(path) + after = os.stat(path, follow_symlinks=False) + except OSError as exc: + raise SystemExit( + f"unreadable file: {display_escape(path, 500)}: " + f"{display_escape(exc, 500)}" + ) from exc + if ( + before.st_dev, + before.st_ino, + before.st_mode, + before.st_size, + before.st_mtime_ns, + ) != ( + after.st_dev, + after.st_ino, + after.st_mode, + after.st_size, + after.st_mtime_ns, + ): + raise SystemExit( + f"file changed while reading: {display_escape(path, 500)}" + ) + data = os.fsencode(target) + return "symlink", file_mode, len(data), hashlib.sha256(data).hexdigest() + if not stat.S_ISREG(before.st_mode): + return "other", file_mode, before.st_size, "" + + descriptor: int | None = None + digest = hashlib.sha256() + try: + flags = ( + os.O_RDONLY + | getattr(os, "O_BINARY", 0) + | getattr(os, "O_CLOEXEC", 0) + | getattr(os, "O_NOFOLLOW", 0) + ) + descriptor = os.open(path, flags) + opened = os.fstat(descriptor) + if ( + not stat.S_ISREG(opened.st_mode) + or (before.st_dev, before.st_ino) != (opened.st_dev, opened.st_ino) + ): + raise OSError("file changed while opening") + while chunk := os.read(descriptor, 1024 * 1024): + digest.update(chunk) + after = os.fstat(descriptor) + if ( + opened.st_dev, + opened.st_ino, + opened.st_mode, + opened.st_size, + opened.st_mtime_ns, + ) != ( + after.st_dev, + after.st_ino, + after.st_mode, + after.st_size, + after.st_mtime_ns, + ): + raise OSError("file changed while reading") + except OSError as exc: + raise SystemExit( + f"unreadable file: {display_escape(path, 500)}: " + f"{display_escape(exc, 500)}" + ) from exc + finally: + if descriptor is not None: + os.close(descriptor) + return "file", file_mode, before.st_size, digest.hexdigest() + + +def source_tree_snapshot( + repo: Path, +) -> tuple[ + str, + str, + tuple[tuple[str, object], ...], +]: + head_result = git_result( + repo, + "rev-parse", + "--verify", + "HEAD", + check=False, + ) + head = head_result.stdout.strip() + if head_result.returncode != 0: + symbolic_result = git_result( + repo, + "symbolic-ref", + "-q", + "HEAD", + check=False, + ) + symbolic_head = symbolic_result.stdout.strip() + if symbolic_result.returncode != 0 or not symbolic_head: + raise SystemExit("unable to resolve HEAD for source snapshot") + ref_result = git_result( + repo, + "show-ref", + "--verify", + "--quiet", + symbolic_head, + check=False, + ) + if ref_result.returncode != 1: + raise SystemExit("unable to verify unborn HEAD for source snapshot") + head = f"unborn:{symbolic_head}" + index_entries = git( + repo, + "ls-files", + "--stage", + "-z", + ) + tracked = git_path_list(repo, "ls-files", "-z") + index_modes = { + rel: metadata.split(" ", 1)[0] + for record in index_entries.split("\0") + if record and "\t" in record + for metadata, rel in (record.split("\t", 1),) + } + untracked = git_path_list( + repo, + *global_excludes_git_args(repo), + "ls-files", + "--others", + "--exclude-standard", + "-z", + ) + fingerprints = tuple( + ( + rel, + source_tree_snapshot(repo / rel) + if index_modes.get(rel) == "160000" + and (repo / rel / ".git").exists() + else source_file_fingerprint(repo / rel), + ) + for rel in sorted(set(tracked + untracked)) + ) + return head, index_entries, fingerprints + + +def branch_bundle(repo: Path, base_ref: str) -> tuple[str, bool]: + base_ref = validate_git_ref(repo, base_ref, "base") + diff_range = f"{base_ref}...HEAD" + branch_patch = git( + repo, + "diff", + *SAFE_DIFF_FLAGS, + "--patch", + "--end-of-options", + diff_range, + ) + branch_paths = git_path_list( + repo, + "diff", + *SAFE_DIFF_FLAGS, + "--name-only", + "-z", + "--end-of-options", + diff_range, + ) + require_no_binary_diff( + "branch diff", + git( + repo, + "diff", + *SAFE_DIFF_FLAGS, + "--numstat", + "-z", + "--end-of-options", + diff_range, + ), + ) + require_no_gitlink_diff( + "branch diff", + git( + repo, + "diff", + *SAFE_DIFF_FLAGS, + "--raw", + "-z", + "--end-of-options", + diff_range, + ), + ) + omitted_tracked = bool(tracked_sensitive_paths(branch_paths)) + branch_patch = validate_review_patch("branch diff", branch_paths, branch_patch) + return "\n\n".join( + [ + "# Branch Diff", + ( + REVIEW_SECURITY_REDACTION + if secret_text_risk(base_ref) + else f"base: {base_ref}" + ), + ( + REVIEW_SECURITY_REDACTION + if omitted_tracked + else redact_secret_like_review_metadata( + git( + repo, + "diff", + *SAFE_DIFF_FLAGS, + "--stat", + "--end-of-options", + diff_range, + ) + ) + ), + branch_patch, + ] + ), False + + +def commit_bundle(repo: Path, commit_ref: str) -> tuple[str, bool]: + commit_ref = validate_git_ref(repo, commit_ref, "commit") + parents = git(repo, "rev-list", "--parents", "-n", "1", commit_ref).split() + if len(parents) > 2: + raise SystemExit( + "commit review does not accept merge commits; review the branch diff " + "or an individual parent-relative commit instead" + ) + commit_patch = git( + repo, + "show", + *SAFE_DIFF_FLAGS, + "--patch", + "--format=fuller", + "--end-of-options", + commit_ref, + ) + commit_paths = git_path_list( + repo, + "show", + *SAFE_DIFF_FLAGS, + "--name-only", + "--format=", + "-z", + "--end-of-options", + commit_ref, + ) + require_no_binary_diff( + "commit diff", + git( + repo, + "show", + *SAFE_DIFF_FLAGS, + "--numstat", + "--format=", + "-z", + "--end-of-options", + commit_ref, + ), + ) + require_no_gitlink_diff( + "commit diff", + git( + repo, + "show", + *SAFE_DIFF_FLAGS, + "--raw", + "--format=", + "-z", + "--end-of-options", + commit_ref, + ), + ) + omitted_tracked = bool(tracked_sensitive_paths(commit_paths)) + commit_patch = validate_review_patch("commit diff", commit_paths, commit_patch) + commit_summary = git( + repo, + "show", + *SAFE_DIFF_FLAGS, + "--stat", + "--format=fuller", + "--end-of-options", + commit_ref, + ) + if omitted_tracked or secret_text_risk(commit_summary): + commit_summary = REVIEW_SECURITY_REDACTION + return "\n\n".join( + [ + "# Commit Diff", + ( + REVIEW_SECURITY_REDACTION + if secret_text_risk(commit_ref) + else f"commit: {commit_ref}" + ), + commit_summary, + commit_patch, ] + ), False + + +def review_paths(repo: Path, target: str, target_ref: str | None, commit_ref: str) -> set[str]: + names: set[str] = set() + if target == "local": + names.update(git_path_list(repo, "diff", *SAFE_DIFF_FLAGS, "--name-only", "--cached", "-z")) + names.update(git_path_list(repo, "diff", *SAFE_DIFF_FLAGS, "--name-only", "-z")) + names.update(safe_untracked_files(repo)) elif target == "branch": assert target_ref - sources = [git(repo, "diff", "--name-only", f"{target_ref}...HEAD")] + target_ref = validate_git_ref(repo, target_ref, "base") + names.update( + git_path_list( + repo, + "diff", + *SAFE_DIFF_FLAGS, + "--name-only", + "-z", + "--end-of-options", + f"{target_ref}...HEAD", + ) + ) else: - sources = [git(repo, "show", "--name-only", "--format=", commit_ref)] - for source in sources: - for line in source.splitlines(): - path = line.strip() - if path: - names.add(path) + commit_ref = validate_git_ref(repo, commit_ref, "commit") + names.update( + git_path_list( + repo, + "show", + *SAFE_DIFF_FLAGS, + "--name-only", + "--format=", + "-z", + "--end-of-options", + commit_ref, + ) + ) return names -def load_extra_prompt(args: argparse.Namespace) -> str: +def validate_evidence_file(repo: Path, raw_path: str, label: str) -> tuple[Path, str, bool]: + original = Path(raw_path) + if original.is_absolute() or ".." in original.parts or not original.parts: + raise SystemExit(f"{label} must be a repo-relative path: {raw_path}") + raw_rel = original.as_posix() + if path_has_sensitive_part(raw_rel): + raise SystemExit(f"refusing to include sensitive {label}: {raw_rel}") + if raw_repo_path_has_symlink_component(repo, original): + raise SystemExit(f"refusing to include symlinked {label}: {raw_path}") + path = (repo / original).resolve() + if not is_within(path, repo.resolve()): + raise SystemExit(f"{label} must be inside the reviewed repository: {raw_path}") + rel = str(path.relative_to(repo.resolve())) + content, truncated, risk = file_bundle_snapshot(repo, path, rel) + if risk: + raise SystemExit(f"refusing to include unsafe {label}: {rel} ({risk})") + require_no_secret_values(f"{label} {rel}", content) + return path, content, truncated + + +def load_extra_prompt(args: argparse.Namespace, repo: Path) -> tuple[str, bool]: chunks: list[str] = [] + input_truncated = False for value in args.prompt or []: + require_no_secret_values("--prompt", value) chunks.append(value) for path in args.prompt_file or []: - chunks.append(Path(path).read_text()) - return "\n\n".join(chunks) + resolved, content, truncated = validate_evidence_file(repo, path, "--prompt-file") + input_truncated = input_truncated or truncated + chunks.append(f"# Prompt file: {resolved.relative_to(repo.resolve())}\n{content}") + return "\n\n".join(chunks), input_truncated -def load_datasets(args: argparse.Namespace) -> str: +def load_datasets(args: argparse.Namespace, repo: Path) -> tuple[str, bool]: chunks: list[str] = [] + input_truncated = False for spec in args.dataset or []: - path = Path(spec) - if path.is_dir(): - raise SystemExit(f"--dataset must be a file, got directory: {path}") - chunks.append(f"# Dataset: {path}\n{read_text(path)}") - return "\n\n".join(chunks) + path, content, truncated = validate_evidence_file(repo, spec, "--dataset") + input_truncated = input_truncated or truncated + chunks.append(f"# Dataset: {path.relative_to(repo.resolve())}\n{content}") + return "\n\n".join(chunks), input_truncated -def build_prompt(repo: Path, target: str, target_ref: str | None, bundle: str, extra_prompt: str, datasets: str) -> str: - target_line = f"{target} {target_ref}" if target_ref else target +def review_scope_policy() -> str: return textwrap.dedent( + """ + Review scope discipline: + - This helper is a closeout gate. Do not turn a narrow patch into a broad + redesign request. + - Report a finding only when this diff introduces or exposes a concrete + defect that must be fixed before this target can land. + - If the best fix requires a new protocol, config, storage, public API, + release process, migration, owner-boundary move, or canonical contract, + say that directly in the finding and keep the finding tied to the + smallest changed line that proves the current patch is not landable. + - Do not ask for sibling-surface hardening, cleanup, refactors, or + follow-up architecture work unless the current diff is incorrect + without that work. + - Prefer the smallest correct pre-merge fix. A broader ideal design is + not an actionable finding unless the current patch cannot safely land. + - If this is release-branch or release-process work, apply freeze + discipline. Report only release blockers, exact backport regressions, + install/upgrade breakage, crashes, data loss, concrete security + exposure, or release-infrastructure failures. Non-blocking design, + cleanup, and hardening concerns belong on main as follow-ups. + """ + ).strip() + + +def utf8_size(text: str) -> int: + return len(text.encode("utf-8")) + + +def split_utf8_fragment( + text: str, + limit: int, + first_limit: int | None = None, +) -> list[str]: + current_limit = first_limit or limit + if min(limit, current_limit) < 4: + raise SystemExit("review chunk byte limit is too small") + fragments: list[str] = [] + current: list[str] = [] + current_bytes = 0 + for character in text: + character_bytes = utf8_size(character) + if current and current_bytes + character_bytes > current_limit: + fragments.append("".join(current)) + current = [] + current_bytes = 0 + current_limit = limit + current.append(character) + current_bytes += character_bytes + if current: + fragments.append("".join(current)) + return fragments + + +def review_bundle_units(bundle: str) -> list[str]: + section_boundaries = { + "# Git Status\n", + "# Staged Diff\n", + "# Unstaged Diff\n", + "# Untracked Files\n", + "# Untracked File\n", + "# Branch Diff\n", + "# Commit Diff\n", + } + units: list[str] = [] + current: list[str] = [] + for line in literal_lf_lines(bundle): + boundary = line.startswith("diff --git ") or line in section_boundaries + if boundary and current: + units.append("".join(current)) + current = [] + current.append(line) + if current: + units.append("".join(current)) + return units + + +def literal_lf_lines(text: str) -> list[str]: + parts = text.split("\n") + lines = [part + "\n" for part in parts[:-1]] + if parts[-1]: + lines.append(parts[-1]) + return lines + + +def update_review_chunk_context( + context: list[str], + line: str, + next_new_line: int | None, + next_old_line: int | None, + in_hunk: bool, +) -> tuple[int | None, int | None, bool]: + if line.startswith("diff --git "): + context[:] = [line] + return None, None, False + if line == "# Untracked File\n": + context[:] = [line] + return None, None, False + if context == ["# Untracked File\n"] and line.startswith("path: "): + context.append(line) + return None, None, False + if not context: + return next_new_line, next_old_line, in_hunk + if context[0] == "# Untracked File\n": + match = re.match(r"^source-line (\d+): ", line) + if match: + return int(match.group(1)) + 1, None, False + return next_new_line, None, False + if not in_hunk and line.startswith(("--- ", "+++ ")): + header_prefix = line[:4] + context[:] = [entry for entry in context if not entry.startswith(header_prefix)] + context.append(line) + return next_new_line, next_old_line, False + if line.startswith("@@ "): + context[:] = [entry for entry in context if not entry.startswith("@@ ")] + context.append(line) + match = re.match(r"^@@ -(\d+)(?:,\d+)? \+(\d+)(?:,\d+)? @@", line) + if not match: + return None, None, True + return int(match.group(2)), int(match.group(1)), True + if in_hunk and line.startswith(" "): + return increment_line(next_new_line), increment_line(next_old_line), True + if in_hunk and line.startswith("+"): + return increment_line(next_new_line), next_old_line, True + if in_hunk and line.startswith("-"): + return next_new_line, increment_line(next_old_line), True + return next_new_line, next_old_line, in_hunk + + +def increment_line(line: int | None) -> int | None: + return line + 1 if line is not None else None + + +def compact_review_chunk_context(context: list[str]) -> list[str]: + if not context or context[0] == "# Untracked File\n": + return list(context) + new_header = next((entry for entry in context if entry.startswith("+++ ")), None) + old_header = next((entry for entry in context if entry.startswith("--- ")), None) + hunk_header = next((entry for entry in context if entry.startswith("@@ ")), None) + path_header = new_header if new_header and new_header != "+++ /dev/null\n" else old_header + compact = [path_header or context[0]] + if hunk_header: + compact.append(hunk_header) + return compact + + +def review_chunk_context( + context: list[str], + next_new_line: int | None, + next_old_line: int | None, + *, + continued_line: bool = False, + diff_line_marker: str | None = None, +) -> str: + lines = compact_review_chunk_context(context) + if context and context[0] == "# Untracked File\n" and next_new_line is not None: + lines.append(f"[Continuation begins at untracked source line {next_new_line}.]\n") + elif ( + next_new_line is not None + and next_new_line >= 1 + and next_old_line is not None + and next_old_line >= 1 + and next_new_line != next_old_line + ): + lines.append( + f"[Continuation position: new-file line {next_new_line}; " + f"old-file line {next_old_line}.]\n" + ) + elif next_new_line is not None and next_new_line >= 1: + lines.append(f"[Continuation begins at new-file line {next_new_line}.]\n") + elif next_old_line is not None and next_old_line >= 1: + lines.append( + f"[Continuation begins at old-file line {next_old_line}; " + "use this positive line for deleted content.]\n" + ) + if continued_line: + if diff_line_marker: + lines.append( + "[The change content below continues a unified-diff line whose " + f"original marker is `{diff_line_marker}`.]\n" + ) + else: + lines.append("[The change content below continues the preceding long line.]\n") + text = "".join(lines) + if utf8_size(text) > MAX_REVIEW_CHUNK_CONTEXT_BYTES: + raise SystemExit( + "review continuation context exceeds the bounded prompt allowance; " + "shorten the changed path or split the review target" + ) + return text + + +def split_oversized_review_unit( + unit: str, + limit: int, + first_limit: int | None = None, +) -> list[ReviewChunk]: + chunks: list[ReviewChunk] = [] + current: list[str] = [] + current_bytes = 0 + current_context = "" + context: list[str] = [] + next_new_line: int | None = None + next_old_line: int | None = None + in_hunk = False + + def current_limit() -> int: + return first_limit if not chunks and first_limit is not None else limit + + def flush() -> None: + nonlocal current, current_bytes, current_context + if current: + chunks.append(ReviewChunk("".join(current), current_context)) + current = [] + current_bytes = 0 + current_context = "" + + for line in literal_lf_lines(unit): + line_bytes = utf8_size(line) + diff_line_marker = line[0] if in_hunk and line.startswith(("+", "-", " ")) else None + untracked_source_line = None + if context and context[0] == "# Untracked File\n": + match = re.match(r"^source-line (\d+): ", line) + if match: + untracked_source_line = int(match.group(1)) + chunk_limit = current_limit() + if current and current_bytes + line_bytes > chunk_limit: + if line_bytes <= limit: + flush() + chunk_limit = current_limit() + else: + remaining_line_bytes = chunk_limit - current_bytes + if remaining_line_bytes < 4: + flush() + chunk_limit = current_limit() + else: + fragments = split_utf8_fragment( + line, + limit, + first_limit=remaining_line_bytes, + ) + current.append(fragments[0]) + flush() + continued_context = review_chunk_context( + context, + untracked_source_line or next_new_line, + next_old_line, + continued_line=True, + diff_line_marker=diff_line_marker, + ) + chunks.extend( + ReviewChunk(fragment, continued_context) + for fragment in fragments[1:-1] + ) + if len(fragments) > 1: + current = [fragments[-1]] + current_bytes = utf8_size(fragments[-1]) + current_context = continued_context + next_new_line, next_old_line, in_hunk = update_review_chunk_context( + context, + line, + next_new_line, + next_old_line, + in_hunk, + ) + continue + if line_bytes > chunk_limit: + flush() + fragments = split_utf8_fragment( + line, + limit, + first_limit=chunk_limit, + ) + for index, fragment in enumerate(fragments[:-1]): + chunks.append( + ReviewChunk( + fragment, + review_chunk_context( + context, + untracked_source_line or next_new_line, + next_old_line, + continued_line=index > 0, + diff_line_marker=diff_line_marker, + ), + ) + ) + current = [fragments[-1]] + current_bytes = utf8_size(fragments[-1]) + current_context = review_chunk_context( + context, + untracked_source_line or next_new_line, + next_old_line, + continued_line=len(fragments) > 1, + diff_line_marker=diff_line_marker, + ) + next_new_line, next_old_line, in_hunk = update_review_chunk_context( + context, + line, + next_new_line, + next_old_line, + in_hunk, + ) + continue + if not current: + current_context = review_chunk_context(context, next_new_line, next_old_line) + current.append(line) + current_bytes += line_bytes + next_new_line, next_old_line, in_hunk = update_review_chunk_context( + context, + line, + next_new_line, + next_old_line, + in_hunk, + ) + flush() + return chunks + + +def split_review_bundle(bundle: str, limit: int) -> list[ReviewChunk]: + if utf8_size(bundle) <= limit: + return [ReviewChunk(bundle)] + chunks: list[ReviewChunk] = [] + pending: ReviewChunk | None = None + + def flush_pending() -> None: + nonlocal pending + if pending is not None: + chunks.append(pending) + pending = None + + for unit in review_bundle_units(bundle): + unit_bytes = utf8_size(unit) + pending_bytes = utf8_size(pending.content) if pending else 0 + remaining = limit - pending_bytes + if pending and unit_bytes <= remaining: + pending = ReviewChunk(pending.content + unit, pending.context) + continue + if pending and remaining >= 256: + first_line = literal_lf_lines(unit)[0] + if utf8_size(first_line) > remaining and utf8_size(first_line) <= limit: + flush_pending() + pieces = split_oversized_review_unit(unit, limit) + chunks.extend(pieces[:-1]) + pending = pieces[-1] + continue + pieces = split_oversized_review_unit( + unit, + limit, + first_limit=remaining, + ) + first, *rest = pieces + pending = ReviewChunk(pending.content + first.content, pending.context) + flush_pending() + if rest: + chunks.extend(rest[:-1]) + pending = rest[-1] + continue + flush_pending() + if unit_bytes <= limit: + pending = ReviewChunk(unit) + continue + pieces = split_oversized_review_unit(unit, limit) + chunks.extend(pieces[:-1]) + pending = pieces[-1] + flush_pending() + if "".join(chunk.content for chunk in chunks) != bundle: + raise SystemExit("internal error: review bundle chunking omitted or reordered input") + return chunks + + +def render_review_prompt( + repo: Path, + target: str, + target_ref: str | None, + chunk: ReviewChunk, + extra_prompt: str, + datasets: str, + chunk_position: tuple[int, int] | None = None, +) -> str: + safe_target_ref = ( + REVIEW_SECURITY_REDACTION + if target_ref and secret_text_risk(target_ref) + else target_ref + ) + target_line = f"{target} {safe_target_ref}" if safe_target_ref else target + branch = current_branch(repo) + if secret_text_risk(branch): + branch = REVIEW_SECURITY_REDACTION + scope_policy = review_scope_policy() + chunk_policy = "" + if chunk_position: + index, total = chunk_position + chunk_policy = textwrap.dedent( + f""" + Oversized review bundle chunk: {index}/{total} + - The complete validated change is distributed across all {total} chunks. + - Original change bytes appear exactly once across the chunk sequence. + - Continuation context may repeat file and hunk headers; it is not extra change content. + - Report every actionable defect demonstrated by this chunk. Reports from all chunks are merged after every pass finishes. + """ + ).strip() + if chunk.context: + chunk_policy += "\n\n# Continuation Context\n" + chunk.context + instructions = textwrap.dedent( f""" You are a senior code reviewer. Review the provided git change bundle only. @@ -633,30 +8672,138 @@ def build_prompt(repo: Path, target: str, target_ref: str | None, bundle: str, e - Do not modify files. - Do not invoke nested reviewers or review tools. - Forbidden nested review commands include: codex review, autoreview, claude review, oracle review. - - You may use read-only tools and web search to inspect files, dependency contracts, upstream docs, current behavior, and security implications. + - The review sandbox is intentionally empty. The change bundle and explicit prompt or datasets are the only reviewed-repository source. Read-only tools cannot access unchanged repository files. + - You may use read-only tools and web search to inspect external dependency contracts, upstream docs, current public behavior, and security implications. + - Do not report a missing import, symbol, definition, call site, config entry, or other unchanged context solely because it is absent from the change bundle. Such a finding requires direct proof in the bundle or explicit datasets. - Shell commands, if available, must be read-only inspection commands. Do not run tests, formatters, package installs, generators, network mutation commands, git mutation commands, or commands that write files. - Report only actionable defects introduced or exposed by this change. - Prefer high-signal findings over style feedback. + - Report EVERY distinct actionable defect in this single pass, ordered most severe first. Each review round costs the caller a full fix-test-review cycle; withholding a known defect until a later round wastes one. + - Before returning, sweep the bundle once more for independent defects in other files or failure modes that you may have stopped scanning for after an earlier find. - Include security findings: injection, secret leaks, authz/authn bypass, path traversal, unsafe deserialization, unsafe filesystem or shell use, privacy leaks, and credential handling. - Do not reject legitimate functionality merely because it touches shell, filesystem, network, auth, or sensitive data. Report a security finding only when the patch creates a concrete exploitable risk, removes an important safety check, or lacks validation at a trust boundary. + - Security-sensitive bundle material may be redacted or omitted before review. Continue reviewing the material that is present. A redaction notice is not itself a defect and does not prove either safety or vulnerability in the omitted material. - For each finding, use the smallest file/line location that demonstrates the issue. - If there are no actionable findings, return an empty findings array and mark the patch correct. Review target: {target_line} - Repository: {repo} + Current branch: {branch} + Review sandbox: . (intentionally contains no reviewed repository files) + + {scope_policy} + + {chunk_policy} {extra_prompt} {datasets} # Change Bundle - {bundle} """ ).strip() + return instructions + "\n" + chunk.content + + +def build_prompt(repo: Path, target: str, target_ref: str | None, bundle: str, extra_prompt: str, datasets: str) -> str: + prompt = render_review_prompt( + repo, + target, + target_ref, + ReviewChunk(bundle), + extra_prompt, + datasets, + ) + prompt_bytes = len(prompt.encode("utf-8")) + if prompt_bytes > MAX_REVIEW_PROMPT_BYTES: + raise SystemExit( + f"review input is {prompt_bytes} bytes, exceeding the {MAX_REVIEW_PROMPT_BYTES}-byte aggregate limit; " + "reduce the change, prompt files, or datasets" + ) + return prompt + + +def build_review_prompts( + repo: Path, + target: str, + target_ref: str | None, + bundle: str, + extra_prompt: str, + datasets: str, +) -> list[str]: + full_prompt = render_review_prompt( + repo, + target, + target_ref, + ReviewChunk(bundle), + extra_prompt, + datasets, + ) + if utf8_size(full_prompt) <= MAX_REVIEW_PROMPT_BYTES: + return [full_prompt] + + empty_chunk_prompt = render_review_prompt( + repo, + target, + target_ref, + ReviewChunk(""), + extra_prompt, + datasets, + (999_999, 999_999), + ) + content_limit = ( + MAX_REVIEW_PROMPT_BYTES + - utf8_size(empty_chunk_prompt) + - MAX_REVIEW_CHUNK_CONTEXT_BYTES + - 4_096 + ) + if content_limit < 16_000: + raise SystemExit( + "review prompt files and datasets leave too little room for change chunks; " + "reduce the extra review context" + ) + if utf8_size(bundle) > content_limit * MAX_REVIEW_PASSES: + raise SystemExit( + f"review bundle requires more than {MAX_REVIEW_PASSES} bounded passes; " + "reduce or split the change before review" + ) + + for _attempt in range(4): + chunks = split_review_bundle(bundle, content_limit) + if len(chunks) > MAX_REVIEW_PASSES: + raise SystemExit( + f"review bundle requires {len(chunks)} bounded passes; " + f"limit {MAX_REVIEW_PASSES}; reduce or split the change before review" + ) + prompts = [ + render_review_prompt( + repo, + target, + target_ref, + chunk, + extra_prompt, + datasets, + (index, len(chunks)), + ) + for index, chunk in enumerate(chunks, start=1) + ] + largest = max(utf8_size(prompt) for prompt in prompts) + if largest <= MAX_REVIEW_PROMPT_BYTES: + return prompts + content_limit -= largest - MAX_REVIEW_PROMPT_BYTES + 1_024 + if content_limit < 16_000: + break + raise SystemExit( + "unable to partition the review bundle within the aggregate prompt limit" + ) -def write_json_temp(data: dict[str, Any]) -> Path: - handle = tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) +def write_json_temp(data: dict[str, Any], temp_root: Path) -> Path: + handle = tempfile.NamedTemporaryFile( + "w", + suffix=".json", + delete=False, + dir=temp_root, + ) with handle: json.dump(data, handle) return Path(handle.name) @@ -666,17 +8813,202 @@ def toml_quoted_key_segment(value: str) -> str: return json.dumps(value) -def codex_config_isolation_flags(repo: Path) -> list[str]: +def toml_inline_string_table(values: dict[str, str]) -> str: + entries = ", ".join(f"{key}={json.dumps(value)}" for key, value in sorted(values.items())) + return "{" + entries + "}" + + +def codex_config_isolation_flags(repo: Path, runtime_root: Path) -> list[str]: + tool_env = toml_inline_string_table(codex_tool_git_env()) + state_home = runtime_root / "state" + log_dir = runtime_root / "log" + state_home.mkdir(parents=True, exist_ok=True) + log_dir.mkdir(parents=True, exist_ok=True) return [ "-c", "project_doc_max_bytes=0", "-c", + f"sqlite_home={json.dumps(str(state_home.resolve()))}", + "-c", + f"log_dir={json.dumps(str(log_dir.resolve()))}", + "-c", + "features.shell_snapshot=false", + "-c", + "features.hooks=false", + "-c", + "features.plugins=false", + "-c", + "skills.include_instructions=false", + "-c", + "skills.config=[]", + "-c", f"projects.{toml_quoted_key_segment(str(repo.resolve()))}.trust_level=\"untrusted\"", + "-c", + 'shell_environment_policy.inherit="core"', + "-c", + "shell_environment_policy.ignore_default_excludes=false", + "-c", + f"shell_environment_policy.set={tool_env}", + "-c", + "shell_environment_policy.experimental_use_profile=false", + "-c", + "allow_login_shell=false", + "-c", + 'default_permissions="autoreview"', + "-c", + 'permissions.autoreview.filesystem={":minimal"="read",":workspace_roots"="read"}', ] +def parse_codex_auth_config_fallback(text: str) -> dict[str, Any]: + config: dict[str, Any] = {} + pending_key: str | None = None + pending_value: list[str] = [] + for raw_line in text.splitlines(): + line = raw_line.strip() + if pending_key is not None: + pending_value.append(raw_line) + try: + config[pending_key] = ast.literal_eval("\n".join(pending_value)) + except SyntaxError: + continue + except ValueError: + pending_key = None + pending_value = [] + continue + pending_key = None + pending_value = [] + continue + if not line or line.startswith("#"): + continue + if line.startswith("["): + break + match = re.fullmatch( + r"(cli_auth_credentials_store|forced_login_method|forced_chatgpt_workspace_id)\s*=\s*(.+)", + line, + ) + if not match: + continue + key, value_text = match.groups() + try: + config[key] = ast.literal_eval(value_text) + except SyntaxError: + if value_text.lstrip().startswith("["): + pending_key = key + pending_value = [value_text] + except ValueError: + continue + return config + + +def load_codex_auth_config(path: Path) -> dict[str, Any]: + try: + text = path.read_text() + except OSError: + return {} + try: + import tomllib + except ModuleNotFoundError: + return parse_codex_auth_config_fallback(text) + try: + config = tomllib.loads(text) + except ValueError: + return {} + return config if isinstance(config, dict) else {} + + +def codex_source_home(repo: Path) -> Path | None: + raw = os.environ.get("CODEX_HOME", "").strip() + candidate = Path(raw).expanduser() if raw else Path.home() / ".codex" + try: + resolved = candidate.resolve() + except OSError: + return None + return ( + resolved + if resolved.is_dir() and external_env_path(repo, str(resolved)) + else None + ) + + +def codex_auth_config_flags(repo: Path, *, force_file: bool = False) -> list[str]: + codex_home = codex_source_home(repo) + if codex_home is None: + return ["-c", 'cli_auth_credentials_store="file"'] if force_file else [] + config = load_codex_auth_config(codex_home / "config.toml") + + allowed_values = { + "forced_login_method": {"chatgpt", "api"}, + } + flags: list[str] = ( + ["-c", 'cli_auth_credentials_store="file"'] if force_file else [] + ) + if not force_file: + value = config.get("cli_auth_credentials_store") + if isinstance(value, str) and value in {"file", "keyring", "auto", "ephemeral"}: + flags.extend(["-c", f"cli_auth_credentials_store={json.dumps(value)}"]) + for key, allowed in allowed_values.items(): + value = config.get(key) + if isinstance(value, str) and value in allowed: + flags.extend(["-c", f"{key}={json.dumps(value)}"]) + workspace_ids = config.get("forced_chatgpt_workspace_id") + if isinstance(workspace_ids, str) and workspace_ids.strip(): + flags.extend(["-c", f"forced_chatgpt_workspace_id={json.dumps(workspace_ids.strip())}"]) + elif isinstance(workspace_ids, list): + normalized_workspace_ids = [ + value.strip() + for value in workspace_ids + if isinstance(value, str) and value.strip() + ] + if normalized_workspace_ids: + flags.extend(["-c", f"forced_chatgpt_workspace_id={json.dumps(normalized_workspace_ids)}"]) + return flags + + +def prepare_codex_runtime_auth( + repo: Path, + runtime_codex_home: Path, +) -> bool: + source_home = codex_source_home(repo) + if source_home is None: + return False + config = load_codex_auth_config(source_home / "config.toml") + credential_store = config.get("cli_auth_credentials_store") + if credential_store not in {None, "file"}: + return False + source_auth = source_home / "auth.json" + try: + source_stat = source_auth.lstat() + except OSError: + return False + if not stat.S_ISREG(source_stat.st_mode): + return False + try: + data, truncated = read_prefix(source_auth, 1_000_000) + parsed = json.loads(data) + except (OSError, SystemExit, json.JSONDecodeError): + return False + if truncated or not isinstance(parsed, dict): + return False + runtime_codex_home.mkdir(parents=True, exist_ok=True) + runtime_auth = runtime_codex_home / "auth.json" + # Codex refreshes file auth in place. A filesystem link preserves those + # native writes without a copy-back race against another Codex process. + try: + os.link(source_auth, runtime_auth) + except OSError: + try: + runtime_auth.symlink_to(source_auth) + except OSError as exc: + raise SystemExit( + "unable to isolate Codex file authentication without " + "discarding refreshed credentials" + ) from exc + return True + + def codex_exec_isolation_flags() -> list[str]: - return ["--ignore-rules"] + return ["--ignore-user-config", "--ignore-rules", "--skip-git-repo-check"] def claude_review_isolation_flags() -> list[str]: @@ -711,20 +9043,30 @@ def parse_cli_version(text: str) -> tuple[int, int, int] | None: def ensure_claude_isolation_supported(args: argparse.Namespace, repo: Path) -> None: claude_bin = resolve_command(args.claude_bin, repo) - result = run([claude_bin, "--version"], repo, check=False) + engine_env = safe_engine_env( + repo, + [Path(claude_bin).parent], + engine="claude", + ) + temp_root = safe_temp_root(repo) + result = run([claude_bin, "--version"], temp_root, check=False, env=engine_env) + selected_models = [args.model, *(getattr(args, "fallback_model", "") or "").split(",")] + uses_fable = any(model in {"claude-fable-5", "fable"} for model in selected_models) + minimum_version = CLAUDE_FABLE_MIN_VERSION if uses_fable else CLAUDE_SAFE_MODE_MIN_VERSION + version_reason = "for claude-fable-5" if uses_fable else "for --safe-mode" if result.returncode != 0: - raise SystemExit(f"claude engine requires Claude Code >= {format_version(CLAUDE_SAFE_MODE_MIN_VERSION)}; --version failed") + raise SystemExit(f"claude engine requires Claude Code >= {format_version(minimum_version)}; --version failed") version = parse_cli_version(result.stdout or result.stderr) if version is None: - raise SystemExit(f"claude engine requires Claude Code >= {format_version(CLAUDE_SAFE_MODE_MIN_VERSION)} for --safe-mode; could not parse --version output") - if version < CLAUDE_SAFE_MODE_MIN_VERSION: + raise SystemExit(f"claude engine requires Claude Code >= {format_version(minimum_version)} {version_reason}; could not parse --version output") + if version < minimum_version: raise SystemExit( - f"claude engine requires Claude Code >= {format_version(CLAUDE_SAFE_MODE_MIN_VERSION)} " - f"for --safe-mode (found {format_version(version)})" + f"claude engine requires Claude Code >= {format_version(minimum_version)} " + f"{version_reason} (found {format_version(version)})" ) - help_result = run([claude_bin, "--help"], repo, check=False) + help_result = run([claude_bin, "--help"], temp_root, check=False, env=engine_env) help_text = f"{help_result.stdout}\n{help_result.stderr}" - required_flags = ["--safe-mode", "--setting-sources", "--strict-mcp-config", "--disallowedTools"] + required_flags = ["--safe-mode", "--setting-sources", "--strict-mcp-config", "--disallowedTools", "--tools"] missing = [flag for flag in required_flags if flag not in help_text] if help_result.returncode != 0 or missing: detail = ", ".join(missing) if missing else "--help failed" @@ -733,10 +9075,14 @@ def ensure_claude_isolation_supported(args: argparse.Namespace, repo: Path) -> N def ensure_pi_isolation_supported(args: argparse.Namespace, repo: Path) -> str: pi_bin = resolve_command(args.pi_bin, repo) - with tempfile.TemporaryDirectory(prefix="autoreview-pi-probe.") as tempdir: + engine_env = safe_engine_env(repo, [Path(pi_bin).parent], engine="pi") + with tempfile.TemporaryDirectory( + prefix="autoreview-pi-probe.", + dir=safe_temp_root(repo), + ) as tempdir: probe_cwd = Path(tempdir) - result = run([pi_bin, "--version"], probe_cwd, check=False) - help_result = run([pi_bin, "--help"], probe_cwd, check=False) + result = run([pi_bin, "--version"], probe_cwd, check=False, env=engine_env) + help_result = run([pi_bin, "--help"], probe_cwd, check=False, env=engine_env) if result.returncode != 0: raise SystemExit(f"pi engine requires Pi >= {format_version(PI_TRUST_ISOLATION_MIN_VERSION)}; --version failed") version = parse_cli_version(f"{result.stdout}\n{result.stderr}") @@ -751,7 +9097,6 @@ def ensure_pi_isolation_supported(args: argparse.Namespace, repo: Path) -> str: required_flags = [ "--print", *pi_review_isolation_flags(), - "--tools", "--no-tools", "--thinking", ] @@ -766,19 +9111,139 @@ def format_version(version: tuple[int, int, int]) -> str: return ".".join(str(part) for part in version) -def run_codex(args: argparse.Namespace, repo: Path, prompt: str) -> str: - if not args.tools: - raise SystemExit("--no-tools is not supported by the Codex engine; use --engine claude --no-tools for a no-tools run") - schema_path = write_json_temp(SCHEMA) - output_path = Path(tempfile.NamedTemporaryFile("w", suffix=".json", delete=False).name) - cmd = [resolve_command(args.codex_bin, repo), "--ask-for-approval", "never"] +SAFE_CODEX_CONFIG_KEYS = { + "hide_agent_reasoning", + "model_auto_compact_token_limit", + "model_auto_compact_token_limit_scope", + "model_context_window", + "model_reasoning_effort", + "model_reasoning_summary", + "model_verbosity", + "personality", + "plan_mode_reasoning_effort", + "service_tier", + "show_raw_agent_reasoning", + "tool_output_token_limit", +} + + +def codex_config_overrides(args: argparse.Namespace) -> list[str]: + raw = list(getattr(args, "codex_config", None) or []) + if not raw: + raw = os.environ.get("AUTOREVIEW_CODEX_CONFIG", "").split(";") + overrides: list[str] = [] + for item in raw: + item = item.strip() + if not item: + continue + key, sep, value = item.partition("=") + key = key.strip() + if not sep or not value.strip() or not re.fullmatch(r"[A-Za-z0-9_][A-Za-z0-9_.-]*", key): + raise SystemExit(f"invalid Codex config override (expected key=value): {item}") + if key not in SAFE_CODEX_CONFIG_KEYS: + raise SystemExit( + f"unsafe Codex config override refused: {key}; " + "only model and response tuning keys are allowed" + ) + overrides.append(item) + return overrides + + +def codex_config_keys(args: argparse.Namespace) -> list[str]: + return [override.partition("=")[0].strip() for override in codex_config_overrides(args)] + + +def codex_speed_override(args: argparse.Namespace) -> str | None: + speed = getattr(args, "codex_speed", None) or os.environ.get("AUTOREVIEW_CODEX_SPEED", "").strip() or None + if speed is None: + return None + speed = speed.strip().lower() + if speed not in {"fast", "flex", "default"}: + raise SystemExit(f"invalid Codex speed: {speed} (valid: fast, flex, default)") + return f'service_tier="{speed}"' + + +def codex_error_messages(result: subprocess.CompletedProcess[str]) -> list[str]: + messages: list[str] = [] + for stream, accept_plain_text in ( + (result.stderr, True), + (result.stdout, False), + ): + for raw_line in stream.splitlines(): + line = raw_line.strip() + if not line: + continue + if not line.startswith("{"): + if accept_plain_text: + messages.append(line) + continue + try: + event = json.loads(line) + except json.JSONDecodeError: + continue + if not isinstance(event, dict) or event.get("type") not in { + "error", + "turn.failed", + }: + continue + message = event.get("message") + if isinstance(message, str): + messages.append(message) + error = event.get("error") + if isinstance(error, str): + messages.append(error) + elif isinstance(error, dict) and isinstance(error.get("message"), str): + messages.append(error["message"]) + return messages + + +def codex_model_access_failure(result: subprocess.CompletedProcess[str], model: str) -> bool: + for message in codex_error_messages(result): + lowered = message.lower() + if model.lower() not in lowered: + continue + if any( + marker in lowered + for marker in ( + "does not exist or you do not have access", + "do not have access to", + "don't have access to", + "does not appear in the list of models available to your account", + "not supported when using codex", + ) + ): + return True + return False + + +def codex_command( + args: argparse.Namespace, + source_repo: Path, + review_root: Path, + runtime_root: Path, + schema_path: Path, + output_path: Path, + model: str | None, + *, + force_file_auth: bool = False, +) -> list[str]: + cmd = [resolve_command(args.codex_bin, source_repo), "--ask-for-approval", "never"] if args.web_search: cmd.append("--search") - if args.model: - cmd.extend(["--model", args.model]) + if model: + cmd.extend(["--model", model]) + # User overrides go before the isolation flags so isolation stays authoritative on conflicts. + for override in codex_config_overrides(args): + cmd.extend(["-c", override]) + # Dedicated settings win over the generic config escape hatch. if args.thinking: cmd.extend(["-c", f'model_reasoning_effort="{args.thinking}"']) - cmd.extend(codex_config_isolation_flags(repo)) + # After --codex-config so an explicit speed wins over a service_tier value in the raw overrides. + speed_override = codex_speed_override(args) + if speed_override is not None: + cmd.extend(["-c", speed_override]) + cmd.extend(codex_config_isolation_flags(review_root, runtime_root)) + cmd.extend(codex_auth_config_flags(source_repo, force_file=force_file_auth)) cmd.append("exec") if args.stream_engine_output: cmd.append("--json") @@ -787,9 +9252,7 @@ def run_codex(args: argparse.Namespace, repo: Path, prompt: str) -> str: *codex_exec_isolation_flags(), "--ephemeral", "-C", - str(repo), - "-s", - "read-only", + str(review_root), "--output-schema", str(schema_path), "--output-last-message", @@ -797,22 +9260,126 @@ def run_codex(args: argparse.Namespace, repo: Path, prompt: str) -> str: "-", ] ) - result = run_with_heartbeat( - cmd, - repo, - input_text=prompt, - label="codex", - stream_output=args.stream_engine_output, - stream_display=CodexStreamDisplay() if args.stream_engine_output else None, - ) + return cmd + + +def run_codex(args: argparse.Namespace, repo: Path, prompt: str) -> str: + if not args.tools: + raise SystemExit("--no-tools is not supported by the Codex engine; use --engine claude --no-tools for a no-tools run") + temp_root = safe_temp_root(repo) + schema_path = write_json_temp(SCHEMA, temp_root) + with tempfile.NamedTemporaryFile( + "w", + suffix=".json", + delete=False, + dir=temp_root, + ) as output_file: + output_path = Path(output_file.name) + models = [args.model] + fallback_model = getattr(args, "fallback_model", None) + if fallback_model and fallback_model != args.model: + models.append(fallback_model) + primary_failure: subprocess.CompletedProcess[str] | None = None try: - output = output_path.read_text() + # The validated bundle is the sole repository input. The empty + # workspace keeps ignored credentials and linked-worktree metadata + # outside the model's readable filesystem boundary. + with tempfile.TemporaryDirectory( + prefix="autoreview-codex-workspace.", + dir=temp_root, + ) as workspace_dir, tempfile.TemporaryDirectory( + prefix="autoreview-codex-runtime.", + dir=temp_root, + ) as runtime_dir: + review_root = Path(workspace_dir) + runtime_root = Path(runtime_dir) + runtime_home = runtime_root / "home" + runtime_config = runtime_home / ".config" + runtime_data = runtime_home / ".local" / "share" + runtime_state = runtime_home / ".local" / "state" + runtime_cache = runtime_home / ".cache" + runtime_codex_home = runtime_root / "codex-home" + for path in ( + runtime_home, + runtime_config, + runtime_data, + runtime_state, + runtime_cache, + runtime_codex_home, + ): + path.mkdir(parents=True, exist_ok=True) + file_auth_linked = prepare_codex_runtime_auth(repo, runtime_codex_home) + source_codex_home = codex_source_home(repo) + active_codex_home = ( + runtime_codex_home + if file_auth_linked or source_codex_home is None + else source_codex_home + ) + for index, model in enumerate(models): + output_path.write_text("") + cmd = codex_command( + args, + repo, + review_root, + runtime_root, + schema_path, + output_path, + model, + force_file_auth=file_auth_linked, + ) + result = run_with_heartbeat( + cmd, + review_root, + input_text=prompt, + label="codex", + stream_output=args.stream_engine_output, + stream_display=CodexStreamDisplay() if args.stream_engine_output else None, + env=safe_engine_env( + repo, + [Path(cmd[0]).parent], + engine="codex", + extra={ + "HOME": str(runtime_home), + "USERPROFILE": str(runtime_home), + "XDG_CACHE_HOME": str(runtime_cache), + "XDG_CONFIG_HOME": str(runtime_config), + "XDG_DATA_HOME": str(runtime_data), + "XDG_STATE_HOME": str(runtime_state), + # Keyring namespaces are derived from canonical CODEX_HOME. + # Linked file auth uses the isolated home; keyring/auto must + # retain the source namespace until Codex supports an auth split. + "CODEX_HOME": str(active_codex_home), + }, + ), + resolve_root=repo, + ) + output = output_path.read_text() + if result.returncode == 0: + return output or result.stdout + if ( + index == 0 + and len(models) > 1 + and model + and codex_model_access_failure(result, model) + ): + primary_failure = result + print( + f"codex model {model} is unavailable for this account; retrying with {models[1]}", + file=sys.stderr, + ) + continue + detail = result.stderr or result.stdout + if primary_failure is not None: + primary_detail = primary_failure.stderr or primary_failure.stdout + raise SystemExit( + f"codex engine failed with primary model ({primary_failure.returncode})\n{primary_detail}\n" + f"codex fallback model failed ({result.returncode})\n{detail}" + ) + raise SystemExit(f"codex engine failed ({result.returncode})\n{detail}") finally: schema_path.unlink(missing_ok=True) output_path.unlink(missing_ok=True) - if result.returncode != 0: - raise SystemExit(f"codex engine failed ({result.returncode})\n{result.stderr or result.stdout}") - return output or result.stdout + raise AssertionError("unreachable") def run_claude(args: argparse.Namespace, repo: Path, prompt: str) -> str: @@ -828,7 +9395,8 @@ def run_claude(args: argparse.Namespace, repo: Path, prompt: str) -> str: json.dumps(SCHEMA), ] if args.tools: - cmd.extend(["--allowedTools", claude_allowed_tools(args)]) + allowed_tools = claude_allowed_tools(args) + cmd.extend(["--tools", claude_tool_inventory(args), "--allowedTools", allowed_tools]) else: cmd.extend(["--tools", ""]) if args.stream_engine_output: @@ -839,85 +9407,41 @@ def run_claude(args: argparse.Namespace, repo: Path, prompt: str) -> str: cmd.extend(["--fallback-model", args.fallback_model]) if args.thinking: cmd.extend(["--effort", args.thinking]) - result = run_with_heartbeat( - cmd, - repo, - input_text=prompt, - label="claude", - stream_output=args.stream_engine_output, - stream_display=ClaudeStreamDisplay() if args.stream_engine_output else None, - ) + with tempfile.TemporaryDirectory( + prefix="autoreview-claude-workspace.", + dir=safe_temp_root(repo), + ) as tempdir: + result = run_with_heartbeat( + cmd, + Path(tempdir), + input_text=prompt, + label="claude", + stream_output=args.stream_engine_output, + stream_display=ClaudeStreamDisplay() if args.stream_engine_output else None, + env=safe_engine_env( + repo, + [Path(cmd[0]).parent], + engine="claude", + ), + resolve_root=repo, + ) if result.returncode != 0: raise SystemExit(f"claude engine failed ({result.returncode})\n{result.stderr or result.stdout}") return result.stdout def run_droid(args: argparse.Namespace, repo: Path, prompt: str) -> str: - raise SystemExit("droid engine is disabled until a verified read-only isolation flag set is configured") - prompt_path = Path(tempfile.NamedTemporaryFile("w", suffix=".txt", delete=False).name) - prompt_path.write_text(prompt) - cmd = [ - resolve_command(args.droid_bin, repo), - "exec", - "--cwd", - str(repo), - "--output-format", - "stream-json" if args.stream_engine_output else "json", - "-f", - str(prompt_path), - ] - if args.model: - cmd.extend(["--model", args.model]) - if args.thinking: - cmd.extend(["-r", args.thinking]) - if not args.tools: - cmd.extend(["--disabled-tools", "*"]) - result = run_with_heartbeat(cmd, repo, label="droid", stream_output=args.stream_engine_output) - prompt_path.unlink(missing_ok=True) - if result.returncode != 0: - raise SystemExit(f"droid engine failed ({result.returncode})\n{result.stderr or result.stdout}") - return result.stdout + raise SystemExit( + "droid engine is unavailable: the current Droid CLI cannot disable project instructions and all tools; " + "use codex, claude, or pi" + ) def run_copilot(args: argparse.Namespace, repo: Path, prompt: str) -> str: - if args.thinking: - raise SystemExit("--thinking is not supported by the copilot engine") - if not args.tools: - raise SystemExit("--no-tools is not supported by the copilot engine; copilot requires a read-only file view tool to load the review bundle without exposing it in argv") - with tempfile.TemporaryDirectory(prefix="autoreview-copilot.") as tempdir: - prompt_path = Path(tempdir) / "prompt.txt" - prompt_path.write_text(prompt) - os.chmod(prompt_path, 0o600) - cmd = [ - resolve_command(args.copilot_bin, repo), - "-C", - tempdir, - "-p", - "Read ./prompt.txt and follow it exactly. Return only the requested JSON object.", - "--output-format", - "json", - "--stream", - "on" if args.stream_engine_output else "off", - "--no-ask-user", - "--disable-builtin-mcps", - ] - if args.model: - cmd.extend(["--model", args.model]) - cmd.extend( - [ - "--available-tools=read_agent,rg,view,web_fetch", - "--allow-tool=read_agent", - "--allow-tool=rg", - "--allow-tool=view", - "--allow-tool=web_fetch", - ] - ) - if args.web_search: - cmd.append("--allow-all-urls") - result = run_with_heartbeat(cmd, Path(tempdir), label="copilot", stream_output=args.stream_engine_output) - if result.returncode != 0: - raise SystemExit(f"copilot engine failed ({result.returncode})\n{result.stderr or result.stdout}") - return result.stdout + raise SystemExit( + "copilot engine is unavailable: its file tools cannot be confined to the reviewed bundle " + "without exposing ignored repository secrets; use codex, claude, or pi" + ) def build_opencode_cmd(args: argparse.Namespace, repo: Path) -> list[str]: @@ -938,21 +9462,180 @@ def build_opencode_cmd(args: argparse.Namespace, repo: Path) -> list[str]: def run_opencode(args: argparse.Namespace, repo: Path, prompt: str) -> str: - if not args.tools: - raise SystemExit("--no-tools is not supported by the opencode engine") - cmd = build_opencode_cmd(args, repo) - with tempfile.TemporaryDirectory(prefix="autoreview-opencode-run.") as tempdir: - result = run_with_heartbeat( - cmd, - Path(tempdir), - input_text=prompt, - label="opencode", - stream_output=args.stream_engine_output, - env=opencode_review_env(args.web_search), - ) + raise SystemExit( + "opencode engine is unavailable: the current CLI contract does not prove " + "project-config isolation and its generic fetch tool cannot be restricted " + "away from private or metadata endpoints; use codex, claude, or pi" + ) + + +def cursor_local_mcp_paths(repo: Path) -> list[Path]: + candidates = [ + repo / ".cursor" / "mcp.json", + repo / ".mcp.json", + repo / "mcp.json", + ] + return [path for path in candidates if path.exists()] + + +def cursor_home_candidates() -> set[Path]: + home_candidates = [Path.home()] + for name in ("HOME", "USERPROFILE"): + if value := os.environ.get(name): + home_candidates.append(Path(value)) + if drive := os.environ.get("HOMEDRIVE"): + if home_path := os.environ.get("HOMEPATH"): + home_candidates.append(Path(f"{drive}{home_path}")) + return set(home_candidates) + + +def cursor_global_mcp_paths() -> list[Path]: + paths = {home / ".cursor" / "mcp.json" for home in cursor_home_candidates()} + return sorted((path for path in paths if path.exists()), key=str) + + +def json_file_declares_hooks(path: Path) -> bool: + try: + parsed = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return True + if not isinstance(parsed, dict): + return False + if parsed.get("hooks"): + return True + enabled_plugins = parsed.get("enabledPlugins") + return isinstance(enabled_plugins, dict) and any(bool(enabled) for enabled in enabled_plugins.values()) + + +def cursor_global_hook_paths() -> list[Path]: + paths: set[Path] = set() + for home in cursor_home_candidates(): + cursor_hooks = home / ".cursor" / "hooks.json" + if cursor_hooks.exists(): + paths.add(cursor_hooks) + for name in ("settings.json", "settings.local.json"): + claude_settings = home / ".claude" / name + if claude_settings.exists() and json_file_declares_hooks(claude_settings): + paths.add(claude_settings) + return sorted(paths, key=str) + + +def cursor_local_hook_paths(repo: Path) -> list[Path]: + candidates = [ + repo / ".cursor" / "hooks.json", + repo / ".claude" / "settings.json", + repo / ".claude" / "settings.local.json", + ] + return [path for path in candidates if path.exists()] + + +def cursor_local_permission_paths(repo: Path) -> list[Path]: + path = repo / ".cursor" / "cli.json" + return [path] if path.exists() else [] + + +def format_repo_paths(repo: Path, paths: list[Path]) -> str: + return "; ".join(str(path.relative_to(repo)) for path in paths) + + +def cursor_result_event(text: str) -> dict[str, Any] | None: + stripped = text.strip() + if not stripped: + return None + try: + parsed = json.loads(stripped) + except json.JSONDecodeError: + parsed = None + if isinstance(parsed, dict) and parsed.get("type") == "result": + return parsed + for line in reversed(stripped.splitlines()): + line = line.strip() + if not line: + continue + try: + event = json.loads(line) + except json.JSONDecodeError: + continue + if isinstance(event, dict) and event.get("type") == "result": + return event + return None + + +def print_cursor_metadata(text: str) -> None: + event = cursor_result_event(text) + if not event: + return + parts: list[str] = [] + for key in ("session_id", "request_id"): + value = event.get(key) + if isinstance(value, str) and value: + parts.append(f"{key}={value}") + if parts: + print("cursor metadata: " + " ".join(parts), file=sys.stderr) + + +def cursor_help_text(cursor_bin: str, repo: Path, engine_env: dict[str, str]) -> str: + with tempfile.TemporaryDirectory(prefix="autoreview-cursor-probe.") as tempdir: + result = run([cursor_bin, "--help"], Path(tempdir), check=False, env=engine_env) if result.returncode != 0: - raise SystemExit(f"opencode engine failed ({result.returncode})\n{result.stderr or result.stdout}") - return result.stdout + output = (result.stderr or result.stdout).strip() + raise SystemExit(f"cursor engine could not read CLI help from {cursor_bin}: {output[:1000]}") + return result.stdout + result.stderr + + +def ensure_cursor_supported( + args: argparse.Namespace, + repo: Path, + cursor_bin: str, + engine_env: dict[str, str], +) -> None: + help_text = cursor_help_text(cursor_bin, repo, engine_env) + missing: list[str] = [] + if "--print" not in help_text and "-p" not in help_text: + missing.append("--print/-p") + if "--output-format" not in help_text: + missing.append("--output-format") + if "--mode" not in help_text: + missing.append("--mode") + if "--sandbox" not in help_text: + missing.append("--sandbox") + if args.model and "--model" not in help_text and "-m" not in help_text: + missing.append("--model/-m") + if missing: + raise SystemExit( + "cursor engine requires CLI support for " + + ", ".join(missing) + + ". Current Cursor CLI help does not advertise the required option(s)." + ) + + +def build_cursor_cmd( + args: argparse.Namespace, + repo: Path, + cursor_bin: str, + engine_env: dict[str, str], +) -> list[str]: + ensure_cursor_supported(args, repo, cursor_bin, engine_env) + cmd = [ + cursor_bin, + "--print", + "--output-format", + "stream-json" if args.stream_engine_output else "json", + "--mode", + "ask", + "--sandbox", + "enabled", + ] + if args.model: + cmd.extend(["--model", args.model]) + return cmd + + +def run_cursor(args: argparse.Namespace, repo: Path, prompt: str) -> str: + raise SystemExit( + "cursor engine is unavailable: Cursor read permissions can target absolute host paths " + "and the CLI does not expose a proven repository-only filesystem sandbox" + ) def run_pi(args: argparse.Namespace, repo: Path, prompt: str) -> str: @@ -966,17 +9649,25 @@ def run_pi(args: argparse.Namespace, repo: Path, prompt: str) -> str: cmd.extend(["--model", args.model]) if args.thinking: cmd.extend(["--thinking", args.thinking]) - if args.tools: - cmd.extend(["--tools", "read,grep,find,ls"]) - else: - cmd.append("--no-tools") - with tempfile.TemporaryDirectory(prefix="autoreview-pi-run.") as tempdir: + # Pi's built-in read tools accept absolute paths and have no repository + # confinement, so an untrusted review prompt must never receive them. + cmd.append("--no-tools") + with tempfile.TemporaryDirectory( + prefix="autoreview-pi-run.", + dir=safe_temp_root(repo), + ) as tempdir: result = run_with_heartbeat( cmd, Path(tempdir), input_text=prompt, label="pi", stream_output=args.stream_engine_output, + resolve_root=repo, + env=safe_engine_env( + repo, + [Path(cmd[0]).parent], + engine="pi", + ), ) if result.returncode != 0: raise SystemExit(f"pi engine failed ({result.returncode})\n{result.stderr or result.stdout}") @@ -991,7 +9682,7 @@ class CodexStreamDisplay: def __call__(self, name: str, line: str) -> str | None: if name != "stdout": - return line + return stream_display_escape(line) try: event = json.loads(line) except json.JSONDecodeError: @@ -1025,7 +9716,7 @@ class CodexStreamDisplay: def visible(self, text: str) -> str: self.last_visible = time.monotonic() - return text + return stream_display_escape(text) class ClaudeStreamDisplay: @@ -1037,7 +9728,7 @@ class ClaudeStreamDisplay: def __call__(self, name: str, line: str) -> str | None: if name != "stdout": - return line + return stream_display_escape(line) try: event = json.loads(line) except json.JSONDecodeError: @@ -1058,32 +9749,92 @@ class ClaudeStreamDisplay: return self.hidden_activity() chunks: list[str] = [] for item in message.get("content", []): - if not isinstance(item, dict): - continue - if item.get("type") == "text" and isinstance(item.get("text"), str): + if not isinstance(item, dict): + continue + if item.get("type") == "text" and isinstance(item.get("text"), str): + chunks.append(item["text"].rstrip()) + if chunks: + return self.visible(self.flush_hidden() + "\n".join(chunks) + "\n") + return self.hidden_activity() + + def result_summary(self, event: dict[str, Any]) -> str: + usage = event.get("usage") + fields: list[str] = [] + if isinstance(usage, dict): + for key in ( + "input_tokens", + "cache_read_input_tokens", + "cache_creation_input_tokens", + "output_tokens", + ): + value = usage.get(key) + if isinstance(value, int): + fields.append(f"{key}={value}") + cost = event.get("total_cost_usd") + if isinstance(cost, (int, float)) and not isinstance(cost, bool): + fields.append(f"cost_usd={cost:.6f}") + return "claude usage: " + " ".join(fields) + "\n" if fields else "claude turn completed\n" + + def hidden_activity(self) -> str | None: + self.hidden_events += 1 + if time.monotonic() - self.last_visible < self.activity_seconds: + return None + return self.visible(self.flush_hidden()) + + def flush_hidden(self) -> str: + if not self.hidden_events: + return "" + count = self.hidden_events + self.hidden_events = 0 + return f"claude activity: {count} hidden tool/status events\n" + + def visible(self, text: str) -> str: + self.last_visible = time.monotonic() + return stream_display_escape(text) + + +class CursorStreamDisplay: + def __init__(self, *, activity_seconds: int = 20) -> None: + self.activity_seconds = activity_seconds + self.hidden_events = 0 + self.last_visible = time.monotonic() + self.started = False + + def __call__(self, name: str, line: str) -> str | None: + if name != "stdout": + return line + try: + event = json.loads(line) + except json.JSONDecodeError: + return self.visible(line) + if not isinstance(event, dict): + return self.hidden_activity() + event_type = event.get("type") + if event_type == "system" and not self.started: + self.started = True + model = event.get("model") + suffix = f" model={model}" if isinstance(model, str) and model else "" + return self.visible(f"cursor turn started{suffix}\n") + if event_type == "assistant": + return self.assistant_message(event) + if event_type == "result": + request_id = event.get("request_id") + suffix = f" request_id={request_id}" if isinstance(request_id, str) and request_id else "" + return self.visible(self.flush_hidden() + format_cursor_usage(event.get("usage")) + suffix + "\n") + return self.hidden_activity() + + def assistant_message(self, event: dict[str, Any]) -> str | None: + message = event.get("message") + if not isinstance(message, dict): + return self.hidden_activity() + chunks: list[str] = [] + for item in message.get("content", []): + if isinstance(item, dict) and item.get("type") == "text" and isinstance(item.get("text"), str): chunks.append(item["text"].rstrip()) if chunks: return self.visible(self.flush_hidden() + "\n".join(chunks) + "\n") return self.hidden_activity() - def result_summary(self, event: dict[str, Any]) -> str: - usage = event.get("usage") - fields: list[str] = [] - if isinstance(usage, dict): - for key in ( - "input_tokens", - "cache_read_input_tokens", - "cache_creation_input_tokens", - "output_tokens", - ): - value = usage.get(key) - if isinstance(value, int): - fields.append(f"{key}={value}") - cost = event.get("total_cost_usd") - if isinstance(cost, (int, float)) and not isinstance(cost, bool): - fields.append(f"cost_usd={cost:.6f}") - return "claude usage: " + " ".join(fields) + "\n" if fields else "claude turn completed\n" - def hidden_activity(self) -> str | None: self.hidden_events += 1 if time.monotonic() - self.last_visible < self.activity_seconds: @@ -1095,7 +9846,7 @@ class ClaudeStreamDisplay: return "" count = self.hidden_events self.hidden_events = 0 - return f"claude activity: {count} hidden tool/status events\n" + return f"cursor activity: {count} hidden tool/status events\n" def visible(self, text: str) -> str: self.last_visible = time.monotonic() @@ -1113,11 +9864,54 @@ def format_codex_usage(usage: dict[str, Any]) -> str: return "codex usage: " + " ".join(parts) if parts else "codex usage: unavailable" -def claude_allowed_tools(args: argparse.Namespace) -> str: +def format_cursor_usage(usage: Any) -> str: + if not isinstance(usage, dict): + return "cursor usage: unavailable" + fields = [ + "inputTokens", + "outputTokens", + "cacheReadTokens", + "cacheWriteTokens", + ] + parts = [f"{field}={usage[field]}" for field in fields if isinstance(usage.get(field), int)] + return "cursor usage: " + " ".join(parts) if parts else "cursor usage: unavailable" + + +def claude_tool_name(rule: str) -> str: + match = re.match(r"^([A-Za-z][A-Za-z0-9_-]*)(?:\(|$)", rule) + if not match: + raise SystemExit(f"invalid Claude tool rule: {rule}") + return match.group(1) + + +def claude_tool_rules(args: argparse.Namespace) -> list[str]: tools = [tool.strip() for tool in args.claude_allowed_tools.split(",") if tool.strip()] if not args.web_search: - tools = [tool for tool in tools if tool not in {"WebSearch", "WebFetch"}] - return ",".join(tools) + tools = [tool for tool in tools if claude_tool_name(tool) not in {"WebSearch", "WebFetch"}] + return tools + + +def claude_allowed_tools(args: argparse.Namespace) -> str: + return ",".join(claude_tool_rules(args)) + + +def claude_tool_inventory(args: argparse.Namespace) -> str: + safe_tools = {"WebFetch", "WebSearch"} + names: list[str] = [] + for rule in claude_tool_rules(args): + name = claude_tool_name(rule) + if name not in safe_tools: + raise SystemExit(f"Claude review tool is not read-only: {name}") + if name == "WebFetch" and not re.fullmatch( + r"WebFetch\(domain:[A-Za-z0-9.-]+\)", + rule, + ): + raise SystemExit( + "Claude WebFetch must be constrained to one explicit domain" + ) + if name not in names: + names.append(name) + return ",".join(names) def extract_json(text: str) -> dict[str, Any]: @@ -1127,17 +9921,21 @@ def extract_json(text: str) -> dict[str, Any]: try: parsed = json.loads(stripped) except json.JSONDecodeError as exc: - fenced_report = parse_json_candidate(stripped) - if isinstance(fenced_report, dict) and "findings" in fenced_report: - return fenced_report jsonl_report = extract_json_from_jsonl(stripped) if jsonl_report: return jsonl_report + fenced_report = parse_json_candidate(stripped) + if isinstance(fenced_report, dict) and "findings" in fenced_report: + return fenced_report raise SystemExit(f"review engine returned non-JSON output: {exc}\n{stripped[:2000]}") if isinstance(parsed, dict) and "findings" in parsed: return parsed if isinstance(parsed, dict) and isinstance(parsed.get("structured_output"), dict): return parsed["structured_output"] + if isinstance(parsed, dict) and isinstance(parsed.get("result"), dict): + result_object = parsed["result"] + if "findings" in result_object: + return result_object if isinstance(parsed, dict) and isinstance(parsed.get("result"), str): result_json = parse_json_candidate(parsed["result"]) if isinstance(result_json, dict) and "findings" in result_json: @@ -1160,7 +9958,9 @@ def _report_from_events(events: list[Any]) -> dict[str, Any] | None: (e.g. some `claude --output-format json` versions/configurations return [{type:system,init}, ..., {type:result,...}] rather than a bare object). """ + terminal_candidates: list[str | dict[str, Any]] = [] candidates: list[str | dict[str, Any]] = [] + assistant_candidates: list[str] = [] text_fragments: list[str] = [] for event in events: if not isinstance(event, dict): @@ -1172,20 +9972,37 @@ def _report_from_events(events: list[Any]) -> dict[str, Any] | None: data = event.get("data") if isinstance(data, dict) and isinstance(data.get("content"), str): candidates.append(data["content"]) + message = event.get("message") + if isinstance(message, dict): + for item in message.get("content", []): + if isinstance(item, dict) and item.get("type") == "text" and isinstance(item.get("text"), str): + assistant_candidates.append(item["text"]) if isinstance(event.get("result"), str): - candidates.append(event["result"]) + terminal_candidates.append(event["result"]) + if isinstance(event.get("result"), dict): + terminal_candidates.append(event["result"]) if isinstance(event.get("text"), str): candidates.append(event["text"]) if isinstance(event.get("finalText"), str): candidates.append(event["finalText"]) if isinstance(event.get("structured_output"), dict): - candidates.append(event["structured_output"]) + terminal_candidates.append(event["structured_output"]) if event.get("type") == "text": part = event.get("part") if isinstance(part, dict) and isinstance(part.get("text"), str): candidates.append(part["text"]) if text_fragments: candidates.append("".join(text_fragments)) + for candidate in reversed(terminal_candidates): + if isinstance(candidate, dict): + if "findings" in candidate: + return candidate + continue + parsed = parse_json_candidate(candidate) + if isinstance(parsed, dict) and "findings" in parsed: + return parsed + if terminal_candidates: + raise SystemExit("review engine result was not structured JSON:\n" + str(terminal_candidates[-1])[:2000]) for candidate in reversed(candidates): if isinstance(candidate, dict): if "findings" in candidate: @@ -1194,6 +10011,10 @@ def _report_from_events(events: list[Any]) -> dict[str, Any] | None: parsed = parse_json_candidate(candidate) if isinstance(parsed, dict) and "findings" in parsed: return parsed + for candidate in reversed(assistant_candidates): + parsed = parse_json_candidate(candidate) + if isinstance(parsed, dict) and "findings" in parsed: + return parsed return None @@ -1210,9 +10031,20 @@ def extract_json_from_jsonl(text: str) -> dict[str, Any] | None: return _report_from_events(events) -def write_executable(path: Path, text: str) -> None: - path.write_text(text) +def is_structured_output_failure(message: str) -> bool: + return message.startswith("review engine returned non-JSON output") or message.startswith( + "review engine result was not structured JSON" + ) + + +def write_executable(path: Path, text: str) -> Path: + path.write_text(text, encoding="utf-8") path.chmod(0o755) + if os.name != "nt": + return path + wrapper = path.with_name(f"{path.name}.cmd") + wrapper.write_text(f'@echo off\r\n"{sys.executable}" "{path}" %*\r\n', encoding="utf-8") + return wrapper def create_hostile_repo(repo: Path) -> None: @@ -1266,6 +10098,13 @@ def create_hostile_repo(repo: Path) -> None: (repo / ".mcp.json").write_text(json.dumps({"mcpServers": {"hostile": {"command": "./hostile-mcp"}}}) + "\n") (repo / "hostile-mcp").write_text("#!/bin/sh\necho mcp-ran > .hostile-mcp-ran\nexit 1\n") (repo / "hostile-mcp").chmod(0o755) + (repo / ".cursor" / "rules").mkdir(parents=True) + (repo / ".cursor" / "rules" / "hostile.mdc").write_text("HOSTILE_CURSOR_RULE_SENTINEL_DO_NOT_OBEY\n") + (repo / ".cursor" / "cli.json").write_text(json.dumps({"permissions": {"allow": ["Shell(*)"]}}) + "\n") + (repo / ".cursor" / "mcp.json").write_text( + json.dumps({"mcpServers": {"hostile": {"command": "./hostile-mcp"}}}) + "\n" + ) + (repo / "mcp.json").write_text(json.dumps({"mcpServers": {"hostile": {"command": "./hostile-mcp"}}}) + "\n") (repo / ".pi" / "extensions").mkdir(parents=True) (repo / ".pi" / "skills" / "hostile").mkdir(parents=True) (repo / ".pi" / "prompts").mkdir(parents=True) @@ -1302,6 +10141,8 @@ import sys record = os.environ["AUTOREVIEW_FAKE_RECORD"] args = sys.argv[1:] Path(record).write_text(json.dumps({"argv": args, "cwd": os.getcwd(), "stdin": sys.stdin.read()})) +if mutation := os.environ.get("AUTOREVIEW_FAKE_MUTATE"): + Path(mutation).write_text("mutated during review\n") try: output_path = args[args.index("--output-last-message") + 1] except ValueError: @@ -1329,10 +10170,15 @@ if "--version" in args or "-v" in args: print(os.environ.get("AUTOREVIEW_FAKE_CLAUDE_VERSION", "2.1.170 (Claude Code)")) raise SystemExit(0) if "--help" in args or "-h" in args: - print("--safe-mode\n--setting-sources\n--strict-mcp-config\n--disallowedTools\n--print\n--json-schema") + print("--safe-mode\n--setting-sources\n--strict-mcp-config\n--disallowedTools\n--tools\n--print\n--json-schema") raise SystemExit(0) record = os.environ["AUTOREVIEW_FAKE_RECORD"] -Path(record).write_text(json.dumps({"argv": args, "cwd": os.getcwd(), "stdin": sys.stdin.read()})) +Path(record).write_text(json.dumps({ + "argv": args, + "cwd": os.getcwd(), + "stdin": sys.stdin.read(), + "auto_memory_disabled": os.environ.get("CLAUDE_CODE_DISABLE_AUTO_MEMORY"), +})) report = { "findings": [], "overall_correctness": "patch is correct", @@ -1403,6 +10249,71 @@ print(json.dumps({"type": "text", "part": {"type": "text", "text": json.dumps(re ''' +def fake_cursor_script() -> str: + return r'''#!/usr/bin/env python3 +import json +import os +from pathlib import Path +import sys + +args = sys.argv[1:] +invocations = os.environ.get("AUTOREVIEW_FAKE_CURSOR_INVOCATIONS") +if invocations: + with open(invocations, "a", encoding="utf-8") as file: + file.write( + json.dumps( + { + "argv": args, + "cwd": os.getcwd(), + "environment": { + key: os.environ.get(key) + for key in ("CURSOR_CONFIG_DIR", "GIT_CONFIG_GLOBAL", "NODE_OPTIONS", "PYTHONPATH", "PATH") + }, + } + ) + + "\n" + ) +if "--help" in args or "-h" in args: + print(os.environ.get("AUTOREVIEW_FAKE_CURSOR_HELP", "--print\n--output-format\n--model\n--mode\n--sandbox")) + raise SystemExit(0) +record = os.environ["AUTOREVIEW_FAKE_RECORD"] +stdin = sys.stdin.read() +cursor_config = Path(os.environ["CURSOR_CONFIG_DIR"], "cli-config.json").read_text() +Path(record).write_text( + json.dumps( + { + "argv": args, + "cwd": os.getcwd(), + "stdin": stdin, + "cursor_config": cursor_config, + "environment": { + key: os.environ.get(key) + for key in ("CURSOR_CONFIG_DIR", "GIT_CONFIG_GLOBAL", "NODE_OPTIONS", "PYTHONPATH", "PATH") + }, + } + ) +) +report = { + "findings": [], + "overall_correctness": "patch is correct", + "overall_explanation": "fake cursor clean", + "overall_confidence": 0.99, +} +result = { + "type": "result", + "result": json.dumps(report), + "session_id": "fake-session", + "request_id": "fake-request", + "usage": {"inputTokens": 1, "outputTokens": 2}, +} +if "stream-json" in args: + print(json.dumps({"type": "system", "model": "fake"})) + print(json.dumps(result)) +else: + print(json.dumps(result)) +''' + + def self_test_engine_isolation() -> int: with tempfile.TemporaryDirectory(prefix="autoreview-isolation-test.") as tempdir: root = Path(tempdir) @@ -1413,45 +10324,90 @@ def self_test_engine_isolation() -> int: claude_bin = root / "claude" pi_bin = root / "pi" opencode_bin = root / "opencode" + cursor_bin = root / "cursor-agent" record_path = root / "record.json" pi_invocations_path = root / "pi-invocations.jsonl" - write_executable(codex_bin, fake_codex_script()) - write_executable(claude_bin, fake_claude_script()) - write_executable(pi_bin, fake_pi_script()) - write_executable(opencode_bin, fake_opencode_script()) + hostile_ps_path = root / "hostile-ps-ran" + cursor_invocations_path = root / "cursor-invocations.jsonl" + codex_bin = write_executable(codex_bin, fake_codex_script()) + claude_bin = write_executable(claude_bin, fake_claude_script()) + pi_bin = write_executable(pi_bin, fake_pi_script()) + opencode_bin = write_executable(opencode_bin, fake_opencode_script()) + write_executable(repo / "ps", f"#!/usr/bin/env python3\nfrom pathlib import Path\nPath({str(hostile_ps_path)!r}).write_text('ran')\n") + cursor_bin = write_executable(cursor_bin, fake_cursor_script()) args = argparse.Namespace( codex_bin=str(codex_bin), claude_bin=str(claude_bin), pi_bin=str(pi_bin), opencode_bin=str(opencode_bin), + cursor_bin=str(cursor_bin), tools=True, web_search=True, model=None, thinking=None, stream_engine_output=False, - claude_allowed_tools="Read,Grep,Glob,WebSearch,WebFetch", + claude_allowed_tools="WebSearch,WebFetch(domain:docs.example.com)", + cursor_allow_workspace_instructions=False, ) os.environ["AUTOREVIEW_FAKE_RECORD"] = str(record_path) os.environ["AUTOREVIEW_FAKE_PI_INVOCATIONS"] = str(pi_invocations_path) + os.environ["AUTOREVIEW_FAKE_CURSOR_INVOCATIONS"] = str(cursor_invocations_path) + codex_home = root / "codex-home" + codex_home.mkdir() + (codex_home / "config.toml").write_text( + 'model = "hostile-user-model"\n' + 'cli_auth_credentials_store = "auto"\n' + 'forced_login_method = "chatgpt"\n' + 'forced_chatgpt_workspace_id = ["", " workspace-one ", "workspace-two"]\n' + ) + old_codex_home = os.environ.get("CODEX_HOME") + os.environ["CODEX_HOME"] = str(codex_home) + home_keys = ("HOME", "USERPROFILE", "HOMEDRIVE", "HOMEPATH") + old_home_env = {key: os.environ.get(key) for key in home_keys} + test_home = root / "home" + test_home.mkdir() + os.environ["HOME"] = str(test_home) + os.environ["USERPROFILE"] = str(test_home) + os.environ.pop("HOMEDRIVE", None) + os.environ.pop("HOMEPATH", None) + old_path = os.environ.get("PATH", "") + os.environ["PATH"] = f"{repo}{os.pathsep}{old_path}" try: + sample_process_metrics(repo, os.getpid()) + if hostile_ps_path.exists(): + raise SystemExit("heartbeat metrics isolation self-test failed: repo-local ps executed") + run_codex(args, repo, "review hostile patch") codex_record = json.loads(record_path.read_text()) codex_argv = codex_record["argv"] - expected_project_override = f"projects.{toml_quoted_key_segment(str(repo.resolve()))}.trust_level=\"untrusted\"" for required in [ + "--ignore-user-config", "--ignore-rules", "project_doc_max_bytes=0", - expected_project_override, + 'cli_auth_credentials_store="auto"', + 'forced_login_method="chatgpt"', + 'forced_chatgpt_workspace_id=["workspace-one", "workspace-two"]', + 'default_permissions="autoreview"', + 'permissions.autoreview.filesystem={":minimal"="read",":workspace_roots"="read"}', "--ephemeral", - str(repo), - "read-only", ]: if required not in codex_argv: raise SystemExit(f"codex isolation self-test failed: missing {required}") - if Path(codex_record["cwd"]).resolve() != repo.resolve(): - raise SystemExit("codex isolation self-test failed: wrong cwd") + for forbidden in ["hostile-user-model", "read-only"]: + if forbidden in codex_argv: + raise SystemExit(f"codex isolation self-test failed: leaked {forbidden}") + codex_cwd = Path(codex_record["cwd"]).resolve() + if codex_cwd == repo.resolve() or is_within(codex_cwd, repo.resolve()): + raise SystemExit("codex isolation self-test failed: review ran inside hostile repo") + if str(repo) in codex_argv: + raise SystemExit("codex isolation self-test failed: hostile repo granted to tools") + expected_project_override = ( + f"projects.{toml_quoted_key_segment(str(codex_cwd))}.trust_level=\"untrusted\"" + ) + if expected_project_override not in codex_argv: + raise SystemExit("codex isolation self-test failed: isolated project override missing") run_claude(args, repo, "review hostile patch") claude_record = json.loads(record_path.read_text()) @@ -1459,15 +10415,31 @@ def self_test_engine_isolation() -> int: for required in claude_review_isolation_flags(): if required not in claude_argv: raise SystemExit(f"claude isolation self-test failed: missing {required}") - if Path(claude_record["cwd"]).resolve() != repo.resolve(): - raise SystemExit("claude isolation self-test failed: wrong cwd") + allowed_tools = claude_allowed_tools(args) + for required in ["--tools", allowed_tools, "--allowedTools"]: + if required not in claude_argv: + raise SystemExit(f"claude isolation self-test failed: missing {required}") + tools_index = claude_argv.index("--tools") + if claude_argv[tools_index + 1] != "WebSearch,WebFetch": + raise SystemExit("claude isolation self-test failed: wrong tool inventory") + allowed_index = claude_argv.index("--allowedTools") + if claude_argv[allowed_index + 1] != allowed_tools: + raise SystemExit("claude isolation self-test failed: scoped allowed tools lost") + claude_cwd = Path(claude_record["cwd"]).resolve() + if claude_cwd == repo.resolve() or is_within(claude_cwd, repo.resolve()): + raise SystemExit("claude isolation self-test failed: review ran inside hostile repo") + if claude_record["auto_memory_disabled"] != "1": + raise SystemExit("claude isolation self-test failed: auto-memory not disabled") run_pi(args, repo, f"review hostile patch\nRepository: {repo}") pi_record = json.loads(record_path.read_text()) pi_argv = pi_record["argv"] - for required in ["--print", *pi_review_isolation_flags(), "--tools", "read,grep,find,ls"]: + for required in ["--print", *pi_review_isolation_flags(), "--no-tools"]: if required not in pi_argv: raise SystemExit(f"pi isolation self-test failed: missing {required}") + for forbidden in ["--tools", "read,grep,find,ls"]: + if forbidden in pi_argv: + raise SystemExit(f"pi isolation self-test failed: unsafe {forbidden}") if Path(pi_record["cwd"]).resolve() == repo.resolve(): raise SystemExit("pi isolation self-test failed: review ran inside hostile repo") if str(repo) not in pi_record["stdin"]: @@ -1483,31 +10455,31 @@ def self_test_engine_isolation() -> int: if Path(entry["cwd"]).resolve() == repo.resolve(): raise SystemExit("pi isolation self-test failed: probe ran inside hostile repo") - run_opencode(args, repo, "review hostile patch") - opencode_record = json.loads(record_path.read_text()) - opencode_argv = opencode_record["argv"] - for required in ["run", "--dir", str(repo), "--pure", "--format", "json"]: - if required not in opencode_argv: - raise SystemExit(f"opencode isolation self-test failed: missing {required}") - if "--dangerously-skip-permissions" in opencode_argv: - raise SystemExit("opencode isolation self-test failed: skip-permissions present") - if Path(opencode_record["cwd"]).resolve() == repo.resolve(): - raise SystemExit("opencode isolation self-test failed: review ran inside hostile repo") - if opencode_record["stdin"] != "review hostile patch": - raise SystemExit("opencode isolation self-test failed: prompt not delivered over stdin") - if "review hostile patch" in opencode_argv: - raise SystemExit("opencode isolation self-test failed: prompt leaked into argv") - opencode_env = opencode_record["env"] - if opencode_env.get("OPENCODE_DISABLE_PROJECT_CONFIG") != "1": - raise SystemExit("opencode isolation self-test failed: project config env missing") - if opencode_env.get("OPENCODE_DISABLE_AUTOUPDATE") != "1": - raise SystemExit("opencode isolation self-test failed: autoupdate env missing") - config = json.loads(opencode_env["OPENCODE_CONFIG_CONTENT"]) - if config.get("instructions") != [] or config.get("plugin") != [] or config.get("command") != {}: - raise SystemExit("opencode isolation self-test failed: project-controlled extensions not cleared") - for disabled_tool in ("bash", "edit", "skill", "task", "todowrite", "write"): - if config.get("tools", {}).get(disabled_tool) is not False: - raise SystemExit(f"opencode isolation self-test failed: {disabled_tool} tool not disabled") + if record_path.exists(): + record_path.unlink() + try: + run_opencode(args, repo, "review hostile patch") + except SystemExit as exc: + if "opencode engine is unavailable" not in str(exc): + raise + else: + raise SystemExit("opencode isolation self-test failed: unsafe engine was allowed") + if record_path.exists(): + raise SystemExit("opencode isolation self-test failed: disabled engine was invoked") + if hostile_ps_path.exists(): + raise SystemExit("heartbeat metrics isolation self-test failed: repo-local ps executed") + + if record_path.exists(): + record_path.unlink() + try: + run_cursor(args, repo, "review hostile patch") + except SystemExit as exc: + if "Cursor read permissions" not in str(exc): + raise + else: + raise SystemExit("cursor isolation self-test failed: unconfined reads were allowed") + if record_path.exists(): + raise SystemExit("cursor isolation self-test failed: disabled cursor was invoked") os.environ["AUTOREVIEW_FAKE_CLAUDE_VERSION"] = "2.1.168 (Claude Code)" try: @@ -1518,6 +10490,17 @@ def self_test_engine_isolation() -> int: else: raise SystemExit("claude version floor self-test failed") + os.environ["AUTOREVIEW_FAKE_CLAUDE_VERSION"] = "2.1.169 (Claude Code)" + args.model = "claude-fable-5" + try: + ensure_claude_isolation_supported(args, repo) + except SystemExit as exc: + if ">= 2.1.170" not in str(exc): + raise + else: + raise SystemExit("claude fable version floor self-test failed") + args.model = None + os.environ["AUTOREVIEW_FAKE_CLAUDE_VERSION"] = "Claude Code unknown" try: ensure_claude_isolation_supported(args, repo) @@ -1551,6 +10534,18 @@ def self_test_engine_isolation() -> int: os.environ.pop("AUTOREVIEW_FAKE_PI_VERSION", None) os.environ.pop("AUTOREVIEW_FAKE_PI_HELP", None) os.environ.pop("AUTOREVIEW_FAKE_PI_INVOCATIONS", None) + os.environ.pop("AUTOREVIEW_FAKE_CURSOR_HELP", None) + os.environ.pop("AUTOREVIEW_FAKE_CURSOR_INVOCATIONS", None) + os.environ["PATH"] = old_path + if old_codex_home is None: + os.environ.pop("CODEX_HOME", None) + else: + os.environ["CODEX_HOME"] = old_codex_home + for key, value in old_home_env.items(): + if value is None: + os.environ.pop(key, None) + else: + os.environ[key] = value if parse_cli_version("2.1.169 (Claude Code)") != CLAUDE_SAFE_MODE_MIN_VERSION: raise SystemExit("claude version parsing self-test failed") @@ -1558,6 +10553,23 @@ def self_test_engine_isolation() -> int: raise SystemExit("claude version parsing prefix self-test failed") if parse_cli_version("pi 0.79.0") != PI_TRUST_ISOLATION_MIN_VERSION: raise SystemExit("pi version parsing self-test failed") + fallback_config = parse_codex_auth_config_fallback( + 'cli_auth_credentials_store = "ephemeral"\n' + 'forced_login_method = "api"\n' + 'forced_chatgpt_workspace_id = [\n' + ' "workspace-one",\n' + ' "workspace-two", # trailing comments are valid TOML\n' + ']\n' + 'model = "must-not-be-forwarded"\n' + '[projects."/tmp/hostile"]\n' + 'trust_level = "trusted"\n' + ) + if fallback_config != { + "cli_auth_credentials_store": "ephemeral", + "forced_login_method": "api", + "forced_chatgpt_workspace_id": ["workspace-one", "workspace-two"], + }: + raise SystemExit("codex auth config fallback parser self-test failed") if "none" not in THINKING_LEVELS_BY_ENGINE["codex"]: raise SystemExit("codex thinking self-test failed") print("autoreview engine isolation self-test: ok") @@ -1603,6 +10615,58 @@ def self_test_json_array_parser() -> int: return 0 +def self_test_cursor_jsonl_parser() -> int: + report = { + "findings": [], + "overall_correctness": "patch is correct", + "overall_explanation": "cursor parser self-test", + "overall_confidence": 0.99, + } + result_object = { + "type": "result", + "result": report, + "session_id": "session", + "request_id": "request", + } + if extract_json(json.dumps(result_object)) != report: + raise SystemExit("cursor parser self-test failed for result object") + + result_text = { + "type": "result", + "result": "```json\n" + json.dumps(report) + "\n```", + "session_id": "session", + "request_id": "request", + } + if extract_json(json.dumps(result_text)) != report: + raise SystemExit("cursor parser self-test failed for result text") + + jsonl = "\n".join( + json.dumps(event) + for event in [ + {"type": "system", "model": "fake"}, + {"type": "assistant", "message": {"content": [{"type": "text", "text": json.dumps(report)}]}}, + {"type": "result", "result": "not structured json"}, + ] + ) + try: + extract_json(jsonl) + except SystemExit as exc: + if "review engine result was not structured JSON" not in str(exc): + raise + else: + raise SystemExit("cursor parser self-test failed: assistant draft masked bad result") + + try: + extract_json("analysis before " + json.dumps(report) + " after") + except SystemExit: + pass + else: + raise SystemExit("cursor parser self-test failed: embedded JSON was accepted") + + print("autoreview cursor jsonl parser self-test: ok") + return 0 + + def parse_json_candidate(text: str) -> Any | None: stripped = text.strip() if stripped.startswith("```"): @@ -1634,23 +10698,16 @@ def _assert_opencode_permission(web_search: bool) -> None: raise SystemExit(f"opencode isolation self-test failed: {disabled_tool} tool not disabled") if permission.get("*") != "deny": raise SystemExit("opencode isolation self-test failed: default deny missing") - read_permission = permission.get("read") - if not isinstance(read_permission, dict): - raise SystemExit("opencode isolation self-test failed: read rules missing") - expected_read_rules = { - "*": "allow", - "*.env": "ask", - "*.env.*": "ask", - "*.env.example": "allow", - } - for pattern, action in expected_read_rules.items(): - if read_permission.get(pattern) != action: - raise SystemExit(f"opencode isolation self-test failed: read {pattern} must be {action}") + for filesystem_tool in ("read", "grep", "glob"): + if permission.get(filesystem_tool) != "deny": + raise SystemExit( + f"opencode isolation self-test failed: {filesystem_tool} must be denied" + ) expected_web = "allow" if web_search else "deny" if permission.get("websearch") != expected_web: raise SystemExit(f"opencode isolation self-test failed: websearch must be {expected_web} when web_search={web_search}") - if permission.get("webfetch") != expected_web: - raise SystemExit(f"opencode isolation self-test failed: webfetch must be {expected_web} when web_search={web_search}") + if permission.get("webfetch") != "deny": + raise SystemExit("opencode isolation self-test failed: webfetch must stay denied") def self_test_opencode_isolation() -> None: @@ -1658,7 +10715,7 @@ def self_test_opencode_isolation() -> None: _assert_opencode_permission(False) cmd = build_opencode_cmd( argparse.Namespace( - opencode_bin="opencode", + opencode_bin=sys.executable, stream_engine_output=False, model=None, thinking=None, @@ -1717,6 +10774,8 @@ def self_test_opencode_real_project_isolation(args: argparse.Namespace) -> None: [opencode_bin, "debug", "config", "--pure"], cwd=repo, text=True, + encoding=SUBPROCESS_TEXT_ENCODING, + errors=SUBPROCESS_TEXT_ERRORS, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=subprocess_env(opencode_review_env(False)), @@ -1771,7 +10830,38 @@ def self_test_opencode_jsonl_parser() -> None: print("autoreview opencode jsonl parser self-test: ok") -def validate_report(report: dict[str, Any], repo: Path, changed_paths: set[str], required: list[str]) -> None: +def self_test_heartbeat_metrics() -> None: + cases = { + "": 0.0, + "garbage": 0.0, + "12": 12.0, + "01:02": 62.0, + "01:02.5": 62.5, + "01:02:03": 3723.0, + "2-01:02:03": 176523.0, + } + for value, expected in cases.items(): + actual = parse_ps_time(value) + if actual != expected: + raise SystemExit(f"heartbeat metrics self-test failed: parse_ps_time({value!r})={actual!r}") + + previous = (10.0, 3.0, 1024, "S") + current = (70.0, 18.0, 1536, "R,S") + expected = " cpu=15.0s/60s(25%) rss=2M state=R,S" + actual = format_process_metrics(previous, current) + if actual != expected: + raise SystemExit(f"heartbeat metrics self-test failed: {actual!r}") + if format_process_metrics(previous, None) != "": + raise SystemExit("heartbeat metrics self-test failed: missing sample should not add output") + print("autoreview heartbeat metrics self-test: ok") + + +def _validate_report( + report: dict[str, Any], + repo: Path, + changed_paths: set[str], + required: list[str], +) -> None: allowed_top = {"findings", "overall_correctness", "overall_explanation", "overall_confidence"} extra_top = set(report) - allowed_top if extra_top: @@ -1819,12 +10909,27 @@ def validate_report(report: dict[str, Any], repo: Path, changed_paths: set[str], location = finding.get("code_location") if not isinstance(location, dict): raise SystemExit(f"finding {index} missing code_location") - rel = str(location.get("file_path", "")).strip() + allowed_location = {"file_path", "line"} + if set(location) != allowed_location: + raise SystemExit( + f"finding {index} has invalid code_location keys: " + f"{sorted(location)}" + ) + raw_file_path = location.get("file_path") + if not isinstance(raw_file_path, str) or not raw_file_path.strip(): + raise SystemExit(f"finding {index} has invalid location: {location}") + raw_rel = raw_file_path.strip() + normalized_rel = raw_rel if raw_rel in changed_paths else raw_rel.replace("\\", "/") + while normalized_rel.startswith("./"): + normalized_rel = normalized_rel[2:] + rel_path = PurePosixPath(normalized_rel) + rel = rel_path.as_posix() line = location.get("line") - if not rel or not isinstance(line, int) or line < 1: + if not isinstance(line, int) or isinstance(line, bool) or line < 1: raise SystemExit(f"finding {index} has invalid location: {location}") - if Path(rel).is_absolute() or ".." in Path(rel).parts: + if rel_path.is_absolute() or ".." in rel_path.parts or re.match(r"^[A-Za-z]:/", rel): raise SystemExit(f"finding {index} uses invalid file path: {rel}") + location["file_path"] = rel if rel not in changed_paths: ignored_findings.append((index, finding, rel, line)) continue @@ -1834,10 +10939,19 @@ def validate_report(report: dict[str, Any], repo: Path, changed_paths: set[str], for index, finding, rel, line in ignored_findings: title = finding.get("title", "") print( - f"autoreview ignored out-of-scope finding {index}: {title} ({rel}:{line})", + "autoreview ignored out-of-scope finding " + f"{index}: {display_escape(title, 140)} " + f"({display_escape(rel, 500)}:{line})", + file=sys.stderr, + ) + print( + display_escape( + finding.get("body", ""), + 500, + multiline=True, + ), file=sys.stderr, ) - print(bounded_field(str(finding.get("body", "")), 500), file=sys.stderr) report["findings"] = kept_findings if not kept_findings and report["overall_correctness"] == "patch is incorrect": note = f"Ignored {len(ignored_findings)} out-of-scope finding(s) outside the reviewed change." @@ -1850,51 +10964,147 @@ def validate_report(report: dict[str, Any], repo: Path, changed_paths: set[str], raise SystemExit(f"required finding text not found: {needle}") +def validate_report( + report: dict[str, Any], + repo: Path, + changed_paths: set[str], + required: list[str], +) -> None: + try: + _validate_report(report, repo, changed_paths, required) + except SystemExit as exc: + if isinstance(exc.code, str): + raise SystemExit( + display_escape(exc.code, 4000, multiline=True) + ) from None + raise + + def number_in_range(value: Any) -> bool: return isinstance(value, (int, float)) and not isinstance(value, bool) and 0 <= value <= 1 def print_report(report: dict[str, Any], *, label: str = "autoreview") -> None: findings = report["findings"] + display_label = display_escape(label, 200) if findings: - print(f"{label} findings: {len(findings)}") + print(f"{display_label} findings: {len(findings)}") elif report["overall_correctness"] == "patch is incorrect": - print(f"{label} verdict: patch is incorrect without discrete findings") + print( + f"{display_label} verdict: " + "patch is incorrect without discrete findings" + ) else: - print(f"{label} clean: no accepted/actionable findings reported") + print( + f"{display_label} clean: " + "no accepted/actionable findings reported" + ) for finding in findings: loc = finding["code_location"] - print(f"[{finding['priority']}] {finding['title']}") - print(f"{loc['file_path']}:{loc['line']}") - print(f"{finding['body']}") + print( + f"[{finding['priority']}] " + f"{display_escape(finding['title'], 140)}" + ) + print(f"{display_escape(loc['file_path'], 500)}:{loc['line']}") + print(display_escape(finding["body"], 2000, multiline=True)) print() print(f"overall: {report['overall_correctness']} ({report['overall_confidence']})") - print(report["overall_explanation"]) + print(display_escape(report["overall_explanation"], 3000, multiline=True)) -def start_parallel_tests(command: str, repo: Path, shell_kind: str) -> tuple[subprocess.Popen, float]: +def start_parallel_tests( + command: str, + repo: Path, + shell_kind: str, +) -> tuple[subprocess.Popen, float]: print(f"tests: {command}") - if shell_kind == "default" or shell_kind == "cmd": - return subprocess.Popen(command, cwd=repo, shell=True), time.time() - if shell_kind == "powershell": - powershell = resolve_command("powershell", repo) - return subprocess.Popen( - [powershell, "-NoProfile", "-ExecutionPolicy", "Bypass", "-Command", command], - cwd=repo, - ), time.time() - if shell_kind == "pwsh": - pwsh = resolve_command("pwsh", repo) - return subprocess.Popen( - [pwsh, "-NoProfile", "-Command", command], - cwd=repo, - ), time.time() - raise SystemExit(f"invalid --parallel-tests-shell/AUTOREVIEW_PARALLEL_TESTS_SHELL: {shell_kind}") + test_home = Path( + tempfile.mkdtemp( + prefix="autoreview-test-home-", + dir=parallel_test_temp_root(repo), + ) + ) + try: + env = safe_test_env(repo, test_home) + popen_kwargs = { + "cwd": repo, + "env": env, + "stderr": subprocess.PIPE, + "text": True, + "encoding": SUBPROCESS_TEXT_ENCODING, + "errors": SUBPROCESS_TEXT_ERRORS, + } + if shell_kind == "default" or shell_kind == "cmd": + proc = subprocess.Popen(command, shell=True, **popen_kwargs) + elif shell_kind == "powershell": + powershell = resolve_command("powershell", repo) + proc = subprocess.Popen( + [powershell, "-NoProfile", "-ExecutionPolicy", "Bypass", "-Command", command], + **popen_kwargs, + ) + elif shell_kind == "pwsh": + pwsh = resolve_command("pwsh", repo) + proc = subprocess.Popen( + [pwsh, "-NoProfile", "-Command", command], + **popen_kwargs, + ) + else: + raise SystemExit( + f"invalid --parallel-tests-shell/AUTOREVIEW_PARALLEL_TESTS_SHELL: {shell_kind}" + ) + except BaseException: + shutil.rmtree(test_home, ignore_errors=True) + raise + if proc.stderr is None: + shutil.rmtree(test_home, ignore_errors=True) + raise SystemExit("parallel test stderr pipe was not created") + stderr_thread = threading.Thread( + target=relay_parallel_test_stderr, + args=(proc.stderr, env["JAVA_TOOL_OPTIONS"]), + daemon=True, + ) + stderr_thread.start() + setattr(proc, "_autoreview_test_home", test_home) + setattr(proc, "_autoreview_stderr_thread", stderr_thread) + return proc, time.time() + + +def relay_parallel_test_stderr(stream: Any, java_tool_options: str) -> None: + suppressed = { + f"Picked up JAVA_TOOL_OPTIONS: {java_tool_options}", + f"NOTE: Picked up JAVA_TOOL_OPTIONS: {java_tool_options}", + } + for line in stream: + if line.rstrip("\r\n") in suppressed: + continue + sys.stderr.write(line) + sys.stderr.flush() def finish_parallel_tests(proc: subprocess.Popen, started: float) -> int: - proc.wait() - print(f"tests exit: {proc.returncode} after {int(time.time() - started)}s") - return int(proc.returncode or 0) + try: + proc.wait() + stderr_thread = getattr(proc, "_autoreview_stderr_thread", None) + if isinstance(stderr_thread, threading.Thread): + stderr_thread.join(timeout=0.25) + print(f"tests exit: {proc.returncode} after {int(time.time() - started)}s") + return int(proc.returncode or 0) + finally: + test_home = getattr(proc, "_autoreview_test_home", None) + if isinstance(test_home, Path): + shutil.rmtree(test_home, ignore_errors=True) + + +def env_truthy(name: str) -> bool: + value = os.environ.get(name) + if value is None: + return False + normalized = value.strip().lower() + if normalized in {"1", "true", "yes", "on"}: + return True + if normalized in {"", "0", "false", "no", "off"}: + return False + raise SystemExit(f"invalid boolean environment value for {name}: {value}") def parse_args() -> argparse.Namespace: @@ -1902,21 +11112,15 @@ def parse_args() -> argparse.Namespace: parser.add_argument("--mode", choices=["auto", "local", "uncommitted", "branch", "commit"], default="auto") parser.add_argument("--base") parser.add_argument("--commit", default="HEAD") - parser.add_argument("--engine", choices=ENGINES, default=os.environ.get("AUTOREVIEW_ENGINE", "codex")) - parser.add_argument("--reviewers", help="Comma-separated review panel, e.g. codex,claude,pi or codex::high.") + parser.add_argument("--engine", choices=ENGINE_CHOICES, default=os.environ.get("AUTOREVIEW_ENGINE", "codex")) + parser.add_argument("--reviewers", help="Comma-separated review panel, e.g. codex,claude,pi or codex:gpt-5.6-sol:high.") parser.add_argument("--panel", action="store_true", help="Run a Codex/Claude review panel unless --engine changes the first reviewer.") - parser.add_argument( - "--preset", - choices=sorted(REVIEW_PRESETS), - default=os.environ.get("AUTOREVIEW_PRESET"), - help="Reviewer preset, e.g. claude-opus.", - ) parser.add_argument( "--model", action="append", - help="Model for all reviewers or engine=model. Repeatable. Defaults to each reviewer CLI's configured model.", + help="Model for all reviewers or engine=model. Repeatable. Defaults: codex=gpt-5.6-sol with an access-only gpt-5.6-terra retry, claude=claude-fable-5.", ) - parser.add_argument("--thinking", action="append", help="Thinking/effort for all reviewers or engine=level. Repeatable. Codex: none, minimal, low, medium, high, xhigh. Claude: low, medium, high, xhigh, max. Droid: off, none, low, medium, high. Pi: off, minimal, low, medium, high, xhigh. OpenCode: minimal, low, medium, high, max.") + parser.add_argument("--thinking", action="append", help="Thinking/effort for all reviewers or engine=level. Repeatable. Codex: none, minimal, low, medium, high, xhigh, max. Claude: low, medium, high, xhigh, max. Droid: off, none, low, medium, high. Pi: off, minimal, low, medium, high, xhigh. OpenCode: minimal, low, medium, high, max. Cursor: none.") parser.add_argument( "--fallback-model", action="append", @@ -1924,21 +11128,41 @@ def parse_args() -> argparse.Namespace: ) parser.add_argument("--allow-partial-panel", action="store_true", help="Continue panel output when one reviewer fails.") parser.add_argument("--codex-bin", default=os.environ.get("CODEX_BIN", "codex")) + parser.add_argument( + "--codex-config", + action="append", + help='Safe Codex model/response tuning "-c key=value" override (TOML value), codex reviewer only. Repeatable. Capability-, command-, and path-bearing keys are refused. Env default: AUTOREVIEW_CODEX_CONFIG (semicolon-separated), e.g. service_tier="fast".', + ) + parser.add_argument( + "--codex-speed", + choices=["fast", "flex", "default"], + help="Codex service tier: fast (priority processing), flex, or default. Env default: AUTOREVIEW_CODEX_SPEED. Silently standard when the model catalog does not list the tier.", + ) parser.add_argument("--claude-bin", default=os.environ.get("CLAUDE_BIN", "claude")) parser.add_argument("--droid-bin", default=os.environ.get("DROID_BIN", "droid")) parser.add_argument("--copilot-bin", default=os.environ.get("COPILOT_BIN", "copilot")) + parser.add_argument( + "--cursor-bin", + "--cursor-agent-bin", + dest="cursor_bin", + default=os.environ.get("CURSOR_BIN") + or os.environ.get("CURSOR_AGENT_BIN", "cursor-agent"), + ) parser.add_argument("--opencode-bin", default=os.environ.get("OPENCODE_BIN", "opencode")) parser.add_argument("--pi-bin", default=os.environ.get("PI_BIN", "pi")) - parser.add_argument("--no-tools", dest="tools", action="store_false", default=True, help="Disable tools for engines that support it. Codex, copilot, and opencode reject no-tools review.") + parser.add_argument("--no-tools", dest="tools", action="store_false", default=True, help="Disable tools for engines that support it. Codex, Droid, copilot, opencode, and cursor reject no-tools review.") + parser.add_argument("--self-test", action="store_true", help="Run deterministic local autoreview self-tests.") parser.add_argument("--self-test-opencode-jsonl-parser", action="store_true", help=argparse.SUPPRESS) parser.add_argument("--self-test-opencode-isolation", action="store_true", help=argparse.SUPPRESS) parser.add_argument("--self-test-opencode-real-project-isolation", action="store_true", help=argparse.SUPPRESS) + parser.add_argument("--self-test-cursor-jsonl-parser", action="store_true", help=argparse.SUPPRESS) + parser.add_argument("--self-test-cursor-isolation", action="store_true", help=argparse.SUPPRESS) parser.add_argument("--no-web-search", dest="web_search", action="store_false", default=True) parser.add_argument( "--claude-allowed-tools", default=os.environ.get( "AUTOREVIEW_CLAUDE_TOOLS", - "Read,Grep,Glob,WebSearch,WebFetch", + "WebSearch", ), ) parser.add_argument("--prompt", action="append", help="Additional review instruction text.") @@ -1950,7 +11174,20 @@ def parse_args() -> argparse.Namespace: "--stream-engine-output", action="store_true", default=os.environ.get("AUTOREVIEW_STREAM_ENGINE_OUTPUT") == "1", - help="Stream review engine output while preserving buffered output for validation. Codex output is filtered to hide tool/file chatter.", + help="Stream review engine output while preserving buffered output for validation. Codex and Claude filter noisy tool/status chatter.", + ) + parser.add_argument( + "--cursor-allow-workspace-instructions", + dest="cursor_allow_workspace_instructions", + action="store_true", + default=None, + help="Legacy compatibility flag. Cursor review is unavailable because reads cannot be confined to the repository.", + ) + parser.add_argument( + "--no-cursor-allow-workspace-instructions", + dest="cursor_allow_workspace_instructions", + action="store_false", + help="Legacy compatibility flag. Cursor review remains unavailable.", ) parser.add_argument("--parallel-tests", help="Run a test command concurrently with review; failure fails the helper.") parser.add_argument( @@ -1961,19 +11198,18 @@ def parse_args() -> argparse.Namespace: ) parser.add_argument("--require-finding", action="append", default=[], help="Require finding text to contain this substring.") parser.add_argument("--expect-findings", action="store_true", help="Treat findings as success; for harness acceptance tests.") - parser.add_argument("--include-risky-paths", action="store_true", help="Include risky file paths such as .env, keys, tokens, credentials, and logs.") - parser.add_argument("--include-risky-untracked", action="store_true", help=argparse.SUPPRESS) parser.add_argument("--dry-run", action="store_true") parser.add_argument("--self-test-config-defaults", action="store_true", help=argparse.SUPPRESS) parser.add_argument("--self-test-fallback-scope", action="store_true", help=argparse.SUPPRESS) parser.add_argument("--self-test-engine-isolation", action="store_true", help=argparse.SUPPRESS) parser.add_argument("--self-test-json-array-parser", action="store_true", help=argparse.SUPPRESS) + parser.add_argument("--self-test-heartbeat-metrics", action="store_true", help=argparse.SUPPRESS) args = parser.parse_args() + args.engine = normalize_engine(args.engine) + if args.cursor_allow_workspace_instructions is None: + args.cursor_allow_workspace_instructions = env_truthy("AUTOREVIEW_CURSOR_ALLOW_WORKSPACE_INSTRUCTIONS") if args.engine not in ENGINES: raise SystemExit(f"invalid --engine/AUTOREVIEW_ENGINE: {args.engine}") - if args.preset and args.preset not in REVIEW_PRESETS: - valid = ", ".join(sorted(REVIEW_PRESETS)) - raise SystemExit(f"invalid --preset/AUTOREVIEW_PRESET: {args.preset} (valid: {valid})") return args @@ -1990,21 +11226,29 @@ def run_engine(args: argparse.Namespace, repo: Path, prompt: str) -> str: return run_pi(args, repo, prompt) if args.engine == "opencode": return run_opencode(args, repo, prompt) + if args.engine == "cursor": + return run_cursor(args, repo, prompt) raise SystemExit(f"unsupported engine: {args.engine}") +def normalize_engine(engine: str) -> str: + return ENGINE_ALIASES.get(engine, engine) + + def env_defaults_for(env_suffix: str) -> tuple[str | None, dict[str, str]]: env_key = env_suffix.replace("-", "_").upper() global_value = os.environ.get(f"AUTOREVIEW_{env_key}") if global_value is not None: global_value = global_value.strip() or None per_engine: dict[str, str] = {} - for engine in ENGINES: - value = os.environ.get(f"AUTOREVIEW_{engine.upper()}_{env_key}") + for configured_engine in ENGINE_CHOICES: + engine = normalize_engine(configured_engine) + configured_key = configured_engine.replace("-", "_").upper() + value = os.environ.get(f"AUTOREVIEW_{configured_key}_{env_key}") if value is None: continue value = value.strip() - if value: + if value and engine not in per_engine: per_engine[engine] = value return global_value, per_engine @@ -2020,8 +11264,9 @@ def parse_keyed_options(values: list[str] | None, option: str) -> tuple[str | No engine, engine_value = value.split("=", 1) engine = engine.strip() engine_value = engine_value.strip() - if engine not in ENGINES: + if engine not in ENGINE_CHOICES: raise SystemExit(f"--{option} uses unknown engine: {engine}") + engine = normalize_engine(engine) if not engine_value: raise SystemExit(f"--{option} for {engine} cannot be empty") if engine in per_engine: @@ -2039,8 +11284,9 @@ def parse_reviewer_token(token: str) -> tuple[str, str | None, str | None]: if len(parts) > 3 or not parts[0]: raise SystemExit(f"invalid reviewer spec: {token}") engine = parts[0] - if engine not in ENGINES: + if engine not in ENGINE_CHOICES: raise SystemExit(f"unknown reviewer engine: {engine}") + engine = normalize_engine(engine) model = parts[1] if len(parts) >= 2 and parts[1] else None thinking = parts[2] if len(parts) == 3 and parts[2] else None return engine, model, thinking @@ -2054,16 +11300,11 @@ def reviewer_args(args: argparse.Namespace) -> list[argparse.Namespace]: env_global_thinking, env_thinking_by_engine = env_defaults_for("thinking") env_global_fallback, env_fallback_by_engine = env_defaults_for("fallback-model") reviewers: list[tuple[str, str | None, str | None]] = [] - selector_count = sum(bool(value) for value in (args.reviewers, args.panel, args.preset)) - if selector_count > 1: - raise SystemExit("--reviewers, --panel, and --preset are mutually exclusive") if args.reviewers: tokens = [token.strip() for token in args.reviewers.split(",") if token.strip()] if len(tokens) == 1 and tokens[0] == "all": - tokens = list(ENGINES) + tokens = list(ALL_REVIEWERS) reviewers = [parse_reviewer_token(token) for token in tokens] - elif args.preset: - reviewers = list(REVIEW_PRESETS[args.preset]) elif args.panel: engines = [args.engine] for engine in ("codex", "claude"): @@ -2085,6 +11326,10 @@ def reviewer_args(args: argparse.Namespace) -> list[argparse.Namespace]: raise SystemExit(f"--fallback-model is only supported for claude, not {engine_list}") if (global_fallback or env_global_fallback) and "claude" not in selected_engines: raise SystemExit("--fallback-model is only supported for claude; no claude reviewer selected") + if getattr(args, "codex_config", None) and "codex" not in selected_engines: + raise SystemExit("--codex-config is only supported for codex; no codex reviewer selected") + if getattr(args, "codex_speed", None) and "codex" not in selected_engines: + raise SystemExit("--codex-speed is only supported for codex; no codex reviewer selected") seen: set[str] = set() result: list[argparse.Namespace] = [] @@ -2106,6 +11351,7 @@ def reviewer_args(args: argparse.Namespace) -> list[argparse.Namespace]: or global_thinking or env_thinking_by_engine.get(engine) or env_global_thinking + or DEFAULT_THINKING_BY_ENGINE.get(engine) ) if engine == "claude": fallback_model = ( @@ -2114,6 +11360,8 @@ def reviewer_args(args: argparse.Namespace) -> list[argparse.Namespace]: or env_fallback_by_engine.get(engine) or env_global_fallback ) + elif engine == "codex" and model == DEFAULT_MODEL_BY_ENGINE["codex"]: + fallback_model = DEFAULT_CODEX_ACCESS_FALLBACK_MODEL else: fallback_model = None if thinking and thinking not in THINKING_LEVELS_BY_ENGINE[engine]: @@ -2124,6 +11372,7 @@ def reviewer_args(args: argparse.Namespace) -> list[argparse.Namespace]: clone.model = model clone.thinking = thinking clone.fallback_model = fallback_model + clone.tools = False if engine in {"droid", "pi"} else args.tools result.append(clone) return result @@ -2139,11 +11388,32 @@ def reviewer_label(args: argparse.Namespace) -> str: return " ".join(parts) -def run_reviewer(args: argparse.Namespace, repo: Path, prompt: str, changed_paths: set[str], required: list[str]) -> dict[str, Any]: - raw = run_engine(args, repo, prompt) - report = extract_json(raw) - validate_report(report, repo, changed_paths, required) - return report +def run_reviewer( + args: argparse.Namespace, + repo: Path, + prompt: str, + changed_paths: set[str], + required: list[str], + input_truncated: bool = False, +) -> dict[str, Any]: + ensure_reviewer_input_complete(args, input_truncated) + attempts = 3 if args.engine == "cursor" else 1 + for attempt in range(1, attempts + 1): + raw = run_engine(args, repo, prompt) + try: + report = extract_json(raw) + validate_report(report, repo, changed_paths, required) + return report + except SystemExit as exc: + if attempt >= attempts or not is_structured_output_failure(str(exc)): + raise + print( + "retrying " + f"{args.engine} structured output validation after attempt " + f"{attempt}: {display_escape(exc, 4000, multiline=True)}", + file=sys.stderr, + ) + raise SystemExit(f"{args.engine} structured output validation failed after {attempts} attempts") def merge_panel_reports(reports: list[tuple[str, dict[str, Any]]]) -> dict[str, Any]: @@ -2174,12 +11444,33 @@ def merge_panel_reports(reports: list[tuple[str, dict[str, Any]]]) -> dict[str, } -def run_panel(args: argparse.Namespace, reviewers: list[argparse.Namespace], repo: Path, prompt: str, changed_paths: set[str]) -> dict[str, Any]: +def merge_chunk_reports(reports: list[tuple[str, dict[str, Any]]]) -> dict[str, Any]: + report = merge_panel_reports(reports) + summary = ", ".join( + f"{label}: {len(chunk_report['findings'])} finding(s)" + for label, chunk_report in reports + ) + report["overall_explanation"] = bounded_field( + f"Chunked review complete. {summary}.", + 3000, + ) + return report + + +def run_panel( + args: argparse.Namespace, + reviewers: list[argparse.Namespace], + repo: Path, + prompt: str, + changed_paths: set[str], + input_truncated: bool, + required: list[str] | None = None, +) -> dict[str, Any]: reports: list[tuple[str, dict[str, Any]]] = [] failures: list[str] = [] with concurrent.futures.ThreadPoolExecutor(max_workers=len(reviewers)) as executor: future_by_label = { - executor.submit(run_reviewer, reviewer, repo, prompt, changed_paths, []): reviewer_label(reviewer) + executor.submit(run_reviewer, reviewer, repo, prompt, changed_paths, [], input_truncated): reviewer_label(reviewer) for reviewer in reviewers } for future in concurrent.futures.as_completed(future_by_label): @@ -2190,16 +11481,29 @@ def run_panel(args: argparse.Namespace, reviewers: list[argparse.Namespace], rep failures.append(f"{label}: {exc}") except Exception as exc: failures.append(f"{label}: {exc}") - if failures and not args.allow_partial_panel: - raise SystemExit("autoreview panel failed\n" + "\n".join(failures)) - if failures: - for failure in failures: - print(f"panel reviewer failed: {failure}") + escaped_failures = [ + display_escape(failure, 4000, multiline=True) + for failure in failures + ] + if escaped_failures and not args.allow_partial_panel: + raise SystemExit( + "autoreview panel failed\n" + "\n".join(escaped_failures) + ) + if escaped_failures: + for failure in escaped_failures: + print( + "panel reviewer failed: " + failure + ) if not reports: raise SystemExit("autoreview panel produced no reports") reports.sort(key=lambda item: item[0]) report = merge_panel_reports(reports) - validate_report(report, repo, changed_paths, args.require_finding) + validate_report( + report, + repo, + changed_paths, + args.require_finding if required is None else required, + ) return report @@ -2207,11 +11511,13 @@ def reviewer_test_args(**overrides: Any) -> argparse.Namespace: defaults = { "reviewers": None, "panel": False, - "preset": None, "engine": "codex", "model": None, "thinking": None, "fallback_model": None, + "codex_config": None, + "codex_speed": None, + "tools": True, } defaults.update(overrides) return argparse.Namespace(**defaults) @@ -2239,18 +11545,38 @@ def preserve_env(keys: list[str]): def self_test_config_defaults() -> None: keys = [ "AUTOREVIEW_MODEL", - "AUTOREVIEW_CODEX_MODEL", - "AUTOREVIEW_CLAUDE_MODEL", "AUTOREVIEW_THINKING", - "AUTOREVIEW_CODEX_THINKING", - "AUTOREVIEW_CLAUDE_THINKING", + "AUTOREVIEW_FALLBACK_MODEL", + "AUTOREVIEW_CODEX_CONFIG", + "AUTOREVIEW_CODEX_SPEED", + *( + f"AUTOREVIEW_{engine.upper()}_{suffix}" + for engine in ENGINES + for suffix in ("MODEL", "THINKING", "FALLBACK_MODEL") + ), ] with preserve_env(keys): + for key in keys: + os.environ.pop(key, None) default_codex = reviewer_args(reviewer_test_args(engine="codex"))[0] - if default_codex.model is not None: + if default_codex.model != "gpt-5.6-sol": raise SystemExit(f"self-test config defaults failed: default codex model={default_codex.model!r}") + if default_codex.fallback_model != "gpt-5.6-terra": + raise SystemExit( + f"self-test config defaults failed: default codex fallback={default_codex.fallback_model!r}" + ) + explicit_sol = reviewer_args(reviewer_test_args(engine="codex", model=["gpt-5.6-sol"]))[0] + if explicit_sol.fallback_model != "gpt-5.6-terra": + raise SystemExit( + f"self-test config defaults failed: explicit Sol access fallback={explicit_sol.fallback_model!r}" + ) + if default_codex.thinking != "high": + raise SystemExit(f"self-test config defaults failed: default codex thinking={default_codex.thinking!r}") + max_effort = reviewer_args(reviewer_test_args(engine="codex", thinking=["max"]))[0] + if max_effort.thinking != "max": + raise SystemExit("self-test config defaults failed: Codex max thinking should be accepted") default_claude = reviewer_args(reviewer_test_args(engine="claude"))[0] - if default_claude.model is not None: + if default_claude.model != "claude-fable-5": raise SystemExit(f"self-test config defaults failed: default claude model={default_claude.model!r}") os.environ["AUTOREVIEW_MODEL"] = "env-global-model" os.environ["AUTOREVIEW_CODEX_MODEL"] = "env-codex-model" @@ -2280,33 +11606,88 @@ def self_test_config_defaults() -> None: )[0] if inline.model != "inline-model" or inline.thinking != "minimal": raise SystemExit("self-test config defaults failed: inline reviewer values should override CLI/env") - preset = reviewer_args(reviewer_test_args(preset="claude-opus")) - if len(preset) != 1 or preset[0].engine != "claude" or preset[0].model != CLAUDE_OPUS_MODEL: - raise SystemExit("self-test config defaults failed: claude-opus preset model incorrect") + os.environ["AUTOREVIEW_CODEX_CONFIG"] = ' service_tier="fast" ; ' + env_overrides = codex_config_overrides(reviewer_test_args(engine="codex")) + if env_overrides != ['service_tier="fast"']: + raise SystemExit(f"self-test config defaults failed: codex config env overrides={env_overrides!r}") + flag_overrides = codex_config_overrides( + reviewer_test_args(engine="codex", codex_config=['model_verbosity="low"']) + ) + if flag_overrides != ['model_verbosity="low"']: + raise SystemExit(f"self-test config defaults failed: codex config flag should override env, got {flag_overrides!r}") + os.environ["AUTOREVIEW_CODEX_CONFIG"] = "no-equals-sign" + rejected = False try: - reviewer_args(reviewer_test_args(reviewers="codex", preset="claude-opus")) - raise SystemExit("self-test config defaults failed: preset should reject other selectors") - except SystemExit as exc: - if "mutually exclusive" not in str(exc): + codex_config_overrides(reviewer_test_args(engine="codex")) + except SystemExit as error: + rejected = "invalid Codex config override" in str(error) + if not rejected: + raise SystemExit("self-test config defaults failed: malformed codex config override accepted") + rejected = False + try: + codex_config_overrides( + reviewer_test_args( + engine="codex", + codex_config=['mcp_servers.review.command="touch /tmp/owned"'], + ) + ) + except SystemExit as error: + rejected = "unsafe Codex config override refused" in str(error) + if not rejected: + raise SystemExit("self-test config defaults failed: capability-bearing codex config override accepted") + os.environ.pop("AUTOREVIEW_CODEX_CONFIG") + try: + reviewer_args(reviewer_test_args(engine="claude", codex_config=['service_tier="fast"'])) + raise SystemExit("self-test config defaults failed: --codex-config accepted without codex reviewer") + except SystemExit as error: + if "only supported for codex" not in str(error): + raise + os.environ["AUTOREVIEW_CODEX_SPEED"] = "fast" + env_speed = codex_speed_override(reviewer_test_args(engine="codex")) + if env_speed != 'service_tier="fast"': + raise SystemExit(f"self-test config defaults failed: codex speed env override={env_speed!r}") + flag_speed = codex_speed_override(reviewer_test_args(engine="codex", codex_speed="flex")) + if flag_speed != 'service_tier="flex"': + raise SystemExit(f"self-test config defaults failed: codex speed flag should override env, got {flag_speed!r}") + os.environ["AUTOREVIEW_CODEX_SPEED"] = "warp" + rejected = False + try: + codex_speed_override(reviewer_test_args(engine="codex")) + except SystemExit as error: + rejected = "invalid Codex speed" in str(error) + if not rejected: + raise SystemExit("self-test config defaults failed: invalid codex speed accepted") + os.environ.pop("AUTOREVIEW_CODEX_SPEED") + try: + reviewer_args(reviewer_test_args(engine="claude", codex_speed="fast")) + raise SystemExit("self-test config defaults failed: --codex-speed accepted without codex reviewer") + except SystemExit as error: + if "only supported for codex" not in str(error): raise print("self-test config defaults: ok") def self_test_fallback_scope() -> None: keys = [ + "AUTOREVIEW_MODEL", "AUTOREVIEW_FALLBACK_MODEL", + "AUTOREVIEW_CODEX_MODEL", "AUTOREVIEW_CLAUDE_FALLBACK_MODEL", "AUTOREVIEW_CODEX_FALLBACK_MODEL", ] with preserve_env(keys): + for key in keys: + os.environ.pop(key, None) os.environ["AUTOREVIEW_FALLBACK_MODEL"] = "env-global-fallback" os.environ["AUTOREVIEW_CLAUDE_FALLBACK_MODEL"] = "env-claude-fallback" base = reviewer_test_args(reviewers="codex,claude") reviewers = reviewer_args(base) codex = next(r for r in reviewers if r.engine == "codex") claude = next(r for r in reviewers if r.engine == "claude") - if codex.fallback_model is not None: - raise SystemExit("self-test fallback scope failed: codex should ignore AUTOREVIEW_FALLBACK_MODEL") + if codex.fallback_model != "gpt-5.6-terra": + raise SystemExit( + f"self-test fallback scope failed: codex access fallback={codex.fallback_model!r}" + ) if claude.fallback_model != "env-claude-fallback": raise SystemExit(f"self-test fallback scope failed: claude fallback={claude.fallback_model!r}") os.environ.pop("AUTOREVIEW_CLAUDE_FALLBACK_MODEL") @@ -2330,7 +11711,7 @@ def self_test_fallback_scope() -> None: panel = reviewer_args(reviewer_test_args(reviewers="codex,claude", fallback_model=["cli-global"])) panel_codex = next(r for r in panel if r.engine == "codex") panel_claude = next(r for r in panel if r.engine == "claude") - if panel_codex.fallback_model is not None or panel_claude.fallback_model != "cli-global": + if panel_codex.fallback_model != "gpt-5.6-terra" or panel_claude.fallback_model != "cli-global": raise SystemExit("self-test fallback scope failed: CLI global fallback should apply only to Claude panel reviewers") try: reviewer_args(reviewer_test_args(engine="codex", fallback_model=["cli-global"])) @@ -2379,8 +11760,64 @@ def self_test_fallback_scope() -> None: print("self-test fallback scope: ok") +def self_test() -> int: + self_test_opencode_jsonl_parser() + self_test_opencode_isolation() + self_test_config_defaults() + self_test_fallback_scope() + self_test_heartbeat_metrics() + self_test_json_array_parser() + return self_test_engine_isolation() + + +def reject_repo_output_paths(args: argparse.Namespace, repo: Path) -> None: + repo_root_path = repo.resolve() + for option, value in ( + ("--json-output", args.json_output), + ("--output", args.output), + ): + if not value: + continue + path = Path(value).expanduser() + resolved = ( + path if path.is_absolute() else Path.cwd() / path + ).resolve() + inside_repo = resolved.is_relative_to(repo_root_path) + if not inside_repo: + for ancestor in (resolved, *resolved.parents): + try: + if os.path.samefile(ancestor, repo_root_path): + inside_repo = True + break + except OSError: + continue + if not inside_repo: + continue + raise SystemExit( + f"{option} must point outside the reviewed repository: " + f"{display_escape(value, 500)}" + ) + + +def atomic_write_text(path: Path, content: str) -> None: + parent = path.parent + descriptor, temporary = tempfile.mkstemp( + dir=parent, + prefix=f".{path.name}.", + ) + temporary_path = Path(temporary) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + handle.write(content) + os.replace(temporary_path, path) + finally: + temporary_path.unlink(missing_ok=True) + + def main() -> int: args = parse_args() + if args.self_test: + return self_test() if args.self_test_opencode_jsonl_parser: self_test_opencode_jsonl_parser() return 0 @@ -2390,18 +11827,26 @@ def main() -> int: if args.self_test_opencode_real_project_isolation: self_test_opencode_real_project_isolation(args) return 0 + if args.self_test_cursor_jsonl_parser: + return self_test_cursor_jsonl_parser() + if args.self_test_cursor_isolation: + return self_test_engine_isolation() if args.self_test_config_defaults: self_test_config_defaults() return 0 if args.self_test_fallback_scope: self_test_fallback_scope() return 0 + if args.self_test_heartbeat_metrics: + self_test_heartbeat_metrics() + return 0 if args.self_test_engine_isolation: return self_test_engine_isolation() if args.self_test_json_array_parser: return self_test_json_array_parser() reviewers = reviewer_args(args) repo = repo_root() + reject_repo_output_paths(args, repo) target, target_ref = choose_target(repo, args.mode, args.base) print(f"autoreview target: {target}") print(f"branch: {current_branch(repo)}") @@ -2413,9 +11858,18 @@ def main() -> int: print(f"fallback_model: {reviewers[0].fallback_model}") if reviewers[0].thinking: print(f"thinking: {reviewers[0].thinking}") + if reviewers[0].engine == "codex": + config_keys = codex_config_keys(reviewers[0]) + if config_keys: + print(f"codex_config_keys: {', '.join(config_keys)}") + speed = codex_speed_override(reviewers[0]) + if speed: + print(f"codex_speed: {speed}") else: print(f"reviewers: {', '.join(reviewer_label(reviewer) for reviewer in reviewers)}") - print(f"tools: {'on' if args.tools else 'off'}") + tool_states = {reviewer.tools for reviewer in reviewers} + tools_label = "mixed" if len(tool_states) > 1 else ("on" if tool_states.pop() else "off") + print(f"tools: {tools_label}") print(f"web_search: {'on' if args.web_search else 'off'}") display_ref = args.commit if target == "commit" else target_ref if display_ref: @@ -2423,42 +11877,106 @@ def main() -> int: if args.dry_run: return 0 + review_source_snapshot = source_tree_snapshot(repo) + run_trufflehog_preflight(repo, target, target_ref, args.commit) if target == "local": - bundle = local_bundle(repo, include_risky_paths=args.include_risky_paths or args.include_risky_untracked) + bundle, bundle_truncated = local_bundle(repo) elif target == "branch": assert target_ref - bundle = branch_bundle(repo, target_ref, include_risky_paths=args.include_risky_paths or args.include_risky_untracked) + bundle, bundle_truncated = branch_bundle(repo, target_ref) else: - bundle = commit_bundle(repo, args.commit, include_risky_paths=args.include_risky_paths or args.include_risky_untracked) + bundle, bundle_truncated = commit_bundle(repo, args.commit) target_ref = args.commit - prompt = build_prompt(repo, target, target_ref, bundle, load_extra_prompt(args), load_datasets(args)) + extra_prompt, prompt_truncated = load_extra_prompt(args, repo) + datasets, datasets_truncated = load_datasets(args, repo) + input_truncated = bundle_truncated or prompt_truncated or datasets_truncated + prompts = build_review_prompts( + repo, + target, + target_ref, + bundle, + extra_prompt, + datasets, + ) changed_paths = review_paths(repo, target, target_ref, args.commit) - print(f"bundle: {len(prompt)} chars") + print(f"bundle: {utf8_size(bundle)} bytes; review passes: {len(prompts)}") + if source_tree_snapshot(repo) != review_source_snapshot: + raise SystemExit( + "source changed while the review bundle was being created; " + "rerun autoreview against the updated tree" + ) tests_proc: tuple[subprocess.Popen, float] | None = None if args.parallel_tests: tests_proc = start_parallel_tests(args.parallel_tests, repo, args.parallel_tests_shell) try: - if len(reviewers) == 1: - report = run_reviewer(reviewers[0], repo, prompt, changed_paths, args.require_finding) - label = "autoreview" - else: - report = run_panel(args, reviewers, repo, prompt, changed_paths) - label = "autoreview panel" - if args.json_output: - Path(args.json_output).write_text(json.dumps(report, indent=2) + "\n") - - if args.output: - original_stdout = sys.stdout - with Path(args.output).open("w") as handle: - sys.stdout = Tee(original_stdout, handle) - print_report(report, label=label) - sys.stdout = original_stdout + chunk_reports: list[tuple[str, dict[str, Any]]] = [] + for index, prompt in enumerate(prompts, start=1): + if len(prompts) > 1: + print( + f"review pass: {index}/{len(prompts)} " + f"({utf8_size(prompt)} prompt bytes)" + ) + required = args.require_finding if len(prompts) == 1 else [] + if len(reviewers) == 1: + chunk_report = run_reviewer( + reviewers[0], + repo, + prompt, + changed_paths, + required, + input_truncated, + ) + else: + chunk_report = run_panel( + args, + reviewers, + repo, + prompt, + changed_paths, + input_truncated, + required, + ) + chunk_reports.append((f"chunk {index}/{len(prompts)}", chunk_report)) + if len(chunk_reports) == 1: + report = chunk_reports[0][1] else: - print_report(report, label=label) + report = merge_chunk_reports(chunk_reports) + validate_report(report, repo, changed_paths, args.require_finding) + label = "autoreview panel" if len(reviewers) > 1 else "autoreview" + if len(chunk_reports) > 1: + label += " chunked" finally: tests_status = finish_parallel_tests(*tests_proc) if tests_proc else 0 + if source_tree_snapshot(repo) != review_source_snapshot: + print( + "source changed after the review bundle was created; " + "rerun autoreview against the updated tree", + file=sys.stderr, + ) + return 1 + + if args.json_output: + atomic_write_text( + Path(args.json_output), + json.dumps(report, indent=2) + "\n", + ) + + if args.output: + rendered = io.StringIO() + original_stdout = sys.stdout + try: + sys.stdout = rendered + print_report(report, label=label) + finally: + sys.stdout = original_stdout + output = rendered.getvalue() + print(output, end="") + atomic_write_text(Path(args.output), output) + else: + print_report(report, label=label) + has_findings = bool(report["findings"]) overall_incorrect = report["overall_correctness"] == "patch is incorrect" if tests_status != 0: @@ -2468,18 +11986,16 @@ def main() -> int: return 1 if has_findings or overall_incorrect else 0 -class Tee: - def __init__(self, *streams: Any) -> None: - self.streams = streams - - def write(self, data: str) -> None: - for stream in self.streams: - stream.write(data) - - def flush(self) -> None: - for stream in self.streams: - stream.flush() +def sanitized_main() -> int: + try: + return main() + except SystemExit as exc: + if isinstance(exc.code, str): + raise SystemExit( + display_escape(exc.code, 4000, multiline=True) + ) from None + raise if __name__ == "__main__": - raise SystemExit(main()) + raise SystemExit(sanitized_main()) diff --git a/skills/autoreview/scripts/autoreview_test.py b/skills/autoreview/scripts/autoreview_test.py new file mode 100644 index 0000000..f3cbc20 --- /dev/null +++ b/skills/autoreview/scripts/autoreview_test.py @@ -0,0 +1,678 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import importlib.util +import json +import os +import runpy +import subprocess +import sys +import tempfile +import unittest +from importlib.machinery import SourceFileLoader +from pathlib import Path +from unittest import mock + + +SCRIPT_PATH = Path(__file__).with_name("autoreview") +LOADER = SourceFileLoader("autoreview_module", str(SCRIPT_PATH)) +SPEC = importlib.util.spec_from_loader(LOADER.name, LOADER) +assert SPEC is not None +AUTOREVIEW = importlib.util.module_from_spec(SPEC) +LOADER.exec_module(AUTOREVIEW) + + +FINAL_REPORT = { + "findings": [], + "overall_correctness": "patch is correct", + "overall_explanation": "clean", + "overall_confidence": 0.9, +} + +DRAFT_REPORT = { + "findings": [ + { + "title": "Draft finding", + "body": "draft", + "priority": "P3", + "confidence": 0.2, + "category": "maintainability", + "code_location": {"file_path": "draft.js", "line": 1}, + } + ], + "overall_correctness": "patch is incorrect", + "overall_explanation": "draft", + "overall_confidence": 0.2, +} + + +class AutoreviewCursorTests(unittest.TestCase): + def test_extract_json_prefers_terminal_result_event(self) -> None: + stream = "\n".join( + [ + json.dumps( + { + "type": "assistant", + "message": {"role": "assistant", "content": [{"type": "text", "text": json.dumps(DRAFT_REPORT)}]}, + } + ), + json.dumps( + { + "type": "result", + "subtype": "success", + "result": json.dumps(FINAL_REPORT), + "session_id": "session-id", + "request_id": "request-id", + } + ), + ] + ) + self.assertEqual(AUTOREVIEW.extract_json(stream), FINAL_REPORT) + + def test_extract_json_can_fallback_to_assistant_message(self) -> None: + stream = json.dumps( + { + "type": "assistant", + "message": {"role": "assistant", "content": [{"type": "text", "text": json.dumps(FINAL_REPORT)}]}, + } + ) + self.assertEqual(AUTOREVIEW.extract_json(stream), FINAL_REPORT) + + def test_extract_json_does_not_fallback_past_bad_terminal_result(self) -> None: + stream = "\n".join( + [ + json.dumps( + { + "type": "assistant", + "message": {"role": "assistant", "content": [{"type": "text", "text": json.dumps(FINAL_REPORT)}]}, + } + ), + json.dumps( + { + "type": "result", + "subtype": "success", + "result": "not json", + } + ), + ] + ) + with self.assertRaises(SystemExit) as exc_info: + AUTOREVIEW.extract_json(stream) + self.assertIn("review engine result was not structured JSON", str(exc_info.exception)) + + +class AutoreviewSecretScannerTests(unittest.TestCase): + def test_boolean_declarations_are_not_credential_material(self) -> None: + secret_field = "is" + "Secret" + client_secret_field = "hasClient" + "Secret" + cases = ( + (f"val {secret_field}: Boolean? = null,", None), + (f"var {client_secret_field}: Boolean = false", None), + (f"abstract val {secret_field}: Boolean?", None), + (f"val {secret_field}: Boolean?", None), + (f"const {client_secret_field}: boolean = true;", "typescript"), + (f"declare const {client_secret_field}: boolean;", "typescript"), + (f"let {secret_field}: Bool? = nil", None), + (f"let {secret_field}: Bool?", None), + ) + + for content, javascript_dialect in cases: + with self.subTest(content=content): + self.assertFalse( + AUTOREVIEW.secret_text_risk( + content, + javascript_dialect=javascript_dialect, + ) + ) + + def test_boolean_and_null_literal_values_are_not_credentials(self) -> None: + cases = ( + ("is" + "Secret", "true"), + ("requires" + "Password", "false"), + ("access" + "Token", "null"), + ) + for field_name, literal in cases: + content = f"{field_name} = {literal}" + with self.subTest(content=content): + self.assertFalse(AUTOREVIEW.secret_text_risk(content)) + + def test_boolean_annotation_does_not_hide_real_credential_literal(self) -> None: + literal_value = "actual-production-" + "secret" + secret_field = "is" + "Secret" + client_secret_field = "hasClient" + "Secret" + cases = ( + (f'val {secret_field}: Boolean? = "{literal_value}",', None), + (f'var {client_secret_field}: Boolean = "{literal_value}"', None), + ( + f'const {client_secret_field}: boolean = "{literal_value}";', + "typescript", + ), + (f'let {secret_field}: Bool? = "{literal_value}"', None), + ) + + for content, javascript_dialect in cases: + with self.subTest(content=content): + self.assertTrue( + AUTOREVIEW.secret_text_risk( + content, + javascript_dialect=javascript_dialect, + ) + ) + + def test_boolean_prefix_values_remain_credentials(self) -> None: + field_name = "client" + "Secret" + for prefix in ("Boolean", "boolean", "Bool"): + literal_value = prefix + "-prod-credential" + content = f"{field_name}: {literal_value}" + with self.subTest(content=content): + self.assertTrue(AUTOREVIEW.secret_text_risk(content)) + + def test_boolean_type_tokens_in_config_remain_credentials(self) -> None: + field_name = "client" + "Secret" + for literal_value in ("Boolean?", "Boolean?=abc1234"): + content = f"{field_name}: {literal_value}" + with self.subTest(content=content): + self.assertTrue(AUTOREVIEW.secret_text_risk(content)) + + +class AutoreviewCompatibilityTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.home_dir = tempfile.TemporaryDirectory(prefix="autoreview-test-home.") + cls.home_patch = mock.patch.object(Path, "home", return_value=Path(cls.home_dir.name)) + cls.home_patch.start() + cls.home_keys = ("HOME", "USERPROFILE", "HOMEDRIVE", "HOMEPATH") + cls.old_home_env = {key: os.environ.get(key) for key in cls.home_keys} + os.environ["HOME"] = cls.home_dir.name + os.environ["USERPROFILE"] = cls.home_dir.name + os.environ.pop("HOMEDRIVE", None) + os.environ.pop("HOMEPATH", None) + + @classmethod + def tearDownClass(cls) -> None: + cls.home_patch.stop() + for key, value in cls.old_home_env.items(): + if value is None: + os.environ.pop(key, None) + else: + os.environ[key] = value + cls.home_dir.cleanup() + + def test_harness_rejects_disabled_cursor_engine(self) -> None: + harness_path = SCRIPT_PATH.with_name("test-review-harness.py") + namespace = runpy.run_path(str(harness_path)) + with self.assertRaises(SystemExit): + namespace["parse_args"](["--engine", "cursor"]) + + def test_cursor_agent_bin_cli_alias(self) -> None: + with mock.patch.object( + sys, + "argv", + ["autoreview", "--cursor-agent-bin", "/tmp/legacy-cursor"], + ): + args = AUTOREVIEW.parse_args() + self.assertEqual(args.cursor_bin, "/tmp/legacy-cursor") + + def test_cursor_agent_bin_env_alias(self) -> None: + with mock.patch.dict( + os.environ, + {"CURSOR_AGENT_BIN": "/tmp/legacy-cursor"}, + clear=False, + ): + os.environ.pop("CURSOR_BIN", None) + with mock.patch.object(sys, "argv", ["autoreview"]): + args = AUTOREVIEW.parse_args() + self.assertEqual(args.cursor_bin, "/tmp/legacy-cursor") + + def test_cursor_agent_reviewer_alias_normalizes_to_cursor(self) -> None: + self.assertEqual( + AUTOREVIEW.parse_reviewer_token("cursor-agent:auto"), + ("cursor", "auto", None), + ) + + def test_cursor_agent_keyed_option_normalizes_to_cursor(self) -> None: + self.assertEqual( + AUTOREVIEW.parse_keyed_options(["cursor-agent=auto"], "model"), + (None, {"cursor": "auto"}), + ) + + def test_codex_config_status_exposes_keys_only(self) -> None: + args = argparse.Namespace(codex_config=['model_verbosity="low"']) + self.assertEqual(AUTOREVIEW.codex_config_keys(args), ["model_verbosity"]) + + def test_codex_retries_terra_after_sol_access_failure(self) -> None: + args = argparse.Namespace( + codex_bin="codex", + codex_config=None, + codex_speed=None, + fallback_model="gpt-5.6-terra", + model="gpt-5.6-sol", + stream_engine_output=False, + thinking="high", + tools=True, + web_search=False, + ) + models: list[str] = [] + + def fake_run(command: list[str], *_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]: + model = command[command.index("--model") + 1] + models.append(model) + if model == "gpt-5.6-sol": + return subprocess.CompletedProcess( + command, + 1, + "", + "The model `gpt-5.6-sol` does not exist or you do not have access to it.", + ) + output_path = Path(command[command.index("--output-last-message") + 1]) + output_path.write_text(json.dumps(FINAL_REPORT)) + return subprocess.CompletedProcess(command, 0, "", "") + + with tempfile.TemporaryDirectory(prefix="autoreview-codex-fallback.") as tmpdir, mock.patch.object( + AUTOREVIEW, + "resolve_command", + return_value="/usr/bin/codex", + ), mock.patch.object(AUTOREVIEW, "codex_auth_config_flags", return_value=[]), mock.patch.object( + AUTOREVIEW, + "prepare_codex_runtime_auth", + return_value=None, + ), mock.patch.object( + AUTOREVIEW, + "run_with_heartbeat", + side_effect=fake_run, + ): + output = AUTOREVIEW.run_codex(args, Path(tmpdir), "review") + + self.assertEqual(json.loads(output), FINAL_REPORT) + self.assertEqual(models, ["gpt-5.6-sol", "gpt-5.6-terra"]) + + def test_codex_runs_outside_repo_with_bundle_only_workspace(self) -> None: + args = argparse.Namespace( + codex_bin="codex", + codex_config=None, + codex_speed=None, + fallback_model=None, + model="gpt-5.6-sol", + stream_engine_output=False, + thinking="high", + tools=True, + web_search=False, + ) + observed: dict[str, object] = {} + + def fake_run( + command: list[str], + cwd: Path, + *_args: object, + **kwargs: object, + ) -> subprocess.CompletedProcess[str]: + observed["cwd"] = cwd + observed["command"] = command + observed["command_cwd"] = Path(command[command.index("-C") + 1]) + observed["workspace_entries"] = list(cwd.iterdir()) + observed["env"] = kwargs["env"] + output_path = Path(command[command.index("--output-last-message") + 1]) + output_path.write_text(json.dumps(FINAL_REPORT)) + return subprocess.CompletedProcess(command, 0, "", "") + + with tempfile.TemporaryDirectory(prefix="autoreview-codex-workspace-test.") as tmpdir: + repo = Path(tmpdir) + (repo / ".env").write_text("OPENAI_API_KEY=ignored-secret\n") + with mock.patch.dict( + os.environ, + {"CODEX_HOME": ""}, + clear=False, + ), mock.patch.object( + AUTOREVIEW, + "resolve_command", + return_value="/usr/bin/codex", + ), mock.patch.object( + AUTOREVIEW, + "codex_auth_config_flags", + return_value=[], + ), mock.patch.object( + AUTOREVIEW, + "prepare_codex_runtime_auth", + return_value=None, + ), mock.patch.object( + AUTOREVIEW, + "codex_source_home", + return_value=None, + ), mock.patch.object( + AUTOREVIEW, + "run_with_heartbeat", + side_effect=fake_run, + ): + output = AUTOREVIEW.run_codex(args, repo, "review") + + self.assertEqual(json.loads(output), FINAL_REPORT) + observed_cwd = observed["cwd"] + command_cwd = observed["command_cwd"] + self.assertIsInstance(observed_cwd, Path) + self.assertIsInstance(command_cwd, Path) + assert isinstance(observed_cwd, Path) + assert isinstance(command_cwd, Path) + self.assertNotEqual(observed_cwd.resolve(), repo.resolve()) + self.assertEqual(observed_cwd, command_cwd) + self.assertEqual(observed["workspace_entries"], []) + env = observed["env"] + self.assertIsInstance(env, dict) + assert isinstance(env, dict) + self.assertNotEqual(env["HOME"], os.environ.get("HOME")) + self.assertEqual(env["USERPROFILE"], env["HOME"]) + self.assertNotEqual(env.get("CODEX_HOME"), str(repo.resolve())) + self.assertEqual(Path(env["CODEX_HOME"]).name, "codex-home") + self.assertNotEqual(env["CODEX_HOME"], str((Path.home() / ".codex").resolve())) + self.assertIn("features.shell_snapshot=false", observed["command"]) + self.assertIn("features.hooks=false", observed["command"]) + self.assertIn("features.plugins=false", observed["command"]) + self.assertIn("skills.include_instructions=false", observed["command"]) + + def test_codex_does_not_fallback_after_unrelated_failure(self) -> None: + args = argparse.Namespace( + codex_bin="codex", + codex_config=None, + codex_speed=None, + fallback_model="gpt-5.6-terra", + model="gpt-5.6-sol", + stream_engine_output=False, + thinking="high", + tools=True, + web_search=False, + ) + models: list[str] = [] + + def fake_run(command: list[str], *_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]: + models.append(command[command.index("--model") + 1]) + return subprocess.CompletedProcess(command, 1, "", "network timeout") + + with tempfile.TemporaryDirectory(prefix="autoreview-codex-fallback.") as tmpdir, mock.patch.object( + AUTOREVIEW, + "resolve_command", + return_value="/usr/bin/codex", + ), mock.patch.object(AUTOREVIEW, "codex_auth_config_flags", return_value=[]), mock.patch.object( + AUTOREVIEW, + "prepare_codex_runtime_auth", + return_value=None, + ), mock.patch.object( + AUTOREVIEW, + "run_with_heartbeat", + side_effect=fake_run, + ): + with self.assertRaisesRegex(SystemExit, "network timeout"): + AUTOREVIEW.run_codex(args, Path(tmpdir), "review") + + self.assertEqual(models, ["gpt-5.6-sol"]) + + def test_codex_does_not_fallback_after_model_capacity_failure(self) -> None: + args = argparse.Namespace( + codex_bin="codex", + codex_config=None, + codex_speed=None, + fallback_model="gpt-5.6-terra", + model="gpt-5.6-sol", + stream_engine_output=False, + thinking="high", + tools=True, + web_search=False, + ) + models: list[str] = [] + + def fake_run(command: list[str], *_args: object, **_kwargs: object) -> subprocess.CompletedProcess[str]: + models.append(command[command.index("--model") + 1]) + return subprocess.CompletedProcess( + command, + 1, + "", + "model_not_available: gpt-5.6-sol is temporarily unavailable due to capacity", + ) + + with tempfile.TemporaryDirectory(prefix="autoreview-codex-fallback.") as tmpdir, mock.patch.object( + AUTOREVIEW, + "resolve_command", + return_value="/usr/bin/codex", + ), mock.patch.object(AUTOREVIEW, "codex_auth_config_flags", return_value=[]), mock.patch.object( + AUTOREVIEW, + "prepare_codex_runtime_auth", + return_value=None, + ), mock.patch.object( + AUTOREVIEW, + "run_with_heartbeat", + side_effect=fake_run, + ): + with self.assertRaisesRegex(SystemExit, "temporarily unavailable"): + AUTOREVIEW.run_codex(args, Path(tmpdir), "review") + + self.assertEqual(models, ["gpt-5.6-sol"]) + + def test_codex_access_fallback_ignores_structured_output_text(self) -> None: + result = subprocess.CompletedProcess( + ["codex"], + 1, + '{"type":"agent_message","text":"gpt-5.6-sol does not exist or you do not have access"}', + '{"type":"agent_message","message":"gpt-5.6-sol does not exist or you do not have access"}', + ) + + self.assertFalse( + AUTOREVIEW.codex_model_access_failure(result, "gpt-5.6-sol") + ) + + def test_codex_access_fallback_accepts_terminal_error_event(self) -> None: + result = subprocess.CompletedProcess( + ["codex"], + 1, + '{"type":"error","message":"gpt-5.6-sol does not exist or you do not have access"}', + "", + ) + + self.assertTrue( + AUTOREVIEW.codex_model_access_failure(result, "gpt-5.6-sol") + ) + + def test_codex_access_fallback_accepts_account_model_list_error(self) -> None: + result = subprocess.CompletedProcess( + ["codex"], + 1, + "", + ( + "The model gpt-5.6-sol does not appear in the list of models " + "available to your account" + ), + ) + + self.assertTrue( + AUTOREVIEW.codex_model_access_failure(result, "gpt-5.6-sol") + ) + + def test_codex_access_fallback_ignores_plain_stdout(self) -> None: + message = "gpt-5.6-sol does not exist or you do not have access" + stdout_result = subprocess.CompletedProcess(["codex"], 1, message, "") + stderr_result = subprocess.CompletedProcess(["codex"], 1, "", message) + + self.assertFalse( + AUTOREVIEW.codex_model_access_failure(stdout_result, "gpt-5.6-sol") + ) + self.assertTrue( + AUTOREVIEW.codex_model_access_failure(stderr_result, "gpt-5.6-sol") + ) + + def test_extract_json_accepts_dict_result_payload(self) -> None: + payload = { + "type": "result", + "subtype": "success", + "result": FINAL_REPORT, + "session_id": "session-id", + "request_id": "request-id", + } + self.assertEqual(AUTOREVIEW.extract_json(json.dumps(payload)), FINAL_REPORT) + + def test_extract_json_rejects_result_string_with_preamble(self) -> None: + payload = { + "type": "result", + "subtype": "success", + "result": "Inspecting the diff first.\n" + json.dumps(FINAL_REPORT), + } + with self.assertRaisesRegex(SystemExit, "result was not structured JSON"): + AUTOREVIEW.extract_json(json.dumps(payload)) + + def test_retry_filter_only_matches_parse_failures(self) -> None: + self.assertTrue(AUTOREVIEW.is_structured_output_failure("review engine returned non-JSON output: nope")) + self.assertTrue(AUTOREVIEW.is_structured_output_failure("review engine result was not structured JSON:\nnope")) + self.assertFalse(AUTOREVIEW.is_structured_output_failure("review JSON missing required key: findings")) + self.assertFalse(AUTOREVIEW.is_structured_output_failure("finding 0 has invalid priority")) + + def test_cursor_workspace_instructions_fail_closed(self) -> None: + with tempfile.TemporaryDirectory(prefix="autoreview-cursor-test.") as tmpdir: + repo = Path(tmpdir) + args = argparse.Namespace( + thinking=None, + tools=True, + web_search=True, + cursor_allow_workspace_instructions=False, + cursor_bin="cursor-agent", + model="auto", + stream_engine_output=False, + ) + with self.assertRaises(SystemExit) as exc_info: + AUTOREVIEW.run_cursor(args, repo, "prompt") + self.assertIn("cursor engine is unavailable", str(exc_info.exception)) + + def test_cursor_local_mcp_requires_explicit_approval(self) -> None: + with tempfile.TemporaryDirectory(prefix="autoreview-cursor-test.") as tmpdir: + repo = Path(tmpdir) + (repo / ".cursor").mkdir() + (repo / ".cursor" / "mcp.json").write_text("{}\n") + args = argparse.Namespace( + thinking=None, + tools=True, + web_search=True, + cursor_allow_workspace_instructions=True, + cursor_bin="cursor-agent", + model="auto", + stream_engine_output=False, + ) + with self.assertRaises(SystemExit) as exc_info: + AUTOREVIEW.run_cursor(args, repo, "prompt") + self.assertIn("cursor engine is unavailable", str(exc_info.exception)) + + def test_cursor_local_hooks_are_always_refused(self) -> None: + with tempfile.TemporaryDirectory(prefix="autoreview-cursor-test.") as tmpdir: + repo = Path(tmpdir) + (repo / ".cursor").mkdir() + (repo / ".cursor" / "hooks.json").write_text("{}\n") + args = argparse.Namespace( + thinking=None, + tools=True, + web_search=True, + cursor_allow_workspace_instructions=True, + cursor_bin="cursor-agent", + model="auto", + stream_engine_output=False, + ) + with self.assertRaises(SystemExit) as exc_info: + AUTOREVIEW.run_cursor(args, repo, "prompt") + self.assertIn("cursor engine is unavailable", str(exc_info.exception)) + + def test_cursor_local_permissions_are_always_refused(self) -> None: + with tempfile.TemporaryDirectory(prefix="autoreview-cursor-test.") as tmpdir: + repo = Path(tmpdir) + (repo / ".cursor").mkdir() + (repo / ".cursor" / "cli.json").write_text("{}\n") + args = argparse.Namespace( + thinking=None, + tools=True, + web_search=True, + cursor_allow_workspace_instructions=True, + cursor_bin="cursor-agent", + model="auto", + stream_engine_output=False, + ) + with self.assertRaises(SystemExit) as exc_info: + AUTOREVIEW.run_cursor(args, repo, "prompt") + self.assertIn("cursor engine is unavailable", str(exc_info.exception)) + + def test_cursor_is_disabled_without_repo_only_read_sandbox(self) -> None: + with tempfile.TemporaryDirectory(prefix="autoreview-cursor-test.") as tmpdir: + root = Path(tmpdir) + repo = root / "repo" + repo.mkdir() + cursor_bin = root / "cursor-agent" + AUTOREVIEW.write_executable(cursor_bin, AUTOREVIEW.fake_cursor_script()) + args = argparse.Namespace( + thinking=None, + tools=True, + web_search=True, + cursor_allow_workspace_instructions=True, + cursor_bin=str(cursor_bin), + model=None, + stream_engine_output=False, + ) + with mock.patch.object(AUTOREVIEW, "cursor_global_hook_paths", return_value=[]): + with self.assertRaisesRegex(SystemExit, "Cursor read permissions"): + AUTOREVIEW.run_cursor(args, repo, "prompt") + + def test_cursor_engine_fails_closed_end_to_end(self) -> None: + with tempfile.TemporaryDirectory(prefix="autoreview-cursor-e2e.") as tmpdir: + root = Path(tmpdir) + repo = root / "repo" + repo.mkdir() + subprocess.run(["git", "init", "--quiet"], cwd=repo, check=True) + subprocess.run(["git", "config", "user.name", "AutoReview Test"], cwd=repo, check=True) + subprocess.run(["git", "config", "user.email", "autoreview@example.invalid"], cwd=repo, check=True) + source = repo / "example.txt" + source.write_text("before\n") + subprocess.run(["git", "add", "example.txt"], cwd=repo, check=True) + subprocess.run(["git", "commit", "--quiet", "-m", "test: seed fixture"], cwd=repo, check=True) + source.write_text("after\n") + + cursor_bin = root / "cursor-agent" + trufflehog_bin = root / "trufflehog" + record_path = root / "record.json" + AUTOREVIEW.write_executable(cursor_bin, AUTOREVIEW.fake_cursor_script()) + AUTOREVIEW.write_executable( + trufflehog_bin, + "#!/usr/bin/env python3\nraise SystemExit(0)\n", + ) + env = os.environ.copy() + env.update( + { + "AUTOREVIEW_FAKE_RECORD": str(record_path), + "AUTOREVIEW_FAKE_CURSOR_INVOCATIONS": str(root / "cursor-invocations.jsonl"), + "GIT_CONFIG_GLOBAL": str(root / "hostile-gitconfig"), + "NODE_OPTIONS": "--require=hostile.js", + "PYTHONPATH": str(root / "hostile-python"), + "PATH": ( + f"{root}{os.pathsep}{repo}{os.pathsep}" + f"{env.get('PATH', '')}" + ), + "HOME": str(root), + "USERPROFILE": str(root), + } + ) + result = subprocess.run( + [ + sys.executable, + str(SCRIPT_PATH), + "--mode", + "local", + "--engine", + "cursor", + "--cursor-bin", + str(cursor_bin), + "--cursor-allow-workspace-instructions", + ], + cwd=repo, + env=env, + text=True, + capture_output=True, + check=False, + ) + + self.assertNotEqual(result.returncode, 0) + self.assertIn("Cursor read permissions", result.stderr) + self.assertFalse(record_path.exists()) + + +if __name__ == "__main__": + unittest.main() diff --git a/skills/autoreview/scripts/test-review-harness.ps1 b/skills/autoreview/scripts/test-review-harness.ps1 index c74ff04..4b859ca 100644 --- a/skills/autoreview/scripts/test-review-harness.ps1 +++ b/skills/autoreview/scripts/test-review-harness.ps1 @@ -3,7 +3,7 @@ param( [ValidateSet('malicious', 'benign')] [string] $Fixture, - [ValidateSet('codex', 'claude', 'droid', 'copilot', 'pi', 'opencode')] + [ValidateSet('codex', 'claude', 'pi')] [string[]] $Engine, [Alias('h')] diff --git a/skills/autoreview/scripts/test-review-harness.py b/skills/autoreview/scripts/test-review-harness.py index cb76eb6..5077d40 100644 --- a/skills/autoreview/scripts/test-review-harness.py +++ b/skills/autoreview/scripts/test-review-harness.py @@ -3,6 +3,7 @@ import argparse import os +import runpy import shutil import stat import subprocess @@ -12,8 +13,8 @@ from pathlib import Path -ENGINES = ("codex", "claude", "droid", "copilot", "pi", "opencode") -DEFAULT_ENGINES = ("codex",) +ENGINES = ("codex", "claude", "pi") +DEFAULT_ENGINES = ("codex", "claude") MALICIOUS_INITIAL = """export function uploadPath(name) { return `uploads/${name.replaceAll("/", "")}`; @@ -118,7 +119,7 @@ def parse_args(argv: list[str]) -> argparse.Namespace: "or a security-sensitive-but-safe patch, then verifies each selected " "engine through autoreview." ), - epilog="Default engine: codex.", + epilog="Default engines: codex, claude.", ) parser.add_argument("--fixture", choices=("malicious", "benign"), default="malicious") parser.add_argument("--engine", action="append", choices=ENGINES, dest="engines") @@ -145,8 +146,23 @@ def create_fixture_repo(repo: Path, fixture: str) -> None: write_fixture_file(repo, MALICIOUS_CHANGED if fixture == "malicious" else BENIGN_CHANGED) +def validate_prompt_policy(repo: Path, autoreview: Path) -> None: + namespace = runpy.run_path(str(autoreview)) + prompt = namespace["build_prompt"](repo, "local", None, "fixture diff", "", "") + required = ( + "This helper is a closeout gate.", + "Do not turn a narrow patch into a broad", + "If this is release-branch or release-process work", + "Non-blocking design,", + ) + missing = [needle for needle in required if needle not in prompt] + if missing: + raise RuntimeError(f"autoreview prompt missing scope policy: {missing}") + + def run_reviews(repo: Path, script_dir: Path, fixture: str, engines: list[str]) -> None: autoreview = script_dir / "autoreview" + validate_prompt_policy(repo, autoreview) for engine in engines: print(f"== {engine} ==", flush=True) command = [ diff --git a/skills/autoreview/tests/fixtures/typescript-benign-config-path-references.ts b/skills/autoreview/tests/fixtures/typescript-benign-config-path-references.ts new file mode 100644 index 0000000..1b45f94 --- /dev/null +++ b/skills/autoreview/tests/fixtures/typescript-benign-config-path-references.ts @@ -0,0 +1,30 @@ +declare const accountId: string; +declare const filePath: string; +declare const secretRef: string; +declare const tryReadSecretFileSync: (...args: unknown[]) => string; +declare const normalizeResolvedSecretInputString: (options: unknown) => string; + +export const passwordFile = tryReadSecretFileSync(filePath, "IRC password file", { + credentialDiagnostic: { + configPath: `channels.irc.accounts.${accountId}.passwordFile`, + }, +}); +export const nickservFile = tryReadSecretFileSync(filePath, "IRC NickServ password file", { + credentialDiagnostic: { + configPath: `channels.irc.accounts.${accountId}.nickserv.passwordFile`, + }, +}); +export const botSecret = normalizeResolvedSecretInputString({ + value: secretRef, + path: `channels.nextcloud-talk.accounts.${accountId}.botSecret`, +}); +export const botSecretFile = tryReadSecretFileSync(filePath, "Nextcloud bot secret file", { + credentialDiagnostic: { + configPath: `channels.nextcloud-talk.accounts.${accountId}.botSecretFile`, + }, +}); +export const tokenFile = tryReadSecretFileSync( + filePath, + `channels.telegram.accounts.${accountId}.tokenFile`, + { rejectSymlink: true }, +); diff --git a/skills/autoreview/tests/fixtures/typescript-benign-references.ts b/skills/autoreview/tests/fixtures/typescript-benign-references.ts new file mode 100644 index 0000000..924bdd7 --- /dev/null +++ b/skills/autoreview/tests/fixtures/typescript-benign-references.ts @@ -0,0 +1,55 @@ +type SecretRef = { source: "env"; id: string }; +type CredentialUnavailableDiagnostic = { path: string; reason: string }; + +declare const tokenRef: SecretRef; +declare const keyRef: SecretRef; +declare const inlinePassword: string; +declare const inlineSecret: string; +declare const accountFileToken: string; +declare const baseFileToken: string; +declare const passwordResolution: { password: string }; +declare const secretResolution: { secret: string }; +declare const tokenResolution: { token: string }; +declare const accountTokenFile: { token: string }; +declare const channelTokenFile: { token: string }; +declare const merged: { apiPassword: string; passwordFile: string }; +declare const tryReadSecretFileSync: (...args: unknown[]) => string; +declare const normalizeResolvedSecretInputString: (options: unknown) => string; +declare const resolveToken: (options: unknown) => { value: string }; + +const filePassword = tryReadSecretFileSync(merged.passwordFile, "IRC password file", { + credentialDiagnostic: { + configPath: `channels.irc.accounts.${accountId}.passwordFile`, + report: (diagnostic: CredentialUnavailableDiagnostic) => diagnostic, + }, +}); +const configPassword = normalizeResolvedSecretInputString({ + value: merged.apiPassword, + path: "channels.nextcloud-talk.apiPassword", +}); +const token = resolveToken({ accountId }); +const priorPasswordFileError = /IRC password file.*must not be a symlink/; + +export type CredentialPlumbing = { + tokenRef?: SecretRef; + keyRef?: SecretRef; + credentialDiagnostics?: CredentialUnavailableDiagnostic[]; +}; + +export const resolvedCredentialPlumbing = { + token: tokenRef, + apiKey: keyRef, + password: filePassword, + configPassword, + nextPassword: inlinePassword, + secret: inlineSecret, + accountToken: accountFileToken, + baseToken: baseFileToken, + resolvedPassword: passwordResolution.password, + resolvedSecret: secretResolution.secret, + resolvedToken: tokenResolution.token, + accountTokenFile: accountTokenFile.token, + channelTokenFile: channelTokenFile.token, + apiPassword: merged.apiPassword, + channelAccessToken: token.value, +}; diff --git a/skills/autoreview/tests/fixtures/typescript-sensitive-literals.ts b/skills/autoreview/tests/fixtures/typescript-sensitive-literals.ts new file mode 100644 index 0000000..1fb677b --- /dev/null +++ b/skills/autoreview/tests/fixtures/typescript-sensitive-literals.ts @@ -0,0 +1,10 @@ +const password = "FAKE-CorrectHorseBattery-Staple-2026!"; +const credential = "FAKE_A7f9K2m4Q8v6N3x5R1p0T9z8"; +const apiKey = "sk-proj-FAKE00000000000000000000000000000000000000000000"; +const githubToken = "ghp_FAKE000000000000000000000000000000"; +const awsAccessKey = "AKIAFAKE000000000000"; +const slackToken = "xoxb-FAKE000000000-FAKE000000000-FAKE000000000000000000000000"; +const authorization = "Bearer eyJhbGciOiJIUzI1NiJ9.RkFLRS1OT1QtQS1SRUFM.TOKENFAKESIGNATURE"; +const resolvedToken = resolveToken({ value: "FAKE_B8g0L3n5R9w7P4y6S2q1U0a9" }); +const filePassword = tryReadSecretFileSync(path, "FAKE-A7f9K2m4Q8v6N3x5R1p0T9z8"); +const password = readPassword("alice", "FAKE correct horse secret battery 2026"); diff --git a/skills/autoreview/tests/test_autoreview_hardening.py b/skills/autoreview/tests/test_autoreview_hardening.py new file mode 100644 index 0000000..d0863a5 --- /dev/null +++ b/skills/autoreview/tests/test_autoreview_hardening.py @@ -0,0 +1,6938 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import contextlib +import io +import json +import os +import re +import runpy +import shutil +import stat +import subprocess +import sys +import tempfile +import threading +import time +import unittest +from unittest import mock +from pathlib import Path + + +SCRIPT = Path(__file__).resolve().parents[1] / "scripts" / "autoreview" +FIXTURES = Path(__file__).with_name("fixtures") +PRIVATE_KEY_BEGIN_TEXT = "BEGIN " + "PRIVATE KEY" +RSA_PRIVATE_KEY_BEGIN_TEXT = "BEGIN RSA " + "PRIVATE KEY" + + +def load_helper() -> dict[str, object]: + return runpy.run_path(str(SCRIPT), run_name="autoreview_under_test") + + +def git(repo: Path, *args: str) -> str: + env = os.environ.copy() + env.update( + { + "GIT_AUTHOR_NAME": "Autoreview Test", + "GIT_AUTHOR_EMAIL": "autoreview@example.invalid", + "GIT_COMMITTER_NAME": "Autoreview Test", + "GIT_COMMITTER_EMAIL": "autoreview@example.invalid", + } + ) + result = subprocess.run( + ["git", *args], + cwd=repo, + env=env, + check=True, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + return result.stdout + + +def init_repo(tempdir: Path) -> Path: + repo = tempdir / "repo" + repo.mkdir() + git(repo, "init", "-q") + git(repo, "config", "user.name", "Autoreview Test") + git(repo, "config", "user.email", "autoreview@example.invalid") + return repo + + +def realistic_secret_value() -> str: + return "A7f9K2m4Q8v6" + "N3x5R1p0T9z8" + + +def add_fake_trufflehog( + helper: dict[str, object], + root: Path, + env: dict[str, str], +) -> None: + helper["write_executable"]( + root / "trufflehog", + "#!/usr/bin/env python3\nraise SystemExit(0)\n", + ) + env["PATH"] = f"{root}{os.pathsep}{env.get('PATH', '')}" + + +class AutoreviewHardeningTests(unittest.TestCase): + def setUp(self) -> None: + self.helper = load_helper() + + def test_trufflehog_missing_binary_has_platform_neutral_guidance(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + with mock.patch.dict( + self.helper["run_trufflehog_preflight"].__globals__, + {"find_command": lambda _name, _repo: None}, + ): + with self.assertRaises(SystemExit) as error: + self.helper["run_trufflehog_preflight"]( + repo, + "local", + None, + "HEAD", + ) + + message = str(error.exception) + self.assertIn("TruffleHog is required but was not found", message) + self.assertIn(self.helper["TRUFFLEHOG_INSTALL_URL"], message) + self.assertNotIn("brew", message.casefold()) + + def test_trufflehog_scans_staged_and_working_versions_separately(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + source = repo / "runtime.txt" + source.write_text("base\n", encoding="utf-8") + git(repo, "add", "runtime.txt") + git(repo, "commit", "-q", "-m", "base") + source.write_text("staged version\n", encoding="utf-8") + git(repo, "add", "runtime.txt") + source.write_text("working version\n", encoding="utf-8") + + original_find_command = self.helper["find_command"] + original_run = self.helper["run"] + scanned: dict[str, str] = {} + + def find_command(name: str, checkout: Path) -> str | None: + if name == "trufflehog": + return "/trusted/trufflehog" + return original_find_command(name, checkout) + + def run_scanner( + command: list[str], + cwd: Path, + **_kwargs: object, + ) -> subprocess.CompletedProcess[str]: + if command[0] != "/trusted/trufflehog": + return original_run(command, cwd, **_kwargs) + self.assertEqual( + command[0:2], + [ + "/trusted/trufflehog", + "git", + ], + ) + self.assertEqual(command[3], "--since-commit") + self.assertEqual(command[5:7], ["--branch", "HEAD"]) + self.assertEqual( + command[7:], + [ + "--no-update", + "--no-color", + "--results=verified,unknown", + "--fail", + "--fail-on-scan-errors", + ], + ) + scan_path = command[2].removeprefix("file://") + if os.name == "nt": + scan_path = scan_path.lstrip("/") + scan_repo = Path(scan_path) + commits = git( + scan_repo, + "log", + "--reverse", + "--format=%H", + ).splitlines() + scanned["staged"] = git( + scan_repo, + "show", + f"{commits[1]}:runtime.txt", + ) + scanned["working"] = git( + scan_repo, + "show", + f"{commits[2]}:runtime.txt", + ) + return subprocess.CompletedProcess(command, 0, "", "") + + with mock.patch.dict( + self.helper["run_trufflehog_preflight"].__globals__, + { + "find_command": find_command, + "run": run_scanner, + }, + ): + self.helper["run_trufflehog_preflight"]( + repo, + "local", + None, + "HEAD", + ) + + self.assertEqual( + scanned, + { + "staged": "staged version\n", + "working": "working version\n", + }, + ) + + def test_trufflehog_scans_only_changed_content_at_reviewed_ref(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + unchanged = repo / "unchanged.txt" + changed = repo / "changed.txt" + unchanged.write_text("unchanged\n", encoding="utf-8") + changed.write_text("base\n", encoding="utf-8") + git(repo, "add", "unchanged.txt", "changed.txt") + git(repo, "commit", "-q", "-m", "base") + base = git(repo, "rev-parse", "HEAD").strip() + changed.write_text("reviewed version\n", encoding="utf-8") + git(repo, "add", "changed.txt") + git(repo, "commit", "-q", "-m", "change") + reviewed_commit = git(repo, "rev-parse", "HEAD").strip() + changed.write_text("later working version\n", encoding="utf-8") + + for target, target_ref, commit_ref in ( + ("branch", base, "HEAD"), + ("commit", None, reviewed_commit), + ): + with self.subTest(target=target), tempfile.TemporaryDirectory() as scan_dir: + scan_repo = Path(scan_dir) + base_commit = self.helper["prepare_trufflehog_history"]( + repo, + target, + target_ref, + commit_ref, + scan_repo, + ) + + commits = git( + scan_repo, + "log", + "--reverse", + "--format=%H", + ).splitlines() + self.assertEqual(commits[0], base_commit) + self.assertEqual(len(commits), 3) + self.assertEqual( + git( + scan_repo, + "show", + f"{commits[1]}:changed.txt", + ), + "reviewed version\n", + ) + with self.assertRaises(subprocess.CalledProcessError): + subprocess.run( + [ + "git", + "show", + f"{commits[1]}:unchanged.txt", + ], + cwd=scan_repo, + check=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + ) + + def test_trufflehog_history_scans_deleted_content_in_reverse_commit(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + source = repo / "removed.txt" + source.write_text("removed baseline content\n", encoding="utf-8") + git(repo, "add", "removed.txt") + git(repo, "commit", "-q", "-m", "base") + base = git(repo, "rev-parse", "HEAD").strip() + source.unlink() + git(repo, "add", "removed.txt") + git(repo, "commit", "-q", "-m", "remove credential") + + with tempfile.TemporaryDirectory() as scan_dir: + scan_repo = Path(scan_dir) + scan_base = self.helper["prepare_trufflehog_history"]( + repo, + "branch", + base, + "HEAD", + scan_repo, + ) + commits = git( + scan_repo, + "log", + "--reverse", + "--format=%H", + ).splitlines() + + self.assertEqual(commits[0], scan_base) + self.assertEqual(len(commits), 3) + self.assertEqual( + git(scan_repo, "show", f"{commits[2]}:removed.txt"), + "removed baseline content\n", + ) + with self.assertRaises(subprocess.CalledProcessError): + subprocess.run( + ["git", "show", f"{commits[1]}:removed.txt"], + cwd=scan_repo, + check=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + ) + + def test_trufflehog_snapshot_supports_directory_to_file_transition(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + nested = repo / "entry" / "nested.txt" + nested.parent.mkdir() + nested.write_text("nested\n", encoding="utf-8") + git(repo, "add", "entry/nested.txt") + git(repo, "commit", "-q", "-m", "directory") + base = git(repo, "rev-parse", "HEAD").strip() + nested.unlink() + nested.parent.rmdir() + (repo / "entry").write_text("file\n", encoding="utf-8") + git(repo, "add", "-A") + git(repo, "commit", "-q", "-m", "file") + + with tempfile.TemporaryDirectory() as scan_dir: + scan_repo = Path(scan_dir) + self.helper["prepare_trufflehog_history"]( + repo, + "branch", + base, + "HEAD", + scan_repo, + ) + commits = git( + scan_repo, + "log", + "--reverse", + "--format=%H", + ).splitlines() + self.assertEqual( + git(scan_repo, "show", f"{commits[1]}:entry"), + "file\n", + ) + + @unittest.skipIf(os.name == "nt", "Windows filenames cannot use Git pathspec magic prefixes") + def test_trufflehog_snapshot_treats_git_paths_as_literals(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + rel = ":(exclude).txt" + source = repo / rel + source.write_text("base\n", encoding="utf-8") + git(repo, "--literal-pathspecs", "add", "--", rel) + git(repo, "commit", "-q", "-m", "base") + source.write_text("staged\n", encoding="utf-8") + git(repo, "--literal-pathspecs", "add", "--", rel) + + with tempfile.TemporaryDirectory() as index_dir: + index_root = Path(index_dir) + self.helper["materialize_index_snapshot"]( + repo, + index_root, + [rel], + ) + self.assertEqual( + (index_root / rel).read_text(encoding="utf-8"), + "staged\n", + ) + + git(repo, "commit", "-q", "-m", "staged") + with tempfile.TemporaryDirectory() as tree_dir: + tree_root = Path(tree_dir) + self.helper["materialize_tree_snapshot"]( + repo, + tree_root, + "HEAD", + [rel], + ) + self.assertEqual( + (tree_root / rel).read_text(encoding="utf-8"), + "staged\n", + ) + + def test_trufflehog_snapshot_force_stages_ignored_materialized_files(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + (repo / ".gitignore").write_text("ignored.txt\n", encoding="utf-8") + (repo / "ignored.txt").write_text("review me\n", encoding="utf-8") + + commit = self.helper["commit_snapshot"](repo, "snapshot") + + self.assertEqual( + git(repo, "show", f"{commit}:ignored.txt"), + "review me\n", + ) + + def test_trufflehog_snapshot_rejects_symlinked_parent_directories(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + outside = root / "outside" + outside.mkdir() + (outside / "review.txt").write_text("outside\n", encoding="utf-8") + parent = repo / "nested" + try: + parent.symlink_to(outside, target_is_directory=True) + except OSError as exc: + if os.name == "nt" and getattr(exc, "winerror", None) == 1314: + self.skipTest("Windows symlink privilege is not available") + raise + + with tempfile.TemporaryDirectory() as snapshot_dir: + with self.assertRaisesRegex(SystemExit, "symlinked parent"): + self.helper["copy_worktree_file"]( + repo, + Path(snapshot_dir), + "nested/review.txt", + ) + + def test_local_trufflehog_snapshot_supports_path_type_transitions(self) -> None: + for transition in ("directory-to-file", "file-to-directory"): + with self.subTest(transition=transition), tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + entry = repo / "entry" + nested = entry / "nested.txt" + if transition == "directory-to-file": + entry.mkdir() + nested.write_text("nested\n", encoding="utf-8") + git(repo, "add", "entry/nested.txt") + else: + entry.write_text("file\n", encoding="utf-8") + git(repo, "add", "entry") + git(repo, "commit", "-q", "-m", "base") + + if transition == "directory-to-file": + nested.unlink() + entry.rmdir() + entry.write_text("file\n", encoding="utf-8") + expected_path = "entry" + expected_content = "file\n" + else: + entry.unlink() + entry.mkdir() + nested.write_text("nested\n", encoding="utf-8") + expected_path = "entry/nested.txt" + expected_content = "nested\n" + + with tempfile.TemporaryDirectory() as scan_dir: + scan_repo = Path(scan_dir) + self.helper["prepare_trufflehog_history"]( + repo, + "local", + None, + "HEAD", + scan_repo, + ) + commits = git( + scan_repo, + "log", + "--reverse", + "--format=%H", + ).splitlines() + self.assertEqual( + git(scan_repo, "show", f"{commits[2]}:{expected_path}"), + expected_content, + ) + + def test_trufflehog_findings_and_errors_do_not_leak_scanner_output(self) -> None: + for returncode, expected in ( + ( + self.helper["TRUFFLEHOG_FINDINGS_EXIT_CODE"], + "found verified or unknown credentials", + ), + (1, "could not complete the credential scan"), + ): + with self.subTest(returncode=returncode), tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + (repo / "runtime.txt").write_text("review me\n", encoding="utf-8") + original_find_command = self.helper["find_command"] + original_run = self.helper["run"] + scanner_output = "detected-value-that-must-not-leak" + + def find_command(name: str, checkout: Path) -> str | None: + if name == "trufflehog": + return "/trusted/trufflehog" + return original_find_command(name, checkout) + + def run_scanner( + command: list[str], + cwd: Path, + **_kwargs: object, + ) -> subprocess.CompletedProcess[str]: + if command[0] != "/trusted/trufflehog": + return original_run(command, cwd, **_kwargs) + return subprocess.CompletedProcess( + command, + returncode, + scanner_output, + scanner_output, + ) + + output = io.StringIO() + with ( + mock.patch.dict( + self.helper["run_trufflehog_preflight"].__globals__, + { + "find_command": find_command, + "run": run_scanner, + }, + ), + contextlib.redirect_stdout(output), + contextlib.redirect_stderr(output), + self.assertRaises(SystemExit) as error, + ): + self.helper["run_trufflehog_preflight"]( + repo, + "local", + None, + "HEAD", + ) + + combined = output.getvalue() + str(error.exception) + self.assertIn(expected, combined) + self.assertNotIn(scanner_output, combined) + + def test_powershell_harness_exposes_runnable_engines_only(self) -> None: + harness = SCRIPT.with_name("test-review-harness.ps1").read_text(encoding="utf-8") + + self.assertIn("[ValidateSet('codex', 'claude', 'pi')]", harness) + for disabled_engine in ("droid", "copilot", "opencode", "cursor"): + self.assertNotIn(f"'{disabled_engine}'", harness) + + def test_local_bundle_omits_sensitive_untracked_file_without_blocking(self) -> None: + for rel in (".env", "tokens/session.dat", "secrets/local.py"): + with self.subTest(rel=rel), tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + path = repo / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("placeholder=true\n", encoding="utf-8") + (repo / "review.py").write_text("print('review me')\n", encoding="utf-8") + + bundle, truncated = self.helper["local_bundle"](repo) + + self.assertIn("# Review Input Redactions", bundle) + self.assertIn(self.helper["REVIEW_SECURITY_REDACTION"], bundle) + self.assertNotIn(rel, bundle) + self.assertNotIn("placeholder=true", bundle) + self.assertIn("print('review me')", bundle) + self.assertFalse(truncated) + + def test_local_bundle_marks_untracked_binary_input_incomplete(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + (repo / "image.bin").write_bytes(b"\x89PNG\r\n\0binary-content") + + bundle, truncated = self.helper["local_bundle"](repo) + + self.assertIn( + '# Untracked File\npath: "image.bin"\n' + 'source-line 1: "[binary file omitted]"', + bundle, + ) + self.assertTrue(truncated) + + def test_local_bundle_rejects_non_utf8_untracked_text(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + (repo / "latin.py").write_bytes(b"print('caf\xe9')\n") + + with self.assertRaisesRegex(SystemExit, "non-UTF-8 file"): + self.helper["local_bundle"](repo) + + def test_local_bundle_uses_validated_untracked_snapshot(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + (repo / "notes.txt").write_text("review me\n", encoding="utf-8") + original_read_prefix = self.helper["read_prefix"] + reads = 0 + + def read_once(path: Path, limit: int) -> tuple[bytes, bool]: + nonlocal reads + reads += 1 + if reads > 1: + raise AssertionError("untracked file was reopened after validation") + return original_read_prefix(path, limit) + + with mock.patch.dict( + self.helper["local_bundle"].__globals__, + {"read_prefix": read_once}, + ): + bundle, truncated = self.helper["local_bundle"](repo) + + expected_record = json.dumps("review me" + os.linesep) + self.assertIn( + '# Untracked File\npath: "notes.txt"\n' + f"source-line 1: {expected_record}", + bundle, + ) + self.assertFalse(truncated) + self.assertEqual(reads, 1) + + def test_tracked_binary_changes_are_blocked_in_all_modes(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + binary = repo / "artifact.bin" + binary.write_bytes(b"\0base") + git(repo, "add", "artifact.bin") + git(repo, "commit", "-q", "-m", "base") + base = git(repo, "rev-parse", "HEAD").strip() + + binary.write_bytes(b"\0changed") + git(repo, "add", "artifact.bin") + with self.assertRaisesRegex(SystemExit, "refusing binary changes"): + self.helper["local_bundle"](repo) + + git(repo, "commit", "-q", "-m", "binary change") + with self.assertRaisesRegex(SystemExit, "refusing binary changes"): + self.helper["commit_bundle"](repo, "HEAD") + with self.assertRaisesRegex(SystemExit, "refusing binary changes"): + self.helper["branch_bundle"](repo, base) + + def test_gitlink_changes_are_blocked_in_all_modes(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + tracked = repo / "tracked.txt" + tracked.write_text("base\n", encoding="utf-8") + git(repo, "add", "tracked.txt") + git(repo, "commit", "-q", "-m", "base") + base = git(repo, "rev-parse", "HEAD").strip() + + git( + repo, + "update-index", + "--add", + "--cacheinfo", + f"160000,{base},vendor/dependency", + ) + with self.assertRaisesRegex(SystemExit, "gitlink/submodule changes"): + self.helper["local_bundle"](repo) + + git(repo, "commit", "-q", "-m", "add gitlink") + with self.assertRaisesRegex(SystemExit, "gitlink/submodule changes"): + self.helper["commit_bundle"](repo, "HEAD") + with self.assertRaisesRegex(SystemExit, "gitlink/submodule changes"): + self.helper["branch_bundle"](repo, base) + + def test_gitlink_guard_parses_combined_raw_modes(self) -> None: + raw_diff = ( + "::100644 100644 160000 " + + ("a" * 40) + + " " + + ("b" * 40) + + " " + + ("c" * 40) + + " MM\0vendor/dependency\0" + ) + + with self.assertRaisesRegex(SystemExit, "gitlink/submodule changes"): + self.helper["require_no_gitlink_diff"]("merge diff", raw_diff) + + def test_codex_config_rejects_capability_bearing_overrides(self) -> None: + for override in ( + 'mcp_servers.review.command="touch /tmp/owned"', + 'notify=["sh", "-c", "touch /tmp/owned"]', + 'model_instructions_file="/tmp/hostile.md"', + 'model_provider="credential-sink"', + 'hooks.PreToolUse.command="touch /tmp/owned"', + ): + with self.subTest(override=override), self.assertRaisesRegex( + SystemExit, + "unsafe Codex config override refused", + ): + self.helper["codex_config_overrides"]( + argparse.Namespace(codex_config=[override]) + ) + + def test_codex_config_accepts_safe_tuning_overrides(self) -> None: + args = argparse.Namespace( + codex_config=[ + 'service_tier="fast"', + 'model_verbosity="low"', + 'model_reasoning_summary="concise"', + ] + ) + + self.assertEqual( + self.helper["codex_config_overrides"](args), + args.codex_config, + ) + + def test_untracked_files_respect_trusted_global_excludes(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + home = root / "home" + home.mkdir() + excludes = root / "global-ignore" + excludes.write_text( + "ignored.local\n!settings.local\n", + encoding="utf-8", + ) + (home / ".gitconfig").write_text( + f"[core]\n\texcludesFile = {excludes.as_posix()}\n", + encoding="utf-8", + ) + (repo / "ignored.local").write_text("private notes\n", encoding="utf-8") + (repo / ".gitignore").write_text("settings.local\n", encoding="utf-8") + (repo / "settings.local").write_text("repo private\n", encoding="utf-8") + git(repo, "add", ".gitignore") + (repo / "visible.txt").write_text("review me\n", encoding="utf-8") + (repo / "hostile-gitconfig").write_text( + "[core]\n\texcludesFile = /does/not/exist\n", + encoding="utf-8", + ) + + with mock.patch.dict( + os.environ, + { + "HOME": str(home), + "USERPROFILE": str(home), + "GIT_CONFIG_GLOBAL": str(repo / "hostile-gitconfig"), + }, + ): + self.assertEqual( + self.helper["safe_untracked_files"](repo), + ["hostile-gitconfig", "visible.txt"], + ) + + def test_dirty_check_respects_trusted_global_excludes(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + home = root / "home" + home.mkdir() + excludes = root / "global-ignore" + excludes.write_text("ignored.local\n", encoding="utf-8") + (home / ".gitconfig").write_text( + f"[core]\n\texcludesFile = {excludes.as_posix()}\n", + encoding="utf-8", + ) + (repo / "ignored.local").write_text("private notes\n", encoding="utf-8") + + with mock.patch.dict( + os.environ, + { + "HOME": str(home), + "USERPROFILE": str(home), + }, + ): + self.assertFalse(self.helper["is_dirty"](repo)) + + def test_oversized_text_is_rejected_without_scanning_binary_tail(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + tail_secret = "\ntoken=" + "A" * 24 + "\n" + content = "x" * (64_000 * 3 - 4) + tail_secret + + untracked = repo / "untracked.txt" + untracked.write_text(content, encoding="utf-8") + with self.assertRaisesRegex(SystemExit, "file too large to scan safely"): + self.helper["safe_untracked_files"](repo) + + untracked.unlink() + binary = repo / "binary.bin" + binary.write_bytes(b"\0" + content.encode()) + self.assertEqual( + self.helper["safe_untracked_files"](repo), + ["binary.bin"], + ) + + binary.unlink() + evidence = repo / "evidence.txt" + evidence.write_text(content, encoding="utf-8") + with self.assertRaisesRegex(SystemExit, "file too large to scan safely"): + self.helper["validate_evidence_file"](repo, "evidence.txt", "--dataset") + + def test_branch_bundle_rejects_unsafe_or_unknown_base_before_diff(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + (repo / "tracked.txt").write_text("base\n", encoding="utf-8") + git(repo, "add", "tracked.txt") + git(repo, "commit", "-q", "-m", "base") + + with self.assertRaisesRegex(SystemExit, "unsafe base ref"): + self.helper["branch_bundle"](repo, "--help") + with self.assertRaisesRegex(SystemExit, "unknown base ref"): + self.helper["branch_bundle"](repo, "origin/main") + + def test_commit_bundle_rejects_merge_commits(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + (repo / "base.txt").write_text("base\n", encoding="utf-8") + git(repo, "add", "base.txt") + git(repo, "commit", "-q", "-m", "base") + base_branch = git(repo, "branch", "--show-current").strip() + git(repo, "checkout", "-q", "-b", "side") + (repo / "side.txt").write_text("side\n", encoding="utf-8") + git(repo, "add", "side.txt") + git(repo, "commit", "-q", "-m", "side") + git(repo, "checkout", "-q", base_branch) + (repo / "main.txt").write_text("main\n", encoding="utf-8") + git(repo, "add", "main.txt") + git(repo, "commit", "-q", "-m", "main") + git(repo, "merge", "-q", "--no-ff", "side", "-m", "merge") + + with self.assertRaisesRegex(SystemExit, "does not accept merge commits"): + self.helper["commit_bundle"](repo, "HEAD") + + def test_git_path_list_preserves_newline_filenames(self) -> None: + if os.name == "nt": + self.skipTest("Windows filesystems do not support newline path components") + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + rel = "line\nbreak.txt" + (repo / rel).write_text("content\n", encoding="utf-8") + git(repo, "add", rel) + + paths = self.helper["git_path_list"](repo, "ls-files", "-z") + + self.assertIn(rel, paths) + + @unittest.skipUnless(sys.platform.startswith("linux"), "requires raw non-UTF-8 filename support") + def test_git_path_list_rejects_non_utf8_output(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + rel = os.fsdecode(b"invalid-\xff.txt") + (repo / rel).write_text("content\n", encoding="utf-8") + git(repo, "add", "--", rel) + + with self.assertRaisesRegex(SystemExit, "non-UTF-8 Git output"): + self.helper["git_path_list"](repo, "ls-files", "-z") + + def test_review_patch_rejects_oversized_content(self) -> None: + with self.assertRaisesRegex(SystemExit, "too large to review safely"): + self.helper["validate_review_patch"]("local staged diff", ["safe.txt"], "x" * 25, 10) + + def test_review_patch_limit_counts_utf8_bytes(self) -> None: + with self.assertRaisesRegex(SystemExit, r"12 bytes; limit 10"): + self.helper["validate_review_patch"]("local staged diff", ["safe.txt"], "界" * 4, 10) + + def test_review_patch_accepts_large_content_without_explicit_limit(self) -> None: + patch = ( + "diff --git a/safe.txt b/safe.txt\n" + "--- a/safe.txt\n" + "+++ b/safe.txt\n" + "@@ -0,0 +1,100000 @@\n" + + "+safe review content\n" * 100_000 + ) + + self.assertEqual( + self.helper["validate_review_patch"]( + "local staged diff", + ["safe.txt"], + patch, + ), + patch, + ) + + def test_review_bundle_chunking_preserves_every_byte_and_diff_context(self) -> None: + bundle = ( + "# Commit Diff\n\n" + "diff --git a/safe.txt b/safe.txt\n" + "--- a/safe.txt\n" + "+++ b/safe.txt\n" + "@@ -0,0 +1,200 @@\n" + + "+safe review content\n" * 200 + ) + + chunks = self.helper["split_review_bundle"](bundle, 300) + + self.assertGreater(len(chunks), 1) + self.assertEqual("".join(chunk.content for chunk in chunks), bundle) + self.assertTrue(all(len(chunk.content.encode("utf-8")) <= 300 for chunk in chunks)) + self.assertTrue( + any( + "+++ b/safe.txt" in chunk.context + and "@@ -0,0 +1,200 @@" in chunk.context + and "Continuation begins at new-file line" in chunk.context + for chunk in chunks[1:] + ) + ) + + def test_untracked_markdown_headings_do_not_create_bundle_boundaries(self) -> None: + bundle = ( + "# Untracked Files\n\n" + "# Untracked File\n" + 'path: "notes.md"\n' + 'source-line 1: "# title\\n"\n' + 'source-line 2: "## section\\n"\n\n' + "# Untracked File\n" + 'path: "todo.md"\n' + 'source-line 1: "# next\\n"' + ) + + units = self.helper["review_bundle_units"](bundle) + + self.assertEqual(len(units), 3) + self.assertIn(r'source-line 2: "## section\n"', units[1]) + self.assertEqual("".join(units), bundle) + + def test_unicode_line_separators_do_not_create_bundle_boundaries(self) -> None: + bundle = ( + "# Untracked Files\n\n" + "# Untracked File\n" + 'path: "notes.txt"\n' + 'source-line 1: "before\u2028diff --git a/fake b/fake"\n\n' + "diff --git a/real.txt b/real.txt\n" + "--- a/real.txt\n" + "+++ b/real.txt\n" + ) + + units = self.helper["review_bundle_units"](bundle) + + self.assertEqual(len(units), 3) + self.assertIn("\u2028diff --git a/fake b/fake", units[1]) + self.assertEqual("".join(units), bundle) + + def test_diff_source_prefixes_do_not_replace_file_context(self) -> None: + context: list[str] = [] + next_new_line = None + next_old_line = None + in_hunk = False + lines = ( + "diff --git a/safe.txt b/safe.txt\n", + "--- a/safe.txt\n", + "+++ b/safe.txt\n", + "@@ -10,2 +10,3 @@\n", + "+++ added source beginning with pluses\n", + "--- deleted source beginning with minuses\n", + " context\n", + ) + + for line in lines: + next_new_line, next_old_line, in_hunk = self.helper[ + "update_review_chunk_context" + ]( + context, + line, + next_new_line, + next_old_line, + in_hunk, + ) + + self.assertEqual(next_new_line, 12) + self.assertEqual(next_old_line, 12) + self.assertIn("--- a/safe.txt\n", context) + self.assertIn("+++ b/safe.txt\n", context) + self.assertNotIn("--- deleted source beginning with minuses\n", context) + + def test_hunk_header_that_fits_fresh_chunk_is_not_split(self) -> None: + unit = ( + "diff --git a/abcdefghijk b/abcdefghijk\n" + "--- a/abcdefghijk\n" + "+++ b/abcdefghijk\n" + "@@ -1 +1 @@\n" + "-old\n" + "+new\n" + ) + + chunks = self.helper["split_oversized_review_unit"](unit, 85) + + self.assertGreater(len(chunks), 1) + self.assertTrue(any("@@ -1 +1 @@\n" in chunk.content for chunk in chunks)) + self.assertEqual("".join(chunk.content for chunk in chunks), unit) + + def test_long_diff_line_continuations_keep_their_original_marker(self) -> None: + for marker in ("+", "-", " "): + with self.subTest(marker=marker): + unit = ( + "diff --git a/large.txt b/large.txt\n" + "--- a/large.txt\n" + "+++ b/large.txt\n" + "@@ -1 +1 @@\n" + f"{marker}{'x' * 400}\n" + ) + + chunks = self.helper["split_oversized_review_unit"](unit, 140) + + self.assertTrue( + any( + f"original marker is `{marker}`" in chunk.context + for chunk in chunks[1:] + ) + ) + self.assertEqual("".join(chunk.content for chunk in chunks), unit) + + def test_modified_file_deletion_context_keeps_old_and_new_offsets(self) -> None: + context: list[str] = [] + next_new_line = None + next_old_line = None + in_hunk = False + for line in ( + "diff --git a/safe.txt b/safe.txt\n", + "--- a/safe.txt\n", + "+++ b/safe.txt\n", + "@@ -10,3 +10,2 @@\n", + "-first deleted line\n", + ): + next_new_line, next_old_line, in_hunk = self.helper[ + "update_review_chunk_context" + ]( + context, + line, + next_new_line, + next_old_line, + in_hunk, + ) + + rendered = self.helper["review_chunk_context"]( + context, + next_new_line, + next_old_line, + ) + + self.assertIn("new-file line 10", rendered) + self.assertIn("old-file line 11", rendered) + + def test_multiple_long_line_tails_pack_into_following_chunks(self) -> None: + limit = 200 + unit = ( + "diff --git a/large.txt b/large.txt\n" + "--- a/large.txt\n" + "+++ b/large.txt\n" + "@@ -1,5 +1,5 @@\n" + + ("+" + "x" * 205 + "\n") * 5 + ) + + chunks = self.helper["split_oversized_review_unit"](unit, limit) + minimum_chunks = (len(unit.encode("utf-8")) + limit - 1) // limit + + self.assertLessEqual(len(chunks), minimum_chunks + 1) + self.assertEqual("".join(chunk.content for chunk in chunks), unit) + self.assertTrue(all(len(chunk.content.encode("utf-8")) <= limit for chunk in chunks)) + + def test_untracked_continuation_context_keeps_source_line(self) -> None: + unit = ( + "# Untracked File\n" + 'path: "notes.txt"\n' + 'source-line 1: "short\\n"\n' + f'source-line 2: "{"x" * 300}"\n' + ) + + chunks = self.helper["split_oversized_review_unit"](unit, 120) + + self.assertGreater(len(chunks), 2) + self.assertTrue( + any( + "Continuation begins at untracked source line 2" in chunk.context + for chunk in chunks[1:] + ) + ) + self.assertEqual("".join(chunk.content for chunk in chunks), unit) + + def test_deleted_file_continuation_uses_positive_old_line(self) -> None: + unit = ( + "diff --git a/removed.txt b/removed.txt\n" + "--- a/removed.txt\n" + "+++ /dev/null\n" + "@@ -40,50 +0,0 @@\n" + + "-deleted content\n" * 50 + ) + + chunks = self.helper["split_oversized_review_unit"](unit, 180) + + deletion_contexts = [ + chunk.context for chunk in chunks[1:] if "old-file line" in chunk.context + ] + self.assertTrue(deletion_contexts) + self.assertTrue(all("line 0" not in context for context in deletion_contexts)) + self.assertTrue(all("--- a/removed.txt" in context for context in deletion_contexts)) + + def test_long_complete_context_is_retained_or_rejected(self) -> None: + path = "nested/" + "x" * 10_000 + ".txt" + context = [ + f'diff --git "a/{path}" "b/{path}"\n', + f'--- "a/{path}"\n', + f'+++ "b/{path}"\n', + "@@ -1 +1 @@\n", + ] + + rendered = self.helper["review_chunk_context"](context, 2, 2) + + self.assertIn(f'+++ "b/{path}"', rendered) + self.assertIn("@@ -1 +1 @@", rendered) + self.assertIn("Continuation begins at new-file line 2", rendered) + + def test_review_bundle_packs_oversized_unit_tails_globally(self) -> None: + limit = 1_000 + units = [] + for index in range(5): + header = ( + f"diff --git a/file-{index}.txt b/file-{index}.txt\n" + f"--- a/file-{index}.txt\n" + f"+++ b/file-{index}.txt\n" + "@@ -0,0 +1 @@\n" + ) + body = "+" + "x" * (1_100 - len(header.encode("utf-8")) - 2) + "\n" + units.append(header + body) + bundle = "".join(units) + + chunks = self.helper["split_review_bundle"](bundle, limit) + + self.assertEqual(len(chunks), 6) + self.assertEqual("".join(chunk.content for chunk in chunks), bundle) + self.assertTrue(all(len(chunk.content.encode("utf-8")) <= limit for chunk in chunks)) + + def test_large_bundle_stays_single_pass_until_prompt_limit(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + prompts = self.helper["build_review_prompts"]( + repo, + "commit", + "HEAD", + "# Commit Diff\n" + "safe review content\n" * 18_000, + "", + "", + ) + + self.assertEqual(len(prompts), 1) + + def test_bundle_above_prompt_limit_uses_complete_bounded_passes(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + prompts = self.helper["build_review_prompts"]( + repo, + "commit", + "HEAD", + "# Commit Diff\n" + "safe review content\n" * 35_000, + "", + "", + ) + + self.assertGreater(len(prompts), 1) + self.assertTrue( + all( + len(prompt.encode("utf-8")) + <= self.helper["MAX_REVIEW_PROMPT_BYTES"] + for prompt in prompts + ) + ) + self.assertTrue(all("Oversized review bundle chunk:" in prompt for prompt in prompts)) + + def test_review_prompt_preserves_bundle_ending_whitespace(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + bundle = "# Commit Diff\n+Markdown hard break \n+\n" + prompt = self.helper["render_review_prompt"]( + repo, + "commit", + "HEAD", + self.helper["ReviewChunk"](bundle), + "", + "", + ) + + self.assertTrue(prompt.endswith(bundle)) + + def test_review_pass_count_is_bounded(self) -> None: + builder = self.helper["build_review_prompts"] + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + with mock.patch.dict(builder.__globals__, {"MAX_REVIEW_PASSES": 1}): + with self.assertRaisesRegex(SystemExit, "more than 1 bounded passes"): + builder( + repo, + "commit", + "HEAD", + "# Commit Diff\n" + "safe review content\n" * 35_000, + "", + "", + ) + + def test_review_patch_does_not_disclose_controls_in_omitted_paths(self) -> None: + path = ".env.\x1b]52;c;VEVTVA==\x07\udc9b" + + redacted = self.helper["validate_review_patch"]( + "local staged diff", + [path], + "", + ) + + self.assertEqual( + redacted, + self.helper["REVIEW_SECURITY_REDACTION"] + "\n", + ) + self.assertNotIn("\x1b", redacted) + self.assertNotIn("\x07", redacted) + self.assertNotIn("\udc9b", redacted) + + def test_review_patch_omits_everything_when_sensitive_paths_cannot_be_mapped( + self, + ) -> None: + patch = ( + "commit metadata that must not survive a mapping failure\n" + "diff --cc .env\n" + "@@@ -1,1 -1,1 +1,1 @@@\n" + "++placeholder=true\n" + ) + + redacted = self.helper["validate_review_patch"]( + "branch diff", + [".env"], + patch, + ) + + self.assertEqual( + redacted, + self.helper["REVIEW_SECURITY_REDACTION"] + "\n", + ) + self.assertNotIn("placeholder", redacted) + self.assertNotIn("commit metadata", redacted) + + def test_review_metadata_redaction_is_independent_of_path_classification( + self, + ) -> None: + credential = "ghp_" + "A" * 24 + metadata = f" M {credential}.txt\n" + + self.assertEqual( + self.helper["redact_secret_like_review_metadata"](metadata), + self.helper["REVIEW_SECURITY_REDACTION"], + ) + self.assertNotIn( + credential, + self.helper["redact_secret_like_review_metadata"](metadata), + ) + + def test_review_patch_redacts_metadata_but_preserves_code_content(self) -> None: + patch = ( + "Authorization: Basic dXNlcjpwYXNzd29yZA==\n" + "diff --git a/src/runtime.ts b/src/runtime.ts\n" + "--- a/src/runtime.ts\n" + "+++ b/src/runtime.ts\n" + "@@ -0,0 +1 @@\n" + '+const token = "ordinary-hardcoded-value-12345";\n' + ) + + validated = self.helper["validate_review_patch"]( + "commit diff", + ["src/runtime.ts"], + patch, + ) + + self.assertNotIn("dXNlcjpwYXNzd29yZA==", validated) + self.assertIn("ordinary-hardcoded-value-12345", validated) + + def test_review_patch_redacts_standard_diff_paths_but_preserves_hunks(self) -> None: + credential = "ghp_" + "A" * 24 + patch = ( + f"diff --git a/{credential}.ts b/{credential}.ts\n" + f"--- a/{credential}.ts\n" + f"+++ b/{credential}.ts\n" + "@@ -0,0 +1 @@\n" + '+const token = "ordinary-hardcoded-value-12345";\n' + ) + + validated = self.helper["validate_review_patch"]( + "commit diff", + ["src/runtime.ts"], + patch, + ) + + self.assertNotIn(credential, validated) + self.assertIn("ordinary-hardcoded-value-12345", validated) + + def test_review_patch_preserves_combined_and_headerless_hunk_content(self) -> None: + credential_shaped_code = '+token = "ordinary-hardcoded-value-12345"\n' + for patch in ( + "@@ -0,0 +1 @@\n" + credential_shaped_code, + "diff --cc src/runtime.ts\n" + "@@@ -0,0 -0,0 +1 @@@\n" + "++token = \"ordinary-hardcoded-value-12345\"\n", + ): + with self.subTest(patch=patch): + validated = self.helper["validate_review_patch"]( + "commit diff", + ["src/runtime.ts"], + patch, + ) + self.assertIn("ordinary-hardcoded-value-12345", validated) + + def test_review_patch_stops_hunk_classification_at_declared_counts(self) -> None: + credential = "ghp_" + "A" * 24 + patch = ( + "@@ -0,0 +1 @@\n" + "+first file content\n" + f"--- a/{credential}.ts\n" + f"+++ b/{credential}.ts\n" + "@@ -0,0 +1 @@\n" + '+token = "ordinary-hardcoded-value-12345"\n' + ) + + validated = self.helper["validate_review_patch"]( + "commit diff", + ["first.ts", "second.ts"], + patch, + ) + + self.assertNotIn(credential, validated) + self.assertIn("ordinary-hardcoded-value-12345", validated) + + def test_review_patch_enforces_limit_after_metadata_redaction(self) -> None: + patch = "ghp_" + "A" * 24 + + with self.assertRaisesRegex(SystemExit, "after metadata redaction"): + self.helper["validate_review_patch"]( + "commit diff", + [], + patch, + 40, + ) + + def test_tracked_sensitive_paths_are_omitted_in_all_modes(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + (repo / "base.txt").write_text("base\n", encoding="utf-8") + git(repo, "add", "base.txt") + git(repo, "commit", "-q", "-m", "base") + base = git(repo, "rev-parse", "HEAD").strip() + + (repo / ".env").write_text("placeholder=true\n", encoding="utf-8") + (repo / "base.txt").write_text("base\nreview me\n", encoding="utf-8") + git(repo, "add", ".env", "base.txt") + local, local_truncated = self.helper["local_bundle"](repo) + self.assertIn(self.helper["REVIEW_SECURITY_REDACTION"], local) + self.assertNotIn(".env", local) + self.assertNotIn("placeholder=true", local) + self.assertIn("+review me", local) + self.assertFalse(local_truncated) + + git(repo, "commit", "-q", "-m", "sensitive path") + for bundle, truncated in ( + self.helper["branch_bundle"](repo, base), + self.helper["commit_bundle"](repo, "HEAD"), + ): + self.assertIn(self.helper["REVIEW_SECURITY_REDACTION"], bundle) + self.assertNotIn(".env", bundle) + self.assertNotIn("placeholder=true", bundle) + self.assertIn("+review me", bundle) + self.assertFalse(truncated) + + def test_secret_named_workflows_are_reviewable_in_all_modes(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + (repo / "base.txt").write_text("base\n", encoding="utf-8") + git(repo, "add", "base.txt") + git(repo, "commit", "-q", "-m", "base") + base = git(repo, "rev-parse", "HEAD").strip() + + workflow = repo / ".github" / "workflows" / "secret-scan.yml" + workflow.parent.mkdir(parents=True) + workflow.write_text("name: Secret scan\n", encoding="utf-8") + untracked_bundle, _ = self.helper["local_bundle"](repo) + self.assertIn("secret-scan.yml", untracked_bundle) + + git(repo, "add", str(workflow.relative_to(repo))) + tracked_bundle, _ = self.helper["local_bundle"](repo) + self.assertIn("secret-scan.yml", tracked_bundle) + + git(repo, "commit", "-q", "-m", "add secret scanner") + branch_bundle, _ = self.helper["branch_bundle"](repo, base) + commit_bundle, _ = self.helper["commit_bundle"](repo, "HEAD") + self.assertIn("secret-scan.yml", branch_bundle) + self.assertIn("secret-scan.yml", commit_bundle) + + def test_case_variant_secret_named_workflows_remain_sensitive(self) -> None: + for rel in ( + ".GitHub/workflows/secret-scan.yml", + ".github/Workflows/secret-scan.yml", + ".github/workflows/secret-scan.YML", + ): + with self.subTest(rel=rel): + self.assertIsNotNone(self.helper["sensitive_repo_path_risk"](rel)) + self.assertIsNotNone( + self.helper["tracked_sensitive_repo_path_risk"](rel) + ) + + def test_tracked_source_names_and_env_templates_remain_reviewable(self) -> None: + for rel in ( + "tokenizer.py", + "token_count.ts", + "src/token/parser.py", + "src/token/session.ts", + "internal/tokens/types.go", + "packages/token/package.json", + "scripts/tokens/session.sh", + "src/tokens/session.mjs", + "credentials/prod.py", + "secrets/runtime.ts", + "src/credentials/provider.py", + "src/secrets/scanner.ts", + "ui/tokens/session.vue", + "proto/token/session.proto", + "password_validator.go", + ".env.example", + "private/parser.py", + ".agents/skills/openclaw-secret-scanning-maintainer/SKILL.md", + "design-tokens/colors.json", + "design-tokens.json", + "design_tokens.json", + "tokens/default.json", + "token_count/generated.py", + ".docker/Dockerfile", + ".docker/scripts/build.sh", + ".github/workflows/secret-scan.yml", + ): + with self.subTest(rel=rel): + self.assertIsNone(self.helper["tracked_sensitive_repo_path_risk"](rel)) + + def test_untracked_token_source_paths_remain_reviewable(self) -> None: + for rel in ( + "src/token/parser.py", + "src/token/session.ts", + "scripts/tokens/session.sh", + "src/tokens/session.mjs", + "ui/tokens/session.vue", + "proto/token/session.proto", + ): + with self.subTest(rel=rel): + self.assertIsNone(self.helper["sensitive_repo_path_risk"](rel)) + + def test_untracked_credential_shaped_source_content_is_reviewed(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + source = 'const token = "ordinary-hardcoded-value-12345";\n' + path = repo / "src" / "runtime.ts" + path.parent.mkdir() + path.write_text(source, encoding="utf-8") + + bundle, truncated = self.helper["local_bundle"](repo) + + self.assertIn("ordinary-hardcoded-value-12345", bundle) + self.assertFalse(truncated) + + def test_untracked_design_token_artifacts_remain_reviewable(self) -> None: + for rel in ( + "design-tokens.json", + "design_tokens.json", + "src/styles/design-tokens.json", + "themes/dark/design_tokens.json", + "tokens/design-tokens.json", + "tokens/design_tokens.json", + ): + with self.subTest(rel=rel): + self.assertIsNone(self.helper["sensitive_repo_path_risk"](rel)) + self.assertIsNone( + self.helper["tracked_sensitive_repo_path_risk"](rel) + ) + self.assertIsNotNone( + self.helper["sensitive_repo_path_risk"](".env/design-tokens.json") + ) + self.assertIsNotNone( + self.helper["tracked_sensitive_repo_path_risk"]( + ".env/design-tokens.json" + ) + ) + self.assertIsNotNone( + self.helper["tracked_sensitive_repo_path_risk"]( + ".env/tokens/design-tokens.json" + ) + ) + + def test_sensitive_named_source_directories_are_blocked_untracked(self) -> None: + for rel in ( + "credentials/prod.py", + "secrets/runtime.ts", + "src/credentials/provider.py", + "src/secrets/scanner.ts", + ): + with self.subTest(rel=rel): + self.assertIsNotNone(self.helper["sensitive_repo_path_risk"](rel)) + + def test_secret_like_path_values_are_blocked(self) -> None: + secret_path = "notes-" + "ghp_" + "A" * 24 + ".txt" + + self.assertEqual( + self.helper["sensitive_repo_path_risk"](secret_path), + "secret-like path", + ) + self.assertEqual( + self.helper["tracked_sensitive_repo_path_risk"](secret_path), + "secret-like path", + ) + + def test_tracked_env_variants_remain_sensitive(self) -> None: + for rel in ( + ".env-local", + ".env_prod", + ".env/production", + ".env/example/production", + ".env/template/prod", + ): + with self.subTest(rel=rel): + self.assertIsNotNone( + self.helper["tracked_sensitive_repo_path_risk"](rel) + ) + + def test_suffixed_credential_data_paths_remain_sensitive(self) -> None: + for rel in ( + "credentials-prod.json", + "service-account-dev.yaml", + "api-key.backup.json", + "token-prod.json", + "tokens.json", + "auth-token.yaml", + "prod-credentials.json", + "google-service-account.json", + "client-secret.yaml", + "credentials/prod.json", + "prod-credentials/client.conf", + "client-secrets/account.ini", + "token/production.json", + "tokens/production.json", + "tokens/session.dat", + "tokens/cache.json", + "token/user.json", + "tokens/device.sqlite", + "tokens/session.jwt", + "tokens/session", + "backup-secrets/prod.json", + "dev_credentials/runtime.yaml", + "client-secrets-old/account.ini", + "client-secrets/account.properties", + "credentials/prod.xml", + "secrets/prod.md", + "credentials.txt", + "client-secret.csv", + ".docker/config.json", + "deployment/.docker/config.json", + ".netrc", + "config/.netrc", + ".git-credentials", + "config/.git-credentials", + ): + with self.subTest(rel=rel): + self.assertIsNotNone( + self.helper["tracked_sensitive_repo_path_risk"](rel) + ) + + def test_secret_detector_handles_quoted_json_keys(self) -> None: + content = '{"' + 'api_key": "' + realistic_secret_value() + '"}' + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_handles_backtick_credential_literals(self) -> None: + content = "const pass" + "word = `" + realistic_secret_value() + "`;" + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_op_backtick_credential_references(self) -> None: + for content in ( + "pass" + "word=`op read op://vault/item/password`", + "pass" + "word=`op read --no-newline 'op://vault/item/password'`", + "pass" + "word=`op read 'op://vault/item name/password'`", + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_safe_backtick_interpolation(self) -> None: + for content in ( + "to" + "ken = `Bearer ${process.env.TOKEN}`", + "pass" + + "word = `${user.credentials.password}:${config.passwordSalt}`", + "api_" + "key = `${config.primary.apiKey}-${config.secondary.apiKey}`", + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_backtick_interpolation_with_literal_secret( + self, + ) -> None: + literal_secret = "hardcoded" + "credential" + for content in ( + "to" + f"ken = `{literal_secret}-${{process.env.TOKEN}}`", + "pass" + + f"word = `${{user.credentials.password}}-{literal_secret}`", + "to" + + f'ken = `Bearer ${{process.env.TOKEN || "{literal_secret}"}}`', + "pass" + "word = `p@ssw0rd-${process.env.PASSWORD}`", + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_op_backtick_shell_fallbacks(self) -> None: + content = ( + "pass" + + "word=`op read op://vault/item/password || echo real-hardcoded-" + + "fallback`" + ) + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_backtick_fallback_literals(self) -> None: + content = ( + "const pass" + + 'word = `${user.password || "' + + "real-hardcoded-fallback" + + '"}`;' + ) + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_member_reference_fallback_literals(self) -> None: + content = ( + "pass" + + 'word = user.credentials.password || "' + + "real-hardcoded-fallback" + + '"' + ) + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_reference_shaped_fallback_literals(self) -> None: + content = ( + "pass" + + 'word = user.credentials.password || "' + + "user.ACTUAL_SECRET_VALUE" + + '"' + ) + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_reference_shaped_backtick_literals(self) -> None: + content = "const pass" + "word = `user.ACTUAL_SECRET_VALUE`;" + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_python_reference_fallback_literals(self) -> None: + for operator in ("or", "and"): + content = ( + "pass" + + f'word = user.credentials.password {operator} "' + + "real-hardcoded-fallback" + + '"' + ) + with self.subTest(operator=operator): + self.assertTrue(self.helper["secret_text_risk"](content)) + + conditional = ( + "pass" + + 'word = user.credentials.password if user else "' + + "real-hardcoded-fallback" + + '"' + ) + self.assertTrue(self.helper["secret_text_risk"](conditional)) + + cast_fallback = ( + "pass" + + 'word = user.credentials.password as string || "' + + "real-hardcoded-fallback" + + '"' + ) + self.assertTrue(self.helper["secret_text_risk"](cast_fallback)) + + def test_secret_detector_allows_nonsecret_fallback_values(self) -> None: + for content in ( + "to" + "ken = retrieve_authentication_token(request) or None", + "pass" + "word = user.credentials.password || null", + "to" + "ken = provider.issue_token() ?? undefined", + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + self.assertIsNone( + self.helper["top_level_fallback_suffix"]( + 'passwordGenerator("ordinary-option-value")' + ) + ) + + def test_secret_detector_stops_fallback_scan_at_sibling_commas(self) -> None: + for content in ( + '{ password: process.env.PASSWORD, label: prefix + "production-east" }', + 'const token = runtimeToken, checksum = value || "aB3$dE5!gH7#";', + 'const password = runtimeToken, {checksum} = value || "aB3$dE5!gH7#";', + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_secret_detector_keeps_fallbacks_before_sibling_commas(self) -> None: + for content in ( + "const to" + + 'ken = runtimeToken || "real-hardcoded-fallback", checksum = value;', + "pass" + + 'word = (lookupPrimary(), lookupSecondary()) || "hardcoded-secret"', + "pass" + + 'word = getSecret() || "hardcoded-secret"', + "pass" + + 'word = primary, secondary == expected or "hardcoded-' + + 'secret"', + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_call_fallback_literals(self) -> None: + for content in ( + "to" + + 'ken = generate_secure_token() || "' + + "real-hardcoded-fallback" + + '"', + "to" + + 'ken = process.env.TOKEN || choose(/\\)/, "' + + "actual-production-secret" + + '")', + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_grouped_fallbacks_after_line_comments( + self, + ) -> None: + for content in ( + "const pass" + + "word = lookup() // comment\n " + + "|| " + + '"top-level-hardcoded-' + + 'secret"', + "const pass" + + 'word = (lookup() // comment\n || "hardcoded-' + + 'secret")', + "const pass" + + "word = (lookup(), // comment\n" + + 'fallback = value || "real-hardcoded-' + + 'secret")', + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_does_not_cross_top_level_line_comments(self) -> None: + for content in ( + "const pass" + + 'word = lookup() // comment\nconst label = value || "hardcoded-' + + 'secret"', + "const pass" + + "word = ({source: lookup(), // note\n" + + 'label: value || "aB3$dE5!gH7#"});', + "const pass" + + "word = {source: lookup(), // note\n" + + 'label: value || "aB3$dE5!gH7#"};', + "const pass" + + "word = ({source: lookup(), // note\n" + + '["label"]: value || "aB3$dE5!gH7#"});', + "const pass" + + "word = ({source: lookup(), // note\n" + + '7: value || "aB3$dE5!gH7#"});', + "const pass" + + "word = ({source: lookup(), // note\n" + + "...defaults,\n" + + 'label: value || "aB3$dE5!gH7#"});', + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + self.assertTrue( + self.helper["starts_sibling_assignment"]( + "...defaults,\nlabel: value" + ) + ) + + def test_secret_detector_rejects_short_call_fallback_literals(self) -> None: + for content in ( + "pass" + 'word = getpass() || "hunter' + '2!"', + "pass" + 'word = None or "actual-production-' + 'password"', + "pass" + 'word = x or "actual-production-' + 'password"', + "pass" + 'word = "" or "actual-production-' + 'password"', + "pass" + 'word = os.getenv("PASSWORD") or "real' + 'pass9"', + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_literal_secrets_in_call_arguments( + self, + ) -> None: + literal_value = "actual-production-" + "secret" + opaque_value = "CORRECT" + "HORSEBATTERYSTAPLE" + for content in ( + "pass" + + f'word = credentialProvider?.getPassword("{literal_value}")', + "to" + + f'ken = provider.issue_token("{literal_value}").strip()', + "to" + + f'ken = provider.issue_token("scope", "{literal_value}")', + "pass" + + f'word = os.getenv("DATABASE_PASSWORD", "{literal_value}")', + "to" + + f'ken = provider.issue_token(this.#scope, "{literal_value}")', + "to" + + f'ken = factory.get("DATABASE_PASSWORD")("{literal_value}")', + "pass" + + 'word = client.get("CORRECT' + + 'HORSEBATTERYSTAPLE")', + "pass" + f'word = OS.GETENV("{opaque_value}")', + "pass" + f'word = factory().os.getenv("{opaque_value}")', + "pass" + f'word = identity ("{literal_value}")', + "pass" + "word=correcthorsebatterystaple\n(echo ok)", + "pass" + "word=correcthorsebatterystaple\r(echo ok)", + "pass" + "word: correcthorsebatterystaple (production)", + "pass" + "word: correcthorsebatterystaple (primary)", + "pass" + "word = correcthorsebatterystaple (primary)", + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_literals_after_javascript_regex_arguments( + self, + ) -> None: + literal_value = "actual-production-" + "secret" + for content in ( + "to" + f'ken = provider.issue_token(/\\)/, "{literal_value}")', + "to" + f'ken = provider.issue_token(/a,b/, "{literal_value}")', + "to" + + f'ken = provider.issue_token(/[),]/gi, "{literal_value}")', + "to" + + f'ken = provider.issue_token(i++ / total, "{literal_value}" // note\n)', + "to" + + f'ken = provider.issue_token(i-- / total, "{literal_value}" // note\n)', + "to" + + f'ken = provider.issue_token(typeof /\\)/, "{literal_value}")', + "to" + + f'ken = provider.issue_token(() => {{ return /\\)/; }}, "{literal_value}")', + "to" + + f'ken = provider.issue_token(function*() {{ yield /\\)/; }}, "{literal_value}")', + "to" + + f'ken = provider.issue_token(of / total, "{literal_value}" // note\n)', + "to" + + f'ken = provider.issue_token(async () => await /\\);/, "{literal_value}")', + "to" + + f'ken = provider.issue_token(async () => await /\\)/\n, "{literal_value}")', + "to" + + f'ken = provider.issue_token(await /\\)/,\n "{literal_value}")', + "to" + + f'ken = provider.issue_token(await /\\)/.test(input), "{literal_value}")', + "to" + + f'ken = provider.issue_token(value! / divisor, "{literal_value}" // note\n)', + "to" + + f'ken = provider.issue_token(! /\\)/, "{literal_value}")', + "to" + + f'ken = provider.issue_token(value / total, "{literal_value}"[0] / count)', + "to" + + f'ken = provider.issue_token(value / total || "{literal_value}"[0] / count)', + "to" + + f'ken = provider.issue_token(counter++ / total || "{literal_value}"[0] / count)', + "to" + + f'ken = provider.issue_token(counter-- / total || "{literal_value}"[0] / count)', + "to" + + f'ken = provider.issue_token(value! / total || "{literal_value}"[0] / count)', + "to" + + f'ken = provider.issue_token(value> / total || "{literal_value}"[0] / count)', + "var await = value; to" + + f'ken = provider.issue_token(await / total || "{literal_value}"[0] / count)', + "var yield = value; to" + + f'ken = provider.issue_token(yield / total || "{literal_value}"[0] / count)', + "to" + + f'ken = provider.issue_token(() => {{ if (ok) /\\)/.test(x); }}, "{literal_value}")', + "to" + + f'ken = provider.issue_token(() => {{ if (x === "(") /\\)/.test(x); }}, "{literal_value}")', + "to" + + f'ken = provider.issue_token(a /\\)/, "{literal_value}")', + "to" + + f'ken = provider.issue_token(() => {{ if (ok) use(); else /\\)/.test(x); }}, "{literal_value}")', + "to" + + f'ken = provider.issue_token(() => {{ do /\\)/.test(x); while (ok); }}, "{literal_value}")', + "to" + + f'ken = provider.issue_token(() => {{ for (const x of /\\)/) use(x); }}, "{literal_value}")', + "to" + + f'ken = provider.issue_token(() => {{ for await (const x of xs) /\\)/.test(x); }}, "{literal_value}")', + "to" + + f'ken = provider.issue_token(() => {{ if /*c*/ (ok) /\\)/.test(x); }}, "{literal_value}")', + "to" + + f'ken = provider.issue_token(() => {{ if (a) /\\(/.test(x); if (b) /\\)/.test(x); }}, "{literal_value}")', + "to" + + f'ken = provider.issue_token(.../\\)/.source, "{literal_value}")', + "to" + + f'ken = provider.issue_token(() => class C extends /\\)/.constructor {{}}, "{literal_value}")', + "// const await = harmless\n" + + "to" + + f'ken = provider.issue_token(await /\\)/, "{literal_value}")', + "to" + + "ken = provider.issue_token(" + + f'() => {{ for (of / total; ok; of++) use(); next / 2; }}, "{literal_value}")', + "to" + + "ken = provider.issue_token(" + + f'() => {{ for (let x = of / total; x; x++) use(); next / 2; }}, "{literal_value}")', + "to" + + "ken = provider.issue_token(" + + f'() => {{ var await=n; if (await / total) /\\)/.test(x); }}, "{literal_value}")', + "to" + + "ken = provider.issue_token(await /\\)/, " + + "x" * 9000 + + f', "{literal_value}")', + "to" + + f'ken = provider.issue_token(await /\\)/, ok /* ) */, "{literal_value}")', + "to" + + f'ken = provider.issue_token(wrapper(await /\\)\\)/, process.env.TOKEN), "{literal_value}")', + "to" + + "ken = provider.issue_token(await /\\)/,\n" + + f'fallback = "{literal_value}")', + "to" + + f'ken = provider.issue_token(await /foo(\\/a\\/bar)\\)/, "{literal_value}")', + "to" + + f'ken = provider.issue_token(await /\\)/, this.#field, "{literal_value}")', + "to" + + "ken = outer(wrapper(await /\\)/, process.env.TOKEN),\n" + + f' "{literal_value}",\n' + + " /foo/)", + "to" + + f'ken = get_token(await /\\)/, /x\\)/, "{literal_value}")', + "to" + + f'ken = get_token(await /\\)/, process.env.TOKEN) || "{literal_value}"', + "to" + + f'ken = get_token(this.#if(x) / total / count, "{literal_value}")', + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_safe_javascript_regex_arguments(self) -> None: + for content in ( + "to" + "ken = provider.issue_token(/\\)/, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(typeof /\\)/, process.env.TOKEN)", + "to" + "ken = provider.issue_token(total / count, process.env.TOKEN)", + "to" + "ken = provider.issue_token(of / total, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(async () => await /\\);/, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(async () => await /\\)/\n, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(await /\\)/,\n process.env.TOKEN)", + "to" + + "ken = provider.issue_token(await /\\)/.test(input), process.env.TOKEN)", + "to" + + "ken = provider.issue_token(value! / divisor, process.env.TOKEN)", + "to" + "ken = provider.issue_token(! /\\)/, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(value / total, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(value / total || process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + "() => { if (ok) /\\)/.test(x); }, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + "items.with(0, x) / total, process.env.TOKEN / count)", + "to" + + "ken = provider.issue_token(" + + "await / total, process.env.TOKEN / count)", + "to" + + "ken = provider.issue_token(" + + "yield / total, process.env.TOKEN / count)", + "to" + + "ken = provider.issue_token(" + + "value> / total, process.env.TOKEN / count)", + "to" + + "ken = provider.issue_token(" + + "value / total, process.env.TOKEN / count)", + "to" + + "ken = provider.issue_token(" + + "a /\\)/, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + "() => { if (ok) use(); else /\\)/.test(x); }, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + "() => { do /\\)/.test(x); while (ok); }, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + "() => { for (const x of /\\)/) use(x); }, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + "() => { for await (const x of xs) /\\)/.test(x); }, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + "() => { if /*c*/ (ok) /\\)/.test(x); }, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + "() => { if (a) /\\(/.test(x); if (b) /\\)/.test(x); }, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + ".../\\)/.source, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + "() => class C extends /\\)/.constructor {}, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + "() => { for (of / total; ok; of++) use(); next / 2; }, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + "() => { for (let x = of / total; x; x++) use(); next / 2; }, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + "() => { for (const {x} of /\\)/) use(x); }, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(" + + "() => { var await=n; if (await / total) /\\)/.test(x); }, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(await /\\)/, " + + "x" * 9000 + + ", process.env.TOKEN)", + "to" + + "ken = provider.issue_token(await /\\)/, ok /* ) */, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(wrapper(await /\\)\\)/, process.env.TOKEN), process.env.TOKEN)", + "to" + + "ken = provider.issue_token(await /\\)/,\n" + + "fallback = process.env.TOKEN)", + "to" + + "ken = provider.issue_token(await /foo(\\/a\\/bar)\\)/, process.env.TOKEN)", + "to" + + "ken = provider.issue_token(await /\\)/, this.#field, process.env.TOKEN)", + "to" + + "ken = outer(wrapper(await /\\)/, process.env.TOKEN),\n" + + " process.env.TOKEN,\n" + + " /foo/)", + "to" + + 'ken = get_token(a / fn(x) / b)\nreport("actual-production-secret")', + "to" + + 'ken = get_token(await /\\)"actual-production-secret"/, process.env.TOKEN)', + "to" + + 'ken = get_token(await /\\)/, /x)"actual-production-secret"/, process.env.TOKEN)', + "to" + + "ken = get_token(await /\\)/, process.env.TOKEN) || process.env.FALLBACK", + "to" + + "ken = get_token(this.#if(x) / total / count, process.env.TOKEN)", + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_regex_parser_accepts_expression_keyword_contexts(self) -> None: + for content in ( + "class C extends /\\)/.constructor {}", + "export default /\\)/;", + ): + with self.subTest(content=content): + start = content.index("/") + self.assertIsNotNone( + self.helper["javascript_regex_literal_end"](content, start) + ) + + def test_call_argument_split_preserves_secret_shaped_regex(self) -> None: + regex = "/password=" + "actual-production-secret" + ",foo/" + + self.assertEqual( + self.helper["split_top_level_call_arguments"]( + f"{regex}, process.env.TOKEN" + ), + [regex, " process.env.TOKEN"], + ) + + def test_call_argument_split_treats_contextual_of_as_identifier(self) -> None: + self.assertEqual( + self.helper["split_top_level_call_arguments"]( + "of / total, other / +count, final" + ), + ["of / total", " other / +count", " final"], + ) + + def test_control_condition_scan_is_cached_per_source(self) -> None: + scan = self.helper["javascript_control_condition_closes"] + scan.cache_clear() + content = " ".join("if (ok) /a/.test(value);" for _ in range(32)) + starts = [match.start() for match in re.finditer(r"/a/", content)] + + for start in starts: + self.assertIsNotNone( + self.helper["javascript_regex_literal_end"](content, start) + ) + + cache = scan.cache_info() + self.assertEqual(cache.misses, 1) + self.assertGreaterEqual(cache.hits, len(starts) - 1) + + def test_credential_uri_contexts_are_scanned_once(self) -> None: + scan = self.helper["string_contexts_at"] + wrapped = mock.Mock(wraps=scan) + content = "\n".join( + f"URL_{index}=postgres://" + f"user:$PASSWORD_{index}@db.example/app" + for index in range(64) + ) + with mock.patch.dict( + self.helper["credentialed_uri_risk"].__globals__, + {"string_contexts_at": wrapped}, + ): + self.assertFalse(self.helper["credentialed_uri_risk"](content)) + + wrapped.assert_called_once() + + def test_secret_detector_scopes_premature_regex_tail_to_current_call( + self, + ) -> None: + literal_value = "actual-production-" + "secret" + for content in ( + "to" + + "ken = get_token(await /\\)/, process.env.TOKEN)\n" + + f'const fixture = "{literal_value}"', + "to" + + 'ken = headers.get("Authorization"); const ratio = a / b\n' + + f'const fixture = "{literal_value}"', + "to" + + "ken = get_token(await /\\)/, process.env.TOKEN)\r\n" + + f'const fixture = "{literal_value}"', + "to" + + 'ken = issue(); route = "/health/status/check";', + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_credential_lookup_keys(self) -> None: + for content in ( + 'pass' + 'word = os.getenv("DATABASE_PASSWORD")', + 'to' + 'ken = headers.get("Authorization")', + 'to' + 'ken = request.headers.get("Authorization")', + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_public_call_arguments(self) -> None: + for content in ( + "access_" + + 'token = credentials.get_token("https://management.azure.com/.default")', + "access_" + + 'token = self._credential.get_token("https://management.azure.com/.default")', + "access_" + 'token = credentials.get_token("scope")', + "access_" + + 'token = credentials.get_token("api://00000000-0000-0000-0000-000000000000/.default")', + "access_" + + 'token = credentials.get_token("3db474b9-6a0c-4840-96ac-1fceb342124f/.default")', + "access_" + + "to" + + 'ken = credentials.get_token("scope-a", ' + + '"https://management.azure.com/.default")', + "access_" + + "to" + + 'ken = credentials.get_token("https://[")', + "pass" + 'word = input("Enter your password: ")', + "pass" + 'word = input("Password: ")', + "pass" + 'phrase = getpass.getpass("Passphrase: ")', + "pass" + + 'word = getpass.getpass(prompt="Enter your password: ")', + "api_" + + 'key = input("Enter your API key: ")', + "api_" + + 'key = getpass.getpass("Enter your API key: ")', + "api_" + + 'key = getpass.getpass(prompt="Enter your API key: ")', + "to" + 'ken = input("Enter API to' + 'ken: ")', + "to" + 'ken = input ("Enter API to' + 'ken: ")', + "api" + 'Key = prompt("Enter API key: ")', + "api" + 'Key = prompt("Enter API key: ", defaultApiKey)', + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_secret_shaped_public_arguments(self) -> None: + for content in ( + "access_" + + "to" + + 'ken = credentials.get_token("https://api.example.test/?access_' + + 'token=hardcoded-secret")', + "access_" + + "to" + + 'ken = credentials.get_token("https://example.test:not-a-port/.default")', + "access_" + + "to" + + 'ken = credentials.get_token("https://example.test/.default?x=%67%68%70")', + "access_" + + "to" + + 'ken = credentials.get_token("https://gl' + + 'pat-abcdefghijklmnopqrst.example.com/.default")', + "access_" + + "to" + + 'ken = credentials.get_token("https://gl%09' + + 'pat-abcdefghijklmnopqrst.example.com/.default")', + "access_" + + "to" + + 'ken = credentials.get_token("https://example.test/' + + 'correct-horse-battery-staple")', + "access_" + + "to" + + 'ken = credentials.get_token("3db474b9-6a0c-4840-96ac-' + + '1fceb342124f/actual-production-secret")', + "pass" + 'word = decode("correct horse battery staple?")', + "api" + + "Key = prompt(" + + '"Enter API key: ", "real' + + 'pass9")', + "pass" + + 'word = prompt("real' + + 'pass9")', + "api" + + "Key = prompt({default: " + + '"real' + + 'pass9"})', + "pass" + + "word = in" + + 'put("correct horse battery staple?")', + "access_" + + "to" + + 'ken = custom_client.get_token("correct-horse-battery-staple")', + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_short_reference_fallback_literals(self) -> None: + for expression in ("env.TOKEN", "getToken()"): + content = ( + "to" + + f'ken = {expression} || "' + + "live-secret-value-123456" + + '"' + ) + with self.subTest(expression=expression): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_parenthesized_fallback_literals(self) -> None: + operator = "o" + "r" + for opening, closing in (("(", ")"), ("((", "))")): + content = ( + "pass" + + f'word = {opening}os.getenv("PASS' + + f'WORD") {operator} "real' + + f'pass9"{closing}' + ) + with self.subTest(opening=opening): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_bare_secret_with_reference_prefix( + self, + ) -> None: + content = "to" + "ken = ab.cd-0123456789abcdefghijklmnop" + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_multiline_call_fallback_literals(self) -> None: + content = ( + "to" + + "ken = provider.issue_token()\n" + + ' || "real-hardcoded-' + + 'fallback"' + ) + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_operator_only_multiline_fallbacks(self) -> None: + content = ( + "pass" + + "word = user.credentials.password ||\n" + + ' "actual-production-' + + 'secret"' + ) + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_nested_multiline_fallbacks(self) -> None: + content = ( + "pass" + + "word = user.credentials.password || getDefault(\n" + + ' "actual-production-' + + 'secret"\n)' + ) + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_comment_separated_call_fallbacks(self) -> None: + content = ( + "to" + + "ken = provider.issue_token()\n" + + " // local fallback\n" + + ' || "real-hardcoded-' + + 'fallback"' + ) + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_optional_call_fallback_literals(self) -> None: + content = ( + "to" + + 'ken = provider?.issue_token() || "real-hardcoded-' + + 'fallback"' + ) + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_ignores_comment_delimiters_in_calls(self) -> None: + content = ( + "to" + + "ken = provider.issue_token(/* ) */ request)" + + ' || "real-hardcoded-' + + 'fallback"' + ) + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_bare_variable_secret_references(self) -> None: + for prefix in ( + "cached", + "current", + "existing", + "loaded", + "previous", + "resolved", + "saved", + "stored", + ): + with self.subTest(prefix=prefix): + self.assertFalse( + self.helper["secret_text_risk"]( + f"refresh_token = {prefix}_refresh_token" + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + "refresh_" + "token = " + "abcdefghijklmnopqrstuvwxyz" + ) + ) + self.assertFalse( + self.helper["secret_text_risk"]( + "const access_" + + "to" + + "ken = generated_password_" + + "value" + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + "ACCESS_" + + "TO" + + "KEN=generated_access_token_" + + realistic_secret_value() + + "_value" + ) + ) + for content in ( + "const token = authenticationToken;", + "const token = longVariableReference;", + "const token = tokenFromEnvironment;", + "const password = databasePassword;", + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_secret_detector_handles_raw_jwt(self) -> None: + content = ".".join( + ( + "eyJhbGciOiJIUzI1NiJ9", + "eyJzdWIiOiIxMjM0NTY3ODkwIn0", + "signatureplaceholder", + ) + ) + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_handles_private_key_header_variants(self) -> None: + for content in ( + "-----BEGIN " + "ENCRYPTED PRIVATE KEY-----", + "-----BEGIN PGP " + "PRIVATE KEY BLOCK-----", + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_dotted_config_keys(self) -> None: + self.assertFalse( + self.helper["secret_text_risk"]( + 'permissions.autoreview.filesystem={":minimal"="read"}' + ) + ) + + def test_secret_detector_allows_typescript_function_parameter_types(self) -> None: + signature = ( + "function formatCredentialLabel(" + + "credential" + + ": ClaudeCliReadableCredential" + + "): string {" + ) + access_key = "AKIA" + "ABCDEFGHIJKLMNOP" + secret_assignment = "const api" + 'Key = "' + access_key + '";' + literal_value = "actual-production-" + "secret" + parameter_name = "api" + "Key" + type_name = "Api" + "Credential" + typed_default = ( + "function connect(" + + parameter_name + + ": " + + type_name + + ' = "' + + literal_value + + '") {}' + ) + benign_default = ( + "function connect(" + + parameter_name + + ": " + + type_name + + " = defaultCredential) {}" + ) + + self.assertFalse(self.helper["secret_text_risk"](signature)) + self.assertFalse(self.helper["secret_text_risk"](benign_default)) + self.assertTrue( + self.helper["secret_text_risk"]( + signature + "\n " + secret_assignment + "\n}" + ) + ) + self.assertTrue(self.helper["secret_text_risk"](typed_default)) + + def test_secret_detector_handles_punctuation_and_multiline_diff_values(self) -> None: + value = "Correct-Horse!" + "@Battery$Staple" + patch = ( + "@@ -1 +1,2 @@\n" + '+"api_key":\n' + '+ "' + value + '"\n' + ) + + self.assertTrue( + any( + self.helper["secret_text_risk"](content) + for content in self.helper["unified_diff_contents"](patch) + ) + ) + + def test_secret_detector_does_not_treat_code_expressions_as_values(self) -> None: + for content in ( + "token = secrets.token_urlsafe(32)", + "token = response", + "password = undefined", + "token = process.env.GITHUB_TOKEN", + 'token = os.environ["GITHUB_TOKEN"]', + 'password = payload.get("password")', + "token = auth_response.credentials.access_token", + "token = response.authentication.accessToken", + "token = request.headers.authorization", + "password = account.credentials.password", + "password = user.credentials.password", + "password = user?.credentials?.password", + "password = `${process.env.PASSWORD}`", + "{ password: process.env.PASSWORD, username }", + "token = process.env.TOKEN as string", + "self.access_token = self.authentication.access_token", + "this.accessToken = this.authentication.accessToken", + "api_key = client.settings.apiKey", + 'token = "$GITHUB_TOKEN"', + 'token = "$env:GITHUB_TOKEN"', + 'token = "${{ secrets.GITHUB_TOKEN }}"', + 'token = "op://Vault/Item/token"', + 'token = "op://Development/AWS/Access Keys/access_key_id"', + 'token_endpoint = "https://accounts.example.com/oauth2/token"', + 'password_policy = "minimum-twelve-characters"', + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + self.assertFalse( + self.helper["secret_text_risk"]( + "pass" + + "word = user.credentials." + + "password\nif password is None:\n reset()" + ) + ) + self.assertFalse( + self.helper["secret_text_risk"]( + "pass" + "word = process.env.PASSWORD " + ) + ) + + def test_review_patch_allows_provider_references_and_test_placeholders( + self, + ) -> None: + token_name = "to" + "ken" + key_name = "api_" + "key" + secret_name = "api_" + "secret" + safe_patch = ( + "diff --git a/provider.ts b/provider.ts\n" + "--- a/provider.ts\n" + "+++ b/provider.ts\n" + "@@ -1 +1,6 @@\n" + f"-const {token_name} = data.session?.access_token;\n" + f"+const {token_name} = data.session?.access_token;\n" + "+const api" + f"Key = providerConfig.{key_name};\n" + "+const api" + "Sec" + f"ret = providerConfig.{secret_name};\n" + f'+const fixture = {{ {key_name}: "test-key" }};\n' + f'+const fixtureSecret = {{ {secret_name}: "test-secret" }};\n' + f'+const session = {{ access_{token_name}: "test-token" }};\n' + ) + + self.assertEqual( + self.helper["validate_review_patch"]( + "branch diff", + ["provider.ts"], + safe_patch, + ), + safe_patch, + ) + + def test_provider_reference_allowlist_still_rejects_real_credentials( + self, + ) -> None: + key_name = "api_" + "key" + literal_value = "actual-production-" + "secret" + structured_value = "ghp_" + "ActualToken1234567890" + unsafe_values = ( + f'const config = {{ {key_name}: "{literal_value}" }};', + f'const config = {{ {key_name}: "{structured_value}" }};', + f'const config = {{ {key_name}: "test-key-extra" }};', + ) + + for content in unsafe_values: + with self.subTest(content=content): + self.assertTrue( + self.helper["secret_text_risk"]( + content, + javascript_dialect="typescript", + ) + ) + + def test_secret_detector_allows_typescript_object_secret_references(self) -> None: + content = ( + "async function configure(context: RuntimeContext) {\n" + " const cliDevice = await login();\n" + " const driverPassword = readPassword();\n" + " return {\n" + " access" + + "Token" + + ": cliDevice.access" + + "Token,\n" + + " pass" + + "word: driverPass" + + "word,\n" + + " ...(context.driverPass" + + "word ? { pass" + + "word: context.driverPass" + + "word } : {}),\n" + + " };\n" + + "}" + ) + yaml_literal = "pass" + "word: actualProductionSecret," + yaml_reference = "pass" + "word: context.driverPass" + "word" + yaml_flow_reference = ( + "{ pass" + "word: context.driverPass" + "word, enabled: true }" + ) + sut_reference = ( + "const pass" + + "word = context.sutPass" + + "word;" + ) + literal_value = "actual-production-" + "secret" + source_literal = ( + "function configure() { return { pass" + + 'word: "' + + literal_value + + '" }; }' + ) + jwt_like = "eyJhbGciOiJIUzI1NiJ9" + ".payload." + "signature" + undeclared_member = ( + "const config = { pass" + + "word: " + + jwt_like + + " };" + ) + jwt_root = jwt_like.split(".", 1)[0] + declared_member = ( + f"const {jwt_root} = {{}};\n" + + "const config = { pass" + + "word: " + + jwt_like + + " };" + ) + prefixed_member = ( + "const session = {};\n" + + "const config = { pass" + + "word: session." + + jwt_like + + " };" + ) + declared_identifier = "CorrectHorseBattery" + "Staple123" + declared_identifier_value = ( + f"const {declared_identifier} = 0;\n" + + "const config = { pass" + + "word: " + + declared_identifier + + " };" + ) + suffixed_reference = ( + "const config = { pass" + + "word: context.driverPass" + + "wordExtra };" + ) + prefixed_reference = ( + "const config = { pass" + + "word: xcontext.driverPass" + + "word };" + ) + final_property = ( + "function configure(context: RuntimeContext) { return { pass" + + "word: context.driverPass" + + "word }; }" + ) + inline_property = ( + "function configure(context: RuntimeContext) { return { pass" + + "word: context.driverPass" + + "word, enabled: true }; }" + ) + asserted_property = ( + "function configure(context: RuntimeContext) { return { pass" + + "word: context.driverPass" + + "word! }; }" + ) + cast_property = ( + "function configure(context: RuntimeContext) { return { pass" + + "word: context.driverPass" + + "word as string }; }" + ) + union_cast_property = ( + "function configure(context: RuntimeContext) { return { pass" + + "word: context.driverPass" + + "word as string | undefined }; }" + ) + next_statement = ( + "function configure(context: RuntimeContext) {\n" + + " const pass" + + "word = context.driverPass" + + "word\n" + + " return consume();\n" + + "}" + ) + line_comment = ( + "const pass" + + "word = context.driverPass" + + "word // supplied by CI\n" + + "consume();" + ) + block_comment = ( + "const pass" + + "word = context.driverPass" + + "word /* supplied by CI */;" + ) + concatenated_literal = ( + "function configure(context: RuntimeContext) { return { pass" + + "word: context.driverPass" + + 'word + "' + + literal_value + + '" }; }' + ) + continued_literal = ( + "function configure(context: RuntimeContext) { return { pass" + + "word: context.driverPass" + + "word\n" + + ' + "' + + literal_value + + '" }; }' + ) + fallback_literal = ( + "function configure(context: RuntimeContext) { return { pass" + + "word: context.driverPass" + + 'word ?? "' + + literal_value + + '" }; }' + ) + assigned_literal = ( + "const pass" + + "word = context.driverPass" + + "word\n" + + ' = "' + + literal_value + + '";' + ) + newline_cast_literal = ( + "const pass" + + "word = context.driverPass" + + "word\n" + + " as string + \"" + + literal_value + + '";' + ) + commented_continuation = ( + "const pass" + + "word = context.driverPass" + + "word // supplied by CI\n" + + ' + "' + + literal_value + + '";' + ) + unicode_comment_continuation = ( + "const pass" + + "word = context.driverPass" + + "word // supplied by CI" + + chr(0x2028) + + ' + "' + + literal_value + + '";' + ) + unicode_space_continuation = ( + "const pass" + + "word = context.driverPass" + + "word\n" + + chr(0x00A0) + + '+ "' + + literal_value + + '";' + ) + multiline_cast_literal = ( + "const pass" + + "word = context.driverPass" + + "word as string\n" + + '+ "' + + literal_value + + '";' + ) + leading_comma_statement = ( + "const pass" + + "word = context.driverPass" + + "word\n" + + ', username = "' + + literal_value + + '";' + ) + unary_statement = ( + "const pass" + + "word = context.driverPass" + + "word\n" + + '!audit("' + + literal_value + + '");' + ) + inequality_literal = ( + "const pass" + + "word = context.driverPass" + + "word\n" + + '!== "' + + literal_value + + '";' + ) + member_call_literal = ( + "const pass" + + "word = context.driverPass" + + 'word["concat"]("safe", "' + + literal_value + + '");' + ) + continued_then_unary_statement = ( + "const pass" + + "word = context.driverPass" + + "word + suffix\n" + + '!audit("' + + literal_value + + '");' + ) + trailing_operator_literal = ( + "const pass" + + "word = context.driverPass" + + "word + suffix +\n" + + ' "' + + literal_value + + '";' + ) + plain_javascript_as_statement = ( + "const pass" + + "word = context.driverPass" + + "word\n" + + 'as("' + + literal_value + + '");' + ) + typescript_dollar_identifier = ( + "const pass" + + "word = context.driverPass" + + "word\n" + + 'as$logger("' + + literal_value + + '");' + ) + + self.assertFalse( + self.helper["secret_text_risk"]( + content, + javascript_dialect="typescript", + ) + ) + self.assertFalse( + self.helper["secret_text_risk"]( + sut_reference, + javascript_dialect="typescript", + ) + ) + self.assertTrue(self.helper["secret_text_risk"](sut_reference)) + self.assertFalse( + self.helper["secret_text_risk"]( + plain_javascript_as_statement, + javascript_dialect="javascript", + ) + ) + self.assertFalse( + self.helper["secret_text_risk"]( + typescript_dollar_identifier, + javascript_dialect="typescript", + ) + ) + self.assertFalse( + self.helper["secret_text_risk"]( + final_property, + javascript_dialect="typescript", + ) + ) + for source_reference in ( + inline_property, + asserted_property, + cast_property, + union_cast_property, + next_statement, + line_comment, + block_comment, + leading_comma_statement, + unary_statement, + continued_then_unary_statement, + ): + with self.subTest(source_reference=source_reference): + self.assertFalse( + self.helper["secret_text_risk"]( + source_reference, + javascript_dialect="typescript", + ) + ) + for unsafe_source_reference in ( + concatenated_literal, + continued_literal, + fallback_literal, + assigned_literal, + newline_cast_literal, + commented_continuation, + unicode_comment_continuation, + unicode_space_continuation, + multiline_cast_literal, + inequality_literal, + member_call_literal, + trailing_operator_literal, + ): + with self.subTest(unsafe_source_reference=unsafe_source_reference): + self.assertTrue( + self.helper["secret_text_risk"]( + unsafe_source_reference, + javascript_dialect="typescript", + ) + ) + self.assertTrue(self.helper["secret_text_risk"](yaml_literal)) + self.assertTrue(self.helper["secret_text_risk"](yaml_reference)) + self.assertTrue(self.helper["secret_text_risk"](yaml_flow_reference)) + self.assertTrue(self.helper["secret_text_risk"](source_literal)) + self.assertTrue(self.helper["secret_text_risk"](undeclared_member)) + self.assertTrue(self.helper["secret_text_risk"](declared_member)) + self.assertTrue(self.helper["secret_text_risk"](prefixed_member)) + self.assertTrue( + self.helper["secret_text_risk"](declared_identifier_value) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + suffixed_reference, + javascript_dialect="typescript", + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + prefixed_reference, + javascript_dialect="typescript", + ) + ) + + def test_secret_detector_allows_lifecycle_named_typescript_references(self) -> None: + key_term = "Api" + "Key" + key_field = key_term[0].lower() + key_term[1:] + credential_term = "Cred" + "ential" + source = ( + f"const resolvedStream{key_term} = resolveAttemptDispatch{key_term}({{\n" + f" {key_field}Info,\n" + " runtimeAuthState,\n" + "});\n" + f"const successful{credential_term} = successfulProfileId\n" + " ? attemptAuthProfileStore.profiles[successfulProfileId]\n" + " : undefined;\n" + f"const successful{key_term}Info = get{key_term}Info();\n" + f"const {key_field} = successful{key_term}Info?.{key_field};\n" + f"const resolved{key_term} = resolveSecretSentinel({key_field});\n" + "return {\n" + f" resolved{key_term}: resolvedStream{key_term},\n" + f" {credential_term.lower()}: successful{credential_term},\n" + f" {key_field}: resolved{key_term},\n" + "};\n" + ) + literal_value = "actual-production-" + "secret" + unsafe_sources = ( + f'const resolved{key_term} = "' + literal_value + '";', + "const config = { pass" + + "word: resolved" + + key_term + + ' + "' + + literal_value + + "\" };", + "const config = { pass" + + "word: Abcdefghijklmnop.Qrstuvwxyzabcdef };", + ) + + self.assertFalse( + self.helper["secret_text_risk"]( + source, + javascript_dialect="typescript", + ) + ) + for unsafe_source in unsafe_sources: + with self.subTest(unsafe_source=unsafe_source): + self.assertTrue( + self.helper["secret_text_risk"]( + unsafe_source, + javascript_dialect="typescript", + ) + ) + + store_reference = ( + "const cred" + + "ential = attemptAuthProfileStore.profiles[successfulProfileId];" + ) + optional_store_reference = ( + "const cred" + + "ential = attemptAuthProfileStore?.[successfulProfileId];" + ) + quoted_store_reference = ( + "const cred" + + 'ential = attemptAuthProfileStore["profiles"][successfulProfileId];' + ) + yaml_store_literal = ( + "pass" + + 'word: attemptAuthProfileStore["' + + literal_value + + '"]' + ) + quoted_secret_key = "N7xQ2mP9vK4r" + "T8wZ" + typescript_store_literal = ( + "const pass" + + 'word = attemptAuthProfileStore["' + + quoted_secret_key + + '"];' + ) + self.assertFalse( + self.helper["secret_text_risk"]( + store_reference, + javascript_dialect="typescript", + ) + ) + self.assertFalse( + self.helper["secret_text_risk"]( + optional_store_reference, + javascript_dialect="typescript", + ) + ) + self.assertFalse( + self.helper["secret_text_risk"]( + quoted_store_reference, + javascript_dialect="typescript", + ) + ) + self.assertTrue(self.helper["secret_text_risk"](yaml_store_literal)) + self.assertTrue( + self.helper["secret_text_risk"]( + typescript_store_literal, + javascript_dialect="typescript", + ) + ) + + def test_secret_detector_allows_typescript_credential_plumbing_fixture(self) -> None: + source = (FIXTURES / "typescript-benign-references.ts").read_text( + encoding="utf-8" + ) + + patch = ( + "diff --git a/src/credential-plumbing.ts b/src/credential-plumbing.ts\n" + "new file mode 100644\n" + "--- /dev/null\n" + "+++ b/src/credential-plumbing.ts\n" + f"@@ -0,0 +1,{len(source.splitlines())} @@\n" + + "".join(f"+{line}\n" for line in source.splitlines()) + ) + validated = self.helper["validate_review_patch"]( + "typescript credential plumbing fixture", + ["src/credential-plumbing.ts"], + patch, + ) + for reference in ( + "filePassword", + "passwordResolution.password", + "tokenResolution.token", + "CredentialUnavailableDiagnostic", + "tokenRef", + "keyRef", + ): + self.assertIn(reference, validated) + + def test_secret_detector_allows_typescript_member_reference_assignment(self) -> None: + source = "legacyXSearchResolvedRecord.apiKey = resolution.value;" + patch = ( + "diff --git a/src/runtime-web-tools.ts b/src/runtime-web-tools.ts\n" + "--- a/src/runtime-web-tools.ts\n" + "+++ b/src/runtime-web-tools.ts\n" + "@@ -20,2 +20,3 @@ function resolveLegacySearch() {\n" + f" {source}\n" + "+const contractDigest = digestRuntimeWebOwnerContract(contract);\n" + ) + + self.assertFalse( + self.helper["secret_text_risk"]( + source, + javascript_dialect="typescript", + ) + ) + self.assertEqual( + self.helper["validate_review_patch"]( + "typescript member reference assignment", + ["src/runtime-web-tools.ts"], + patch, + ), + patch, + ) + + fake_literal = next( + line + for line in (FIXTURES / "typescript-sensitive-literals.ts") + .read_text(encoding="utf-8") + .splitlines() + if line.strip() + ) + self.assertTrue( + self.helper["secret_text_risk"]( + fake_literal, + javascript_dialect="typescript", + ) + ) + def test_review_bundle_preserves_typescript_config_paths(self) -> None: + source = (FIXTURES / "typescript-benign-config-path-references.ts").read_text( + encoding="utf-8" + ) + patch = ( + "diff --git a/src/config-path-references.ts b/src/config-path-references.ts\n" + "new file mode 100644\n" + "--- /dev/null\n" + "+++ b/src/config-path-references.ts\n" + f"@@ -0,0 +1,{len(source.splitlines())} @@\n" + + "".join(f"+{line}\n" for line in source.splitlines()) + ) + + validated = self.helper["validate_review_patch"]( + "typescript config path references", + ["src/config-path-references.ts"], + patch, + ) + + for config_path in ( + "channels.irc.accounts.${accountId}.passwordFile", + "channels.irc.accounts.${accountId}.nickserv.passwordFile", + "channels.nextcloud-talk.accounts.${accountId}.botSecret", + "channels.nextcloud-talk.accounts.${accountId}.botSecretFile", + "channels.telegram.accounts.${accountId}.tokenFile", + ): + self.assertIn(config_path, validated) + + token_term = "To" + "ken" + truncated_call_patch = ( + "diff --git a/src/token.ts b/src/token.ts\n" + "--- a/src/token.ts\n" + "+++ b/src/token.ts\n" + "@@ -40,3 +40,4 @@ function resolveAccountToken() {\n" + f"+ const account{token_term} = resolveRuntime{token_term}Value({{\n" + "+ value: accountConfig.token,\n" + "@@ -70,3 +71,4 @@ function resolveConfigToken() {\n" + f"+ const config{token_term} = resolveRuntime{token_term}Value({{\n" + "+ value: merged.token,\n" + ) + self.assertEqual( + self.helper["validate_review_patch"]( + "typescript truncated credential calls fixture", + ["src/token.ts"], + truncated_call_patch, + ), + truncated_call_patch, + ) + + def test_secret_detector_rejects_sensitive_literal_fixture_corpus(self) -> None: + source = (FIXTURES / "typescript-sensitive-literals.ts").read_text( + encoding="utf-8" + ) + corpus = [line for line in source.splitlines() if line.strip()] + + self.assertGreaterEqual(len(corpus), 7) + for literal_assignment in corpus: + with self.subTest(literal_assignment=literal_assignment): + self.assertTrue( + self.helper["secret_text_risk"]( + literal_assignment, + javascript_dialect="typescript", + ) + ) + truncated_literal = ( + "const incompleteToken = resolveToken({ value: \"" + + realistic_secret_value() + + "\";" + ) + self.assertTrue( + self.helper["secret_text_risk"]( + truncated_literal, + javascript_dialect="typescript", + ) + ) + truncated_short_literal = ( + "const incompleteToken = resolveToken({ value: \"" + + "short" + + "pwd" + + "\";" + ) + self.assertTrue( + self.helper["secret_text_risk"]( + truncated_short_literal, + javascript_dialect="typescript", + ) + ) + + def test_known_secret_fragment_scan_handles_many_javascript_regexes(self) -> None: + fragment = "password file" + regex_count = 2_000 + source = ";".join(f"/{fragment} {index}/" for index in range(regex_count)) + pattern = self.helper["known_secret_fragment_pattern"]([fragment]) + + spans = self.helper["repeated_secret_fragment_spans"]( + source, + pattern, + javascript_dialect="typescript", + ) + + self.assertEqual(len(spans), regex_count) + + def test_lifecycle_reference_scan_is_bounded_for_non_matching_identifier(self) -> None: + source = "const value = resolved" + "A" * 100_000 + "X;" + + started = time.monotonic() + spans = self.helper["javascript_reference_spans"](source) + + self.assertEqual(spans, frozenset()) + self.assertLess(time.monotonic() - started, 5.0) + + def test_review_patch_decodes_git_quoted_source_paths(self) -> None: + property_name = "pass" + "word" + reference = "context.driverPass" + "word" + patch = ( + 'diff --git "a/\\303\\251.ts" "b/\\303\\251.ts"\n' + '--- "a/\\303\\251.ts"\n' + '+++ "b/\\303\\251.ts"\n' + "@@ -40,2 +40,3 @@ function configure(context: RuntimeContext) {\n" + " return {\n" + "+ " + + property_name + + ": " + + reference + + ",\n" + " };\n" + ) + + self.assertEqual( + self.helper["validate_review_patch"]( + "local staged diff", + ["é.ts"], + patch, + ), + patch, + ) + self.assertEqual( + self.helper["javascript_review_dialect"]("module.mts"), + "typescript", + ) + self.assertEqual( + self.helper["javascript_review_dialect"]("module.cts"), + "typescript", + ) + + def test_secret_detector_allows_generated_fixture_credentials(self) -> None: + property_name = "pass" + "word" + variable_name = "to" + "ken" + access_property = "access" + "Token" + generated_fixture = ( + f"function register() {{ return {{ {property_name}: " + + "`matrix-qa-${randomUUID()}` }; }" + ) + generated_marker = ( + f"const {variable_name} = " + + 'buildMatrixQaToken("MATRIX_QA_E2EE_THREAD");' + ) + decoy_fixture = ( + f"const config = {{ {access_property}: " + + 'decoy-' + + 'token" };' + ) + invalid_recovery_fixture = ( + 'const recoveryKey = "not-' + + 'a-valid-matrix-recovery-key";' + ) + literal_value = "actual-production-" + "secret" + fixture_shaped_literal = "PROD_TEST_ACTUAL_" + "SECRET_0123456789" + adversarial_label = "TEST_Q7WX9M2NK4PV8R6DH3JC" + unsafe_template = ( + f"const {property_name} = " + + "`prod-live-secret-${randomUUID()}`;" + ) + unsafe_string_template = ( + f"const {property_name} = " + + "`prod-test-live-secret-${String()}`;" + ) + unsafe_suffix_template = ( + f"const {property_name} = " + + "`prod-live-secret-test-${randomUUID()}`;" + ) + unsafe_call = ( + f"const {variable_name} = " + + f'buildToken("{literal_value}");' + ) + unsafe_fixture_label = ( + f"const {property_name} = " + + f'"{fixture_shaped_literal}";' + ) + unsafe_generator_label = ( + f"const {variable_name} = " + + f'buildMatrixQaToken("{fixture_shaped_literal}");' + ) + unsafe_identity_call = ( + f"const {variable_name} = " + + f'buildTestToken("{adversarial_label}");' + ) + + for content in ( + generated_fixture, + generated_marker, + decoy_fixture, + invalid_recovery_fixture, + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + for content in ( + unsafe_template, + unsafe_string_template, + unsafe_suffix_template, + unsafe_call, + unsafe_fixture_label, + unsafe_generator_label, + unsafe_identity_call, + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_fallback_self_test_ignores_ambient_model_overrides(self) -> None: + with mock.patch.dict( + os.environ, + { + "AUTOREVIEW_MODEL": "ambient-global-model", + "AUTOREVIEW_CODEX_MODEL": "ambient-codex-model", + }, + clear=False, + ): + self.helper["self_test_fallback_scope"]() + + def test_secret_detector_handles_bare_call_keyword_values(self) -> None: + content = "client(api_" + "key=" + realistic_secret_value() + ")" + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_handles_unquoted_underscore_tokens(self) -> None: + content = "token=prod_" + realistic_secret_value() + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_dotted_calls(self) -> None: + for content in ( + "token=secrets.token_urlsafe(32)", + "token = provider.issue_token()", + "token = provider?.issue_token()", + "token = generate_secure_token()", + "token = provider.issue_token().access_token", + "token = generate_secure_token().strip()", + "token = provider.issue_token()?.credentials.access_token", + "access_token = retrieve_authentication_token(request)", + 'token = provider.issue_token(scope="review", retries=2)', + "token = provider.issue_token(\n request,\n retries=2,\n)", + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_spaced_calls_without_language_context( + self, + ) -> None: + for content in ( + "pass" + "word = retrieve_authentication_token (request)", + "to" + "ken: retrieve_authentication_token (request)", + "to" + "ken: derivePBKDF2SHA256Hash (request)", + "to" + "ken: acquireOAuth2TokenV2025 (request)", + "to" + "ken: enterpriseOAuth2ClientV123.getToken ()", + 'pass' + 'word = os.getenv ("DATABASE_PASSWORD")', + "to" + "ken = mint_token ()", + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_rejects_ambiguous_bare_values(self) -> None: + for content in ( + "pass" + "word=CORRECTHORSEBATTERYSTAPLE", + "to" + "ken=prod.opaquecredentialvalue", + "to" + "ken=TOKEN_FROM_ENVIRONMENT_SECRET", + "to" + "ken: prod.A7f9K2m4Q8v6N3x5R1p0T9z8 (production)", + "pass" + "word=correct.horse.battery.password", + "pass" + "word=Correct.horse.battery.staple", + "access_" + "token=abcDefGhijk" + "LmnoPqrst", + "pass" + "word=\"${{ 'Correct.horse.battery.staple' }}\"", + "pass" + "word=\"{{ 'Correct.horse.battery.staple' }}\"", + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_does_not_exempt_expression_text_in_literals(self) -> None: + for value in ( + "correct horse + battery staple", + "prefix-${credential}-suffix", + "secret.format(value)", + ): + with self.subTest(value=value): + content = "pass" + f'word="{value}"' + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_handles_lowercase_passphrases(self) -> None: + content = 'password="' + "correcthorsebatterystaple" + '"' + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_handles_low_diversity_passwords(self) -> None: + for content in ( + 'password="' + "letmeinletmein" + '"', + 'password="' + "hunter2!" + '"', + "password=" + "hunter2!", + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_handles_credentialed_uris(self) -> None: + for content in ( + 'url="postgres://' + "user:pass@" + 'db.example/app"', + "DATABASE_URL=postgres://" + "user:pass@" + "db.example/app", + 'url="redis://' + ":secret@" + 'db.example/app"', + 'url="postgres://' + "user:pa$$word@" + 'db.example/app"', + 'url="postgres://' + + "user:fixed-secret:${DB_PASSWORD}@" + + 'db.example/app"', + 'url="postgres://' + "admin:$ecret123@" + 'db.example/app"', + 'url="postgres://' + "admin:${DB_PASSWORD}@" + 'db.example/app"', + 'url="postgres://' + "admin:{password}@" + 'db.example/app"', + 'url="postgres://' + "admin:%s@" + 'db.example/app"', + 'url="postgres://' + "admin:{}@" + 'db.example/app"', + 'url="https://' + "alice@example.com:secret@" + 'host/app"', + 'url="https://admin:pass' + + 'word@prod.example/private"', + "'database.url': 'postgres:" + + "//user:${DB_PASSWORD}@db.example/app'", + "const cfg = {\n" + + ' url: "postgres:' + + '//admin:$ecret123@db.example/app"\n' + + "}", + "const marker = /`/; " + + 'const url = "postgres:' + + '//user:${DB_PASSWORD}@db.example/app"', + "class C { #field = 1; " + + 'url = "postgres:' + + '//user:${DB_PASSWORD}@db.example/app"; }', + "const url = `postgres:" + + '//user:fixed-secret${process.env["SUFFIX"]}@db.example/app`', + 'const url = "https:' + + '//alice:pa\\"ss@example.com/app"', + "const dsn = `postgres:" + + '//user:${String("hunter2!")}@db.example/app`', + 'return "https:' + + '//user:${API_TOKEN}@host/app"', + 'dsn = "postgres:' + + '//user:{password}@db.example/app".format(' + + "pass" + + 'word="hunter2!")', + 'dsn = "postgres:' + + '//user:{}@db.example/app".format("hunter2!")', + 'dsn = "postgres:' + + '//user:%s@db.example/app" % ("hunter2!")', + 'dsn = fmt.Sprintf("postgres:' + + '//user:%s@db.example/app", "hunter2!")', + "DATABASE_URL='" + + "postgres://" + + "admin:$ecret123@db.example/app" + + "'", + '"dsn": "postgresql:\\/\\/alice:' + + "S3nsitiveValue99@" + + 'db.example/app"', + "database_url: postgres://svc:{" + + "N0tActuallyInterpolation}@db/app", + "const dsn = `https://user:password=" + + "real-hardcoded-secret-${TOKEN}@host`", + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_limits_uri_userinfo_to_authority(self) -> None: + for content in ( + 'url="https://example.com:443?email=user@example.org"', + 'url="https://example.com:443#owner=user@example.org"', + 'url="https://example.com:443" + "?email=user@example.org"', + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_secret_detector_handles_username_only_uri_credentials(self) -> None: + literal_username = "real-hardcoded-" + "secret" + hex_credential = "0123456789abcdef" + "0123456789abcdef01234567" + uuid_credential = "550e8400-e29b-41d4-a716-" + "446655440000" + + for content in ( + "https://actual-production-" + + "token@host/repo", + "https://actual-production-" + + "token" + + ":@host/repo", + "https://Ab9dEf2gHi4jKl6m" + "No8p@host/repo", + "https:" + f"//{hex_credential}@host/repo", + "https:" + f"//{uuid_credential}@host/repo", + "https://" + "$ecret123@host/repo", + "https://token=" + "hardcoded123@host/repo", + "DATABASE_URL=https:" + + f"//token={literal_username}:${{PASSWORD}}@host", + 'curl "https:' + + "//Ab9dEf2gHi4jKl6m" + + 'No8p:${PASSWORD}@host"', + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_ordinary_uri_usernames(self) -> None: + for content in ( + "https://git@github.com/example/repo", + "https://username@host/repo", + "https://username:@host/repo", + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_review_patch_preserves_safe_uri_userinfo(self) -> None: + safe_lines = ( + 'url = f"ssh://{ssh_user}@git.example.invalid/org/repo.git"', + 'url = "https://alice@github.com/example/repo"', + 'url = "https://username:@host/repo"', + 'remote = "ssh://git@github.com/org/repo.git"', + ) + for line in safe_lines: + with self.subTest(line=line): + patch = ( + "diff --git a/fixture.py b/fixture.py\n" + "--- a/fixture.py\n" + "+++ b/fixture.py\n" + "@@ -0,0 +1 @@\n" + f"+{line}\n" + ) + + validated = self.helper["validate_review_patch"]( + "local unstaged diff", + ["fixture.py"], + patch, + ) + + self.assertIn(f"+{line}", validated) + self.assertNotIn("redacted@", validated) + + def test_secret_detector_allows_referenced_uri_credentials(self) -> None: + for content in ( + "postgres:" + "//user:password@localhost/db", + "url=postgres:" + "//user:test-token-placeholder@host/db", + "url=postgres:" + "//user:placeholder@host/db", + "url=`postgres://" + "user:${DB_PASSWORD}@db.example/app`", + 'url=f"postgres://' + 'user:{password}@db.example/app"', + 'url=f"""postgres://' + 'user:{password}@db.example/app"""', + 'dsn=f"connect to postgres://' + + 'user:{password}@db.example/app"', + "DATABASE_URL=postgres://" + "user:$DB_PASSWORD@db.example/app", + "DATABASE_URL=postgres:" + "//user:${DB_PASS}@db.example/app", + "DATABASE_URL=https://" + + "$TOKEN" + + ":@host/repo", + "DATABASE_URL=https://" + + "$TOKEN@host/repo", + "DATABASE_URL=https://" + "${TOKEN}@host/repo", + 'curl "https://${API_USER}:' + + '${API_TOKEN}@host/app"', + "DATABASE_URL=https://john.smith." + + "department1:${PASSWORD}@host", + "DATABASE_URL: postgres://" + + "user:${DB_PASSWORD}@db.example/app", + "DATABASE_URL: postgres://" + + "user:$DB_PASSWORD@db.example/app", + 'DATABASE_URL: "postgres://' + + 'user:${DB_PASSWORD}@db.example/app"', + 'DATABASE_URL: "postgres://' + + 'user:${DB_PASS}@db.example/app"', + "DATABASE_URL: postgres://" + "user:${CRED}@db.example/app", + 'DATABASE_URL: "postgres://' + 'user:${AUTH}@db.example/app"', + "url: postgres://" + "user:${CRED}@db.example/app", + "- DATABASE_URL=postgres://" + + "user:${DB_PASSWORD}@db.example/app", + "url: postgres://" + "user:${DB_PASSWORD}@db.example/app", + "uri: postgres://" + "user:${DB_PASSWORD}@db.example/app", + "dsn: postgres://" + "user:${DB_PASSWORD}@db.example/app", + "# DATABASE_URL: postgres://" + + "user:${DB_PASSWORD}@db.example/app", + "# DATABASE_URL=postgres://" + + "user:${DB_PASSWORD}@db.example/app", + '# DATABASE_URL="postgres://' + + 'user:$DB_PASSWORD@db.example/app"', + 'dsn = "postgres://' + + 'user:%s@db.example/app" % password', + 'dsn = fmt.Sprintf("postgres://' + + 'user:%s@db.example/app", password)', + 'dsn = fmt.Sprintf("postgres://' + + '%s:%s@db.example/app", user, password)', + 'dsn = fmt.Sprintf("postgres://' + + 'user:%s@%s/db", password, host)', + 'dsn = "postgres://' + + '%s:%s@db.example/app" % (user, password)', + 'dsn = "postgres://' + + 'user:{}@db.example/app".format(password)', + 'dsn = "postgres://' + + 'user:{}@{}/db".format(password, host)', + 'dsn = "postgres://' + + 'user:{password}@{host}/db".format(password=password, host=host)', + '$"postgres:' + '//user:{password}@db/app"', + 'format!("postgres:' + '//user:{}@db/app", password)', + '$dsn = "postgres:' + '//user:$password@db/app"', + 'export DATABASE_URL="' + + "postgres://" + + "user:${DB_PASSWORD}@db.example/app" + + '"', + 'DATABASE_URL="jdbc:postgresql://' + + "user:$DB_PASSWORD@db.example/app" + + '"', + "url=`postgres://" + + "user:${process.env.DB_PASSWORD}@db.example/app`", + 'url=f"postgres://' + 'user:{config.password}@db.example/app"', + 'url=f"postgres://' + + 'user:{passwords[0]}@db.example/app"', + "url=f'postgres://" + + 'user:{config["password"]}@db.example/app\'', + "// user's config\n" + + "const url = `postgres://" + + "user:${DB_PASSWORD}@db.example/app`", + "const x = this.#field; " + + "const url = `postgres://" + + "user:${DB_PASSWORD}@db.example/app`", + "class C { #field = 1; " + + "url = `postgres://" + + "user:${DB_PASSWORD}@db.example/app`; }", + "const url = `postgres://" + + "user:${passwords[0]}@db.example/app`", + "const url = `postgres://" + + 'user:${passwords["primary"]}@db.example/app`', + "const dsn = `postgres://" + + "user:${encodeURIComponent(process.env.DB_PASSWORD)}@db.example/app`", + 'dsn = "postgres://' + + 'user:{password}@db.example/app".format(' + + "pass" + + "word=password)", + '$env:DATABASE_URL = "postgres://' + + 'svc:$env:DB_PASSWORD@db.example/app"', + '[string]$dsn = "postgres:' + + '//svc:$env:DB_PASSWORD@db.example/app"', + 'var dsn = $@"postgres:' + + '//svc:{password}@db.example/app";', + 'var dsn = @$"postgres:' + + '//svc:{password}@db.example/app";', + '"dsn": "postgresql:\\/\\/alice:' + + '${DB_PASSWORD}@db.example/app"', + '"dsn": "postgresql:\\/\\/user:' + + 'password@localhost\\/db"', + 'curl "https://' + + 'user:${API_TOKEN}@host/app"', + "curl https://" + "user:$API_TOKEN@host/app", + 'curl -X POST "https:' + '//user:$API_TOKEN@host/app"', + 'curl -X POST "https:' + '//user:$CRED@host/app"', + 'wget "https:' + '//user:${API_TOKEN}@host/app"', + 'git clone https:' + '//user:$TOKEN@host/repo', + 'sudo curl "https:' + '//user:$TOKEN@host/app"', + 'http "https:' + '//user:${API_TOKEN}@host/app"', + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_uri_language_references_require_proven_interpolation_context( + self, + ) -> None: + for content in ( + 'const dsn = "postgres:' + + '//svc:$env:DB_PASSWORD@db.example/app"', + '$dsn = "postgres:' + + '//svc:$env:Sup3rSecret@db.example/app";', + 'var dsn = @"postgres:' + + '//svc:{password}@db.example/app";', + "database_url: postgres://svc:{" + + "password}@db.example/app", + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_uri_shell_inference_rejects_non_shell_language_keywords(self) -> None: + for content in ( + 'assert "postgres:' + '//user:$ecret123@db/app"', + 'print "postgres:' + '//user:$ecret123@db/app"', + 'return "postgres:' + '//user:$ecret123@db/app"', + 'const url = "postgres:' + '//user:$ecret123@db/app"', + ): + with self.subTest(content=content): + self.assertTrue( + self.helper["secret_text_risk"](content) + ) + + def test_uri_defaults_and_plain_strings_are_not_interpolation(self) -> None: + for content in ( + "https:" + "//admin:change" + "me@production.example/", + 'url = "https:' + '//admin:$pass' + 'word@prod.example/"', + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_ignores_arrow_parameter_fallbacks(self) -> None: + self.assertFalse( + self.helper["secret_text_risk"]( + 'token => token || "ordinary-option-value"' + ) + ) + + def test_uri_interpolation_rejects_literal_expressions(self) -> None: + self.assertTrue( + self.helper["secret_text_risk"]( + 'dsn = f"postgres:' + '//user:{ \'literal-' + + 'secret\' }@host/db"' + ) + ) + + def test_secret_detector_handles_basic_authorization_headers(self) -> None: + for content in ( + "Author" + "ization: Basic " + "dXNlcjpwYXNz" + "d29yZA==", + "Author" + "ization: Basic " + "dXNlcjpwYXNz" + "CXdvcmQ=", + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_basic_authentication_prose(self) -> None: + for content in ( + "Authorization: Basic authentication is required", + '"Authorization": "Basic authentication is required"', + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_template_uri_references_skip_format_scans(self) -> None: + original = self.helper["uri_password_is_format_placeholder"] + calls = 0 + + def counted(*args: object) -> bool: + nonlocal calls + calls += 1 + return original(*args) + + self.helper["uri_password_is_format_placeholder"] = counted + try: + content = "const urls = `" + " ".join( + "postgres:" + + f"//user:${{PASSWORD_{index}}}@db{index}.example/app" + for index in range(1000) + ) + "`" + self.assertFalse(self.helper["secret_text_risk"](content)) + self.assertEqual(calls, 0) + finally: + self.helper["uri_password_is_format_placeholder"] = original + + def test_format_uri_references_cache_string_boundaries(self) -> None: + quote_end = self.helper["quoted_string_end"] + quote_end.cache_clear() + content = 'dsn = "' + " ".join( + "postgres:" + f"//user:{{0}}@db{index}.example/app" + for index in range(1000) + ) + '".format(password)' + + self.assertFalse(self.helper["secret_text_risk"](content)) + cache_info = quote_end.cache_info() + self.assertEqual(cache_info.misses, 1) + self.assertGreaterEqual(cache_info.hits, 999) + + def test_secret_detector_handles_aws_secret_access_keys(self) -> None: + content = ( + "AWS_SECRET_ACCESS_" + + "KEY=" + + "A7f9K2m4Q8v6N3x5R1p0T9z8B2c4D6e8F0h2" + ) + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_common_fixture_literals(self) -> None: + for content in ( + 'token: "token-oversized"', + 'API_KEY = "clawrouter-e2e-secret"', + 'token: "very-long-browser-token-0123456789"', + 'token: "config-token"', + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_synthetic_secret_fixture_prefixes_are_generic(self) -> None: + for prefix in self.helper["SYNTHETIC_SECRET_PREFIXES"]: + with self.subTest(prefix=prefix): + self.assertTrue( + self.helper["synthetic_secret_fixture"]( + f"{prefix}-token", + "token", + ) + ) + + self.assertFalse( + self.helper["synthetic_secret_fixture"]( + "test-correct-horse-battery-staple", + "password", + ) + ) + + def test_secret_detector_does_not_trust_in_band_suppressions(self) -> None: + for marker in ("pragma: allowlist secret", "gitleaks:allow"): + with self.subTest(marker=marker): + content = ( + "pass" + + 'word="CorrectHorseBatteryStaple123!" # ' + + marker + ) + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_does_not_treat_quoted_code_text_as_a_reference(self) -> None: + for content in ( + "pass" + 'word="' + "CORRECT_HORSE_BATTERY_STAPLE" + '"', + "to" + 'ken="' + "process.env.PROD_TOKEN" + '"', + "api_" + 'key="' + "config.production_key" + '"', + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + self.assertFalse( + self.helper["secret_text_risk"]('api_key="${OPENAI_API_KEY}"') + ) + + def test_secret_detector_does_not_exempt_placeholder_substrings(self) -> None: + content = "pass" + 'word="prod-sample-' + realistic_secret_value() + '"' + + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_openclaw_redaction_sentinel(self) -> None: + self.assertFalse( + self.helper["secret_text_risk"]('token: "__OPENCLAW_REDACTED__"') + ) + + def test_normalized_secret_scan_does_not_cross_hunks(self) -> None: + patch = ( + "@@ -1 +1 @@\n" + "+password:\n" + "@@ -20 +20 @@\n" + '+"ordinary long string"\n' + ) + + self.assertFalse( + any( + self.helper["secret_text_risk"](content) + for content in self.helper["unified_diff_contents"](patch) + ) + ) + + def test_typescript_credential_property_scan_does_not_cross_hunks(self) -> None: + patch = ( + "diff --git a/src/runtime-web-tools.ts b/src/runtime-web-tools.ts\n" + "--- a/src/runtime-web-tools.ts\n" + "+++ b/src/runtime-web-tools.ts\n" + "@@ -85,12 +84,9 @@ type RuntimeWebProviderSelectionParams<\n" + " toolConfig: TToolConfig;\n" + " }) => { path: string; value: unknown } | undefined;\n" + " /** Resolves inline/env/SecretRef credentials and reports the winning source. */\n" + "- resolveSecretInput: (params: {\n" + "- providerId: string;\n" + "- value: unknown;\n" + "- path: string;\n" + "- envVars: string[];\n" + "- }) => Promise>;\n" + "+ resolveSecretInput: (\n" + "+ params: RuntimeWebResolveSecretInputParams,\n" + "+ ) => Promise>;\n" + " /** Writes the selected credential into the resolved runtime config snapshot. */\n" + " setResolvedCredential: (params: {\n" + " resolvedConfig: OpenClawConfig;\n" + "@@ -418,6 +414,7 @@ function resolveRuntimeWebProviderSelection() {\n" + " let keylessFallbackProvider: TProvider | undefined;\n" + " \n" + " for (const provider of candidates) {\n" + "+ const contractDigest = resolveProviderContractDigest(provider.id);\n" + " const isKeyless = provider.requiresCredential === false;\n" + " if (isKeyless) {\n" + " if (!params.configuredProvider && !params.allowKeylessAutoSelect) {\n" + "@@ -440,6 +437,7 @@ function resolveRuntimeWebProviderSelection() {\n" + " value,\n" + " path,\n" + " envVars: getProviderEnvVars(provider),\n" + "+ contractDigest,\n" + " });\n" + " let selectedCandidatePath = path;\n" + " let selectedCandidateResolution = resolution;\n" + "@@ -457,6 +455,7 @@ function resolveRuntimeWebProviderSelection() {\n" + " value: fallback.value,\n" + " path: fallback.path,\n" + " envVars: getProviderEnvVars(provider),\n" + "+ contractDigest,\n" + " });\n" + " }\n" + " } else if (resolution.source === \"env\" && !resolution.secretRefConfigured) {\n" + ) + + old_content, new_content = self.helper["unified_diff_contents"](patch) + self.assertFalse( + self.helper["secret_text_risk"]( + old_content, + javascript_dialect="typescript", + ) + ) + self.assertFalse( + self.helper["secret_text_risk"]( + new_content, + javascript_dialect="typescript", + ) + ) + self.assertEqual( + self.helper["validate_review_patch"]( + "typescript credential property diff", + ["src/runtime-web-tools.ts"], + patch, + ), + patch, + ) + + def test_typescript_hunk_scan_still_flags_sensitive_literal_fixture(self) -> None: + sensitive_line = next( + line + for line in (FIXTURES / "typescript-sensitive-literals.ts") + .read_text(encoding="utf-8") + .splitlines() + if line.strip() + ) + patch = ( + "@@ -1 +1 @@\n" + "+const tokenRef: SecretRef | undefined = candidate.tokenRef;\n" + "@@ -20 +20 @@\n" + f"+{sensitive_line}\n" + ) + + self.assertTrue( + any( + self.helper["secret_text_risk"]( + content, + javascript_dialect="typescript", + ) + for content in self.helper["unified_diff_contents"](patch) + ) + ) + + def test_normalized_secret_scan_handles_combined_diff_prefixes(self) -> None: + value = "Correct-Horse!" + "@Battery$Staple" + patch = ( + "diff --cc settings.json\n" + "@@@ -1,1 -1,1 +1,2 @@@\n" + '++"api_key":\n' + '++ "' + value + '"\n' + ) + + self.assertTrue( + any( + self.helper["secret_text_risk"](content) + for content in self.helper["unified_diff_contents"](patch) + ) + ) + + def test_normalized_secret_scan_separates_old_and_new_values(self) -> None: + value = "Correct-Horse!" + "@Battery$Staple" + patch = ( + "@@ -1,2 +1,2 @@\n" + " password:\n" + "- placeholder\n" + '+ "' + value + '"\n' + ) + + self.assertTrue( + any( + self.helper["secret_text_risk"](content) + for content in self.helper["unified_diff_contents"](patch) + ) + ) + + def test_secret_detector_handles_compound_json_keys(self) -> None: + for key in ("client_secret", "refresh_token"): + content = '{"' + key + '": "' + realistic_secret_value() + '"}' + with self.subTest(key=key): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_safe_self_references(self) -> None: + self.assertFalse( + self.helper["secret_text_risk"]( + "private" + "_key = private_key or fallback_private_key", + javascript_dialect="javascript", + ) + ) + self.assertFalse( + self.helper["secret_text_risk"]( + "old_private" + + "_key = old_private_key or old_line\nnew_private" + + "_key = new_private_key or new_line", + javascript_dialect="javascript", + ) + ) + colon_assignment = self.helper["SECRET_ASSIGNMENT_PATTERN"].search( + "pass" + "word: password" + ) + self.assertIsNotNone(colon_assignment) + self.assertFalse( + self.helper["safe_self_reference_assignment"]( + "pass" + "word: password", + colon_assignment, + javascript_dialect="javascript", + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + "client" + "secret" + "=" + "clientsecret" + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + "if (ready) send({pass" + + 'word: "' + + realistic_secret_value() + + '"})' + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + "private" + + '_key = private_key or "' + + realistic_secret_value() + + '"' + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + "pass" + + 'word = password\n || "' + + realistic_secret_value() + + '"', + javascript_dialect="javascript", + ) + ) + for trivia in ("\n\n ", "\n /* comment */\n ", " // comment\n "): + with self.subTest(trivia=trivia): + self.assertTrue( + self.helper["secret_text_risk"]( + "pass" + + "word = password" + + trivia + + '|| "' + + realistic_secret_value() + + '"', + javascript_dialect="javascript", + ) + ) + + def test_review_patch_preserves_redaction_placeholder_fallback(self) -> None: + patch = ( + "diff --git a/runtime.py b/runtime.py\n" + "--- a/runtime.py\n" + "+++ b/runtime.py\n" + "@@ -0,0 +1 @@\n" + + "+pass" + + 'word = getenv("PASSWORD") or "redacted"\n' + ) + + self.assertEqual( + self.helper["validate_review_patch"]( + "local unstaged diff", + ["runtime.py"], + patch, + ), + patch, + ) + + def test_review_patch_preserves_ambiguous_short_markerless_lines(self) -> None: + chunks = ["AB12", "CDef", "GH34", "ijKL", "MN56", "opQR"] + patch = ( + "diff --git a/fixture.txt b/fixture.txt\n" + "--- a/fixture.txt\n" + "+++ b/fixture.txt\n" + f"@@ -0,0 +1,{len(chunks)} @@\n" + + "".join(f"+{chunk}\n" for chunk in chunks) + ) + + redacted_patch = self.helper["validate_review_patch"]( + "local unstaged diff", + ["fixture.txt"], + patch, + ) + + self.assertEqual(redacted_patch, patch) + + def test_review_patch_preserves_long_non_pem_identifier_lines(self) -> None: + identifier = "runDangerousOperationWithLongIdentifier" + patch = ( + "diff --git a/runtime.ts b/runtime.ts\n" + "--- a/runtime.ts\n" + "+++ b/runtime.ts\n" + "@@ -0,0 +1 @@\n" + + f"+{identifier}\n" + ) + + redacted = self.helper["validate_review_patch"]( + "local unstaged diff", + ["runtime.ts"], + patch, + ) + + self.assertIn("+" + identifier, redacted) + + def test_review_patch_preserves_hash_and_submodule_lines(self) -> None: + digest = "abcdef0123456789abcdef0123456789abcdef01" + patch = ( + "diff --git a/vendor b/vendor\n" + "--- a/vendor\n" + "+++ b/vendor\n" + "@@ -1 +1,2 @@\n" + + f"+{digest}\n" + + f"+Subproject commit {digest}\n" + ) + + redacted = self.helper["validate_review_patch"]( + "local unstaged diff", + ["vendor"], + patch, + ) + + self.assertIn("+" + digest, redacted) + self.assertIn("+Subproject commit " + digest, redacted) + + def test_review_patch_preserves_unwrapped_alphabetic_identifier(self) -> None: + identifier = "AbCdEfGh" + "IjKlMnOp" + patch = ( + "diff --git a/runtime.ts b/runtime.ts\n" + "--- a/runtime.ts\n" + "+++ b/runtime.ts\n" + "@@ -0,0 +1 @@\n" + + f"+const {identifier} = true;\n" + ) + + redacted_patch = self.helper["validate_review_patch"]( + "local unstaged diff", + ["runtime.ts"], + patch, + ) + + self.assertIn(identifier, redacted_patch) + + def test_review_patch_preserves_punctuation_wrapped_alphabetic_identifier(self) -> None: + identifier = "AbCdEfGh" + "IjKlMnOp" + patch = ( + "diff --git a/runtime.ts b/runtime.ts\n" + "--- a/runtime.ts\n" + "+++ b/runtime.ts\n" + "@@ -0,0 +1 @@\n" + + f"+ {identifier},\n" + ) + + redacted_patch = self.helper["validate_review_patch"]( + "local unstaged diff", + ["runtime.ts"], + patch, + ) + + self.assertIn(identifier, redacted_patch) + + def test_review_patch_preserves_escaped_newline_beside_alphabetic_identifier(self) -> None: + identifier = "AbCdEfGh" + "IjKlMnOp" + patch = ( + "diff --git a/runtime.ts b/runtime.ts\n" + "--- a/runtime.ts\n" + "+++ b/runtime.ts\n" + "@@ -0,0 +1 @@\n" + + f'+[{identifier}, "\\\\n"];\n' + ) + + redacted_patch = self.helper["validate_review_patch"]( + "local unstaged diff", + ["runtime.ts"], + patch, + ) + + self.assertIn(identifier, redacted_patch) + + def test_review_patch_preserves_bare_identifier_in_escaped_pem_concatenation(self) -> None: + identifier = "AbCdEfGh" + "IjKlMnOp" + patch = ( + "diff --git a/runtime.ts b/runtime.ts\n" + "--- a/runtime.ts\n" + "+++ b/runtime.ts\n" + "@@ -0,0 +1 @@\n" + '+const fixture = "-----BEGIN ' + + "PRIVATE KEY-----\\n\" + " + + identifier + + ' + "\\n-----END ' + + 'PRIVATE KEY-----";\n' + ) + + redacted_patch = self.helper["validate_review_patch"]( + "local unstaged diff", + ["runtime.ts"], + patch, + ) + + self.assertIn(identifier, redacted_patch) + + def test_local_bundle_allows_deleted_test_token_fixture(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + path = repo / "fixture.test.ts" + path.write_text('const request = { token: "test-token" };\n', encoding="utf-8") + git(repo, "add", path.name) + git(repo, "commit", "-q", "-m", "base") + + path.write_text('const request = { token: String() };\n', encoding="utf-8") + + bundle, truncated = self.helper["local_bundle"](repo) + + self.assertIn('-const request = { token: "test-token" };', bundle) + self.assertFalse(truncated) + + def test_pi_refuses_truncated_review_input(self) -> None: + reviewer = argparse.Namespace(engine="pi", tools=True) + + with self.assertRaisesRegex(SystemExit, "pi engine refused truncated review input"): + self.helper["ensure_reviewer_input_complete"]( + reviewer, + True, + ) + + self.helper["ensure_reviewer_input_complete"]( + reviewer, + False, + ) + with self.assertRaisesRegex(SystemExit, "codex engine refused truncated review input"): + self.helper["ensure_reviewer_input_complete"]( + argparse.Namespace(engine="codex", tools=True), + True, + ) + with self.assertRaisesRegex(SystemExit, "claude engine refused truncated review input"): + self.helper["ensure_reviewer_input_complete"]( + argparse.Namespace(engine="claude", tools=True), + True, + ) + with self.assertRaisesRegex(SystemExit, "droid engine refused truncated review input"): + self.helper["ensure_reviewer_input_complete"]( + argparse.Namespace(engine="droid", tools=False), + True, + ) + + def test_safe_git_env_preserves_trusted_platform_and_helper_paths(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + repo_bin = repo / "bin" + trusted_bin = root / "trusted-bin" + repo_bin.mkdir() + trusted_bin.mkdir() + with mock.patch.dict( + os.environ, + { + "PATH": os.pathsep.join((str(repo_bin), str(trusted_bin))), + "SYSTEMROOT": "C:\\Windows", + "GIT_DIR": str(repo / ".git"), + "OPENAI_API_KEY": "must-not-reach-git", + }, + clear=False, + ): + env = self.helper["safe_git_env"](repo) + + self.assertNotIn(str(repo_bin.resolve()), env["PATH"].split(os.pathsep)) + self.assertIn(str(trusted_bin.resolve()), env["PATH"].split(os.pathsep)) + self.assertEqual(env["SYSTEMROOT"], "C:\\Windows") + self.assertNotIn("GIT_DIR", env) + self.assertNotIn("OPENAI_API_KEY", env) + + def test_boolean_environment_values_fail_closed(self) -> None: + with mock.patch.dict(os.environ, {"AUTOREVIEW_TEST_BOOL": "flase"}): + with self.assertRaisesRegex(SystemExit, "invalid boolean environment value"): + self.helper["env_truthy"]("AUTOREVIEW_TEST_BOOL") + + def test_droid_fails_closed_without_complete_isolation(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + (repo / "AGENTS.md").write_text("hostile instructions\n", encoding="utf-8") + + with self.assertRaisesRegex( + SystemExit, + r"droid engine is unavailable.*use codex, claude, or pi", + ) as error: + self.helper["run_droid"](argparse.Namespace(), repo, "prompt") + self.assertNotIn("opencode", str(error.exception)) + + def test_prompt_file_keeps_recoverable_repo_path(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + (repo / "review.md").write_text("review context\n", encoding="utf-8") + args = argparse.Namespace(prompt=[], prompt_file=["review.md"]) + + prompt, truncated = self.helper["load_extra_prompt"](args, repo) + + self.assertIn("# Prompt file: review.md", prompt) + self.assertFalse(truncated) + + def test_build_prompt_omits_absolute_repo_path_and_caps_aggregate_input(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + prompt = self.helper["build_prompt"](repo, "local", None, "diff", "", "") + + self.assertIn( + "Review sandbox: . (intentionally contains no reviewed repository files)", + prompt, + ) + self.assertIn("Read-only tools cannot access unchanged repository files", prompt) + self.assertIn( + "Do not report a missing import, symbol, definition, call site, config entry", + prompt, + ) + self.assertNotIn(str(repo), prompt) + with self.assertRaisesRegex(SystemExit, "aggregate limit"): + self.helper["build_prompt"]( + repo, + "local", + None, + "x" * self.helper["MAX_REVIEW_PROMPT_BYTES"], + "", + "", + ) + + def test_cursor_refuses_global_mcp_config(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + global_mcp = root / ".cursor" / "mcp.json" + global_mcp.parent.mkdir() + global_mcp.write_text("{}\n", encoding="utf-8") + args = argparse.Namespace( + thinking=None, + tools=True, + web_search=True, + cursor_allow_workspace_instructions=True, + ) + + with mock.patch.object(Path, "home", return_value=root), mock.patch.dict( + os.environ, + {"HOME": str(root), "USERPROFILE": str(root)}, + ): + with self.assertRaisesRegex(SystemExit, "cursor engine is unavailable"): + self.helper["run_cursor"](args, repo, "prompt") + + def test_cursor_refuses_user_level_hooks(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + settings = root / ".claude" / "settings.json" + settings.parent.mkdir() + settings.write_text('{"hooks":{"PreToolUse":[{"command":"unsafe"}]}}\n', encoding="utf-8") + args = argparse.Namespace( + thinking=None, + tools=True, + web_search=True, + cursor_allow_workspace_instructions=True, + ) + + with mock.patch.object(Path, "home", return_value=root), mock.patch.dict( + os.environ, + {"HOME": str(root), "USERPROFILE": str(root)}, + ): + with self.assertRaisesRegex(SystemExit, "cursor engine is unavailable"): + self.helper["run_cursor"](args, repo, "prompt") + + settings.write_text('{"permissions":{"allow":["Read(**)"]}}\n', encoding="utf-8") + with mock.patch.object(Path, "home", return_value=root), mock.patch.dict( + os.environ, + {"HOME": str(root), "USERPROFILE": str(root)}, + ): + self.assertEqual(self.helper["cursor_global_hook_paths"](), []) + + settings.write_text('{"enabledPlugins":{"review-hooks@example":true}}\n', encoding="utf-8") + with mock.patch.object(Path, "home", return_value=root), mock.patch.dict( + os.environ, + {"HOME": str(root), "USERPROFILE": str(root)}, + ): + self.assertEqual(self.helper["cursor_global_hook_paths"](), [settings]) + + def test_read_text_truncates_without_scanning_tail(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + path = Path(tempdir) / "large.txt" + path.write_bytes(b"x" * 200_000 + b"\0tail") + + text = self.helper["read_text"](path) + + self.assertIn("[truncated at 180000 characters]", text) + self.assertNotEqual(text, "[binary file omitted]") + + def test_read_text_marks_unreadable_input_incomplete(self) -> None: + with mock.patch.dict( + self.helper["read_text_with_status"].__globals__, + {"read_prefix": lambda *_args: (_ for _ in ()).throw(SystemExit("denied"))}, + ): + text, incomplete = self.helper["read_text_with_status"](Path("blocked")) + + self.assertIn("[unreadable:", text) + self.assertTrue(incomplete) + + def test_evidence_file_must_be_repo_relative_and_not_symlinked(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + outside = root / "outside.md" + outside.write_text("outside\n", encoding="utf-8") + + with self.assertRaisesRegex(SystemExit, "repo-relative"): + self.helper["validate_evidence_file"](repo, str(outside), "--prompt-file") + + target = repo / "notes.md" + target.write_text("notes\n", encoding="utf-8") + link = repo / "link.md" + try: + link.symlink_to(target) + except OSError as exc: + if os.name == "nt" and getattr(exc, "winerror", None) == 1314: + self.skipTest("Windows symlink privilege is not available") + raise + with self.assertRaisesRegex(SystemExit, "symlinked"): + self.helper["validate_evidence_file"](repo, "link.md", "--dataset") + + def test_safe_engine_env_strips_process_injection_variables(self) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + try: + os.environ["GIT_DIR"] = "/tmp/unsafe-git-dir" + os.environ["GIT_CONFIG_COUNT"] = "99" + os.environ["DYLD_INSERT_LIBRARIES"] = "/tmp/unsafe.dylib" + os.environ["NODE_OPTIONS"] = "--require=/tmp/unsafe.js" + os.environ["NODE_PATH"] = "/tmp/unsafe-node" + os.environ["LD_AUDIT"] = "/tmp/unsafe-audit.so" + os.environ["LD_LIBRARY_PATH"] = "/tmp/unsafe-lib" + os.environ["RUBYOPT"] = "-r/tmp/unsafe.rb" + os.environ["PERL5OPT"] = "-Munsafe" + os.environ["BUN_OPTIONS"] = "--preload=/tmp/unsafe.js" + os.environ["OPENCODE_CONFIG"] = "/tmp/unsafe-opencode.json" + os.environ["OPENCODE_PERMISSION"] = "allow" + os.environ["OPENCODE_AUTO_SHARE"] = "1" + os.environ["COPILOT_ALLOW_ALL"] = "1" + os.environ["CODEX_HOME"] = "/tmp/codex-auth" + os.environ["DBUS_SESSION_BUS_ADDRESS"] = "unix:path=/run/user/1000/bus" + os.environ["XDG_RUNTIME_DIR"] = "/run/user/1000" + os.environ["CLAUDE_CONFIG_DIR"] = "/tmp/claude-auth" + os.environ["PI_CODING_AGENT_DIR"] = "/tmp/pi-auth" + os.environ["CLAUDE_CODE_USE_FOUNDRY"] = "1" + os.environ["CLOUD_ML_REGION"] = "us-east5" + os.environ["ANTHROPIC_AUTH_TOKEN"] = "test-auth-token" + os.environ["AWS_BEARER_TOKEN_BEDROCK"] = "test-token-placeholder" + os.environ["ANTHROPIC_BEDROCK_BASE_URL"] = ( + "https://bedrock.example.invalid" + ) + os.environ["ANTHROPIC_VERTEX_BASE_URL"] = ( + "https://vertex.example.invalid" + ) + os.environ["AWS_PROFILE"] = "review-profile" + os.environ["AWS_CONFIG_FILE"] = "/tmp/unsafe-aws-config" + os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = ( + "/tmp/unsafe-google-credentials" + ) + os.environ["GOOGLE_EXTERNAL_ACCOUNT_ALLOW_EXECUTABLES"] = "1" + os.environ["OPENROUTER_API_KEY"] = "test-provider-key" + os.environ["GITHUB_TOKEN"] = "test-token-placeholder" + os.environ["HTTPS_PROXY"] = "http://proxy.example.invalid:8080" + os.environ["HTTP_PROXY"] = "proxy.example.invalid:8080" + os.environ["ALL_PROXY"] = "socks5://proxy.example.invalid:1080" + os.environ["DO_NOT_TRACK"] = "1" + os.environ["DISABLE_TELEMETRY"] = "1" + os.environ["CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC"] = "1" + + env = self.helper["safe_engine_env"](repo, engine="codex") + claude_env = self.helper["safe_engine_env"](repo, engine="claude") + pi_env = self.helper["safe_engine_env"](repo, engine="pi") + + self.assertNotEqual(env.get("GIT_DIR"), "/tmp/unsafe-git-dir") + self.assertEqual( + env["GIT_CONFIG_COUNT"], + str(len(self.helper["ENGINE_GIT_CONFIG_OVERRIDES"])), + ) + self.assertNotIn("DYLD_INSERT_LIBRARIES", env) + self.assertNotIn("NODE_OPTIONS", env) + for key in ( + "NODE_PATH", + "LD_AUDIT", + "LD_LIBRARY_PATH", + "RUBYOPT", + "PERL5OPT", + "BUN_OPTIONS", + "OPENCODE_CONFIG", + "OPENCODE_PERMISSION", + "OPENCODE_AUTO_SHARE", + ): + self.assertNotIn(key, env) + self.assertNotIn("COPILOT_ALLOW_ALL", env) + self.assertNotIn("GITHUB_TOKEN", env) + self.assertEqual(env["HTTPS_PROXY"], "http://proxy.example.invalid:8080") + self.assertEqual(env["HTTP_PROXY"], "proxy.example.invalid:8080") + self.assertEqual(env["ALL_PROXY"], "socks5://proxy.example.invalid:1080") + self.assertEqual(env["DO_NOT_TRACK"], "1") + self.assertEqual(env["DISABLE_TELEMETRY"], "1") + self.assertEqual(env["CODEX_HOME"], "/tmp/codex-auth") + if os.name == "nt": + self.assertNotIn("DBUS_SESSION_BUS_ADDRESS", env) + else: + self.assertEqual( + env["DBUS_SESSION_BUS_ADDRESS"], + "unix:path=/run/user/1000/bus", + ) + self.assertEqual(env["XDG_RUNTIME_DIR"], "/run/user/1000") + self.assertEqual( + claude_env["CLAUDE_CONFIG_DIR"], + "/tmp/claude-auth", + ) + self.assertEqual( + claude_env["CLAUDE_CODE_DISABLE_AUTO_MEMORY"], + "1", + ) + self.assertEqual(pi_env["PI_CODING_AGENT_DIR"], "/tmp/pi-auth") + self.assertEqual(claude_env["CLAUDE_CODE_USE_FOUNDRY"], "1") + self.assertEqual(claude_env["CLOUD_ML_REGION"], "us-east5") + self.assertEqual( + claude_env["ANTHROPIC_AUTH_TOKEN"], + "test-auth-token", + ) + self.assertEqual( + claude_env["AWS_BEARER_TOKEN_BEDROCK"], + "test-token-placeholder", + ) + self.assertEqual( + claude_env["ANTHROPIC_BEDROCK_BASE_URL"], + "https://bedrock.example.invalid", + ) + self.assertEqual( + claude_env["ANTHROPIC_VERTEX_BASE_URL"], + "https://vertex.example.invalid", + ) + self.assertEqual(claude_env["AWS_PROFILE"], "review-profile") + self.assertNotIn("AWS_CONFIG_FILE", env) + self.assertNotIn("GOOGLE_APPLICATION_CREDENTIALS", env) + self.assertNotIn( + "GOOGLE_EXTERNAL_ACCOUNT_ALLOW_EXECUTABLES", + env, + ) + self.assertNotIn("OPENROUTER_API_KEY", env) + self.assertEqual( + claude_env["CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC"], + "1", + ) + finally: + os.environ.clear() + os.environ.update(old) + + def test_parallel_tests_use_sanitized_environment_for_every_shell(self) -> None: + observed: list[dict[str, object]] = [] + sanitized_env = { + "PATH": "/usr/bin", + "HOME": "/safe/home", + "JAVA_TOOL_OPTIONS": "'-Duser.home=/safe/home'", + } + + def fake_popen(command: object, **kwargs: object) -> mock.Mock: + observed.append({"command": command, **kwargs}) + proc = mock.Mock() + proc.returncode = 0 + proc.stderr = io.StringIO("") + return proc + + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + with mock.patch.dict( + self.helper["start_parallel_tests"].__globals__, + { + "safe_test_env": lambda actual_repo, test_home: ( + sanitized_env + if actual_repo == repo and not test_home.is_relative_to(repo) + else self.fail("parallel tests sanitized the wrong repository") + ), + "resolve_command": lambda name, actual_repo: ( + f"/usr/bin/{name}" + if actual_repo == repo + else self.fail("parallel tests resolved a shell for the wrong repository") + ), + }, + ), mock.patch("subprocess.Popen", side_effect=fake_popen): + for shell_kind in ("default", "cmd", "powershell", "pwsh"): + proc, started = self.helper["start_parallel_tests"]( + "run tests", repo, shell_kind + ) + test_home = getattr(proc, "_autoreview_test_home") + self.assertTrue(test_home.is_dir()) + self.helper["finish_parallel_tests"](proc, started) + self.assertFalse(test_home.exists()) + + self.assertEqual(len(observed), 4) + for invocation in observed: + self.assertEqual(invocation["cwd"], repo) + self.assertEqual(invocation["env"], sanitized_env) + self.assertEqual(invocation["stderr"], subprocess.PIPE) + self.assertTrue(invocation["text"]) + self.assertTrue(observed[0]["shell"]) + self.assertTrue(observed[1]["shell"]) + self.assertNotIn("shell", observed[2]) + self.assertNotIn("shell", observed[3]) + + def test_parallel_test_finish_does_not_wait_for_inherited_stderr_pipe( + self, + ) -> None: + release = threading.Event() + stderr_thread = threading.Thread(target=release.wait, daemon=True) + stderr_thread.start() + try: + with tempfile.TemporaryDirectory() as tempdir: + test_home = Path(tempdir) / "test-home" + test_home.mkdir() + proc = mock.Mock() + proc.returncode = 0 + proc.wait.return_value = 0 + setattr(proc, "_autoreview_test_home", test_home) + setattr(proc, "_autoreview_stderr_thread", stderr_thread) + + started = time.time() + before = time.monotonic() + result = self.helper["finish_parallel_tests"](proc, started) + elapsed = time.monotonic() - before + + self.assertEqual(result, 0) + self.assertLess(elapsed, 1) + self.assertFalse(test_home.exists()) + finally: + release.set() + stderr_thread.join(timeout=1) + + def test_source_tree_snapshot_detects_parallel_test_mutations(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + source = repo / "source.txt" + source.write_text("before\n", encoding="utf-8") + git(repo, "add", "source.txt") + git(repo, "commit", "-qm", "initial") + before = self.helper["source_tree_snapshot"](repo) + + source.write_text("after\n", encoding="utf-8") + self.assertNotEqual( + self.helper["source_tree_snapshot"](repo), + before, + ) + source.write_text("before\n", encoding="utf-8") + self.assertEqual( + self.helper["source_tree_snapshot"](repo), + before, + ) + + source.write_text("after\n", encoding="utf-8") + git(repo, "add", "source.txt") + git(repo, "commit", "-qm", "mutated") + self.assertNotEqual( + self.helper["source_tree_snapshot"](repo), + before, + ) + + (repo / "generated.txt").write_text("generated\n", encoding="utf-8") + self.assertNotEqual( + self.helper["source_tree_snapshot"](repo), + before, + ) + + def test_rejects_output_paths_inside_reviewed_repository(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + outside = root / "outside.json" + + with self.assertRaisesRegex( + SystemExit, + "--json-output must point outside", + ): + self.helper["reject_repo_output_paths"]( + argparse.Namespace( + json_output=str(repo / "review.json"), + output=None, + ), + repo, + ) + with self.assertRaisesRegex( + SystemExit, + "--output must point outside", + ): + self.helper["reject_repo_output_paths"]( + argparse.Namespace( + json_output=None, + output=str(repo / "review.txt"), + ), + repo, + ) + + self.helper["reject_repo_output_paths"]( + argparse.Namespace( + json_output=str(outside), + output=None, + ), + repo, + ) + alternate_repo = repo.with_name(repo.name.swapcase()) + with ( + mock.patch.object( + os.path, + "samefile", + side_effect=lambda left, right: ( + str(left).casefold() == str(right).casefold() + ), + ), + self.assertRaisesRegex( + SystemExit, + "--json-output must point outside", + ), + ): + self.helper["reject_repo_output_paths"]( + argparse.Namespace( + json_output=str(alternate_repo / "review.json"), + output=None, + ), + repo, + ) + + def test_atomic_output_replaces_hard_link_without_touching_repo_file( + self, + ) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + tracked = repo / "tracked.txt" + tracked.write_text("tracked\n", encoding="utf-8") + outside = root / "review.txt" + os.link(tracked, outside) + + self.helper["atomic_write_text"](outside, "review\n") + + self.assertEqual( + tracked.read_text(encoding="utf-8"), + "tracked\n", + ) + self.assertEqual( + outside.read_text(encoding="utf-8"), + "review\n", + ) + self.assertFalse(os.path.samefile(tracked, outside)) + + def test_partial_panel_failure_output_is_terminal_escaped(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + reviewers = [ + argparse.Namespace( + engine="codex", + model=None, + fallback_model=None, + thinking=None, + ), + argparse.Namespace( + engine="claude", + model=None, + fallback_model=None, + thinking=None, + ), + ] + args = argparse.Namespace( + allow_partial_panel=True, + require_finding=[], + ) + report = { + "findings": [], + "overall_correctness": "patch is correct", + "overall_explanation": "clean", + "overall_confidence": 0.9, + } + + def run_reviewer(reviewer: argparse.Namespace, *_args: object) -> object: + if reviewer.engine == "claude": + raise RuntimeError( + "\x1b]8;;https://example.invalid\x07click" + "\x1b]8;;\x07" + ) + return report + + stdout = io.StringIO() + with ( + mock.patch.dict( + self.helper["run_panel"].__globals__, + {"run_reviewer": run_reviewer}, + ), + contextlib.redirect_stdout(stdout), + ): + self.helper["run_panel"]( + args, + reviewers, + repo, + "prompt", + set(), + False, + ) + + output = stdout.getvalue() + self.assertNotIn("\x1b", output) + self.assertNotIn("\x07", output) + self.assertIn("\\x1b]8;;", output) + self.assertIn("\\x07", output) + + def test_fatal_panel_failure_output_is_terminal_escaped(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + reviewers = [ + argparse.Namespace( + engine="codex", + model=None, + fallback_model=None, + thinking=None, + ) + ] + args = argparse.Namespace( + allow_partial_panel=False, + require_finding=[], + ) + + def run_reviewer(*_args: object) -> object: + raise RuntimeError("\x1b]8;;https://example.invalid\x07click") + + with ( + mock.patch.dict( + self.helper["run_panel"].__globals__, + {"run_reviewer": run_reviewer}, + ), + self.assertRaises(SystemExit) as error, + ): + self.helper["run_panel"]( + args, + reviewers, + repo, + "prompt", + set(), + False, + ) + + message = str(error.exception) + self.assertNotIn("\x1b", message) + self.assertNotIn("\x07", message) + self.assertIn("\\x1b]8;;", message) + self.assertIn("\\x07", message) + + def test_source_tree_snapshot_supports_staged_files_before_first_commit( + self, + ) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + source = repo / "source.txt" + source.write_text("before\n", encoding="utf-8") + git(repo, "add", "source.txt") + + before = self.helper["source_tree_snapshot"](repo) + symbolic_head = git(repo, "symbolic-ref", "HEAD").strip() + self.assertEqual(before[0], f"unborn:{symbolic_head}") + + git(repo, "symbolic-ref", "HEAD", "refs/heads/other") + self.assertNotEqual( + self.helper["source_tree_snapshot"](repo), + before, + ) + git(repo, "symbolic-ref", "HEAD", symbolic_head) + + source.write_text("after\n", encoding="utf-8") + self.assertNotEqual( + self.helper["source_tree_snapshot"](repo), + before, + ) + + @unittest.skipIf(os.name == "nt", "the true command is POSIX-only") + def test_cli_parallel_tests_supports_unborn_repository(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + source = repo / "source.txt" + source.write_text("staged\n", encoding="utf-8") + git(repo, "add", "source.txt") + codex_bin = self.helper["write_executable"]( + root / "codex", + self.helper["fake_codex_script"](), + ) + record_path = root / "record.json" + env = os.environ.copy() + add_fake_trufflehog(self.helper, root, env) + env.update( + { + "AUTOREVIEW_FAKE_RECORD": str(record_path), + "HOME": str(root), + "USERPROFILE": str(root), + } + ) + + result = subprocess.run( + [ + sys.executable, + str(SCRIPT), + "--mode", + "local", + "--engine", + "codex", + "--codex-bin", + str(codex_bin), + "--parallel-tests", + "true", + ], + cwd=repo, + env=env, + text=True, + capture_output=True, + check=False, + ) + + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("autoreview clean", result.stdout) + + @unittest.skipIf(os.name == "nt", "the fake executable is POSIX-only") + def test_cli_detects_source_mutation_without_parallel_tests(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + source = repo / "source.txt" + source.write_text("before\n", encoding="utf-8") + git(repo, "add", "source.txt") + git(repo, "commit", "-qm", "initial") + source.write_text("review me\n", encoding="utf-8") + codex_bin = self.helper["write_executable"]( + root / "codex", + self.helper["fake_codex_script"](), + ) + record_path = root / "record.json" + env = os.environ.copy() + add_fake_trufflehog(self.helper, root, env) + env.update( + { + "AUTOREVIEW_FAKE_MUTATE": str(source), + "AUTOREVIEW_FAKE_RECORD": str(record_path), + "HOME": str(root), + "USERPROFILE": str(root), + } + ) + + result = subprocess.run( + [ + sys.executable, + str(SCRIPT), + "--mode", + "local", + "--engine", + "codex", + "--codex-bin", + str(codex_bin), + ], + cwd=repo, + env=env, + text=True, + capture_output=True, + check=False, + ) + + self.assertEqual(result.returncode, 1, result.stdout) + self.assertIn( + "source changed after the review bundle was created", + result.stderr, + ) + self.assertTrue(record_path.is_file()) + + def test_source_tree_snapshot_hashes_binary_and_untracked_tail_bytes( + self, + ) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + tracked = repo / "tracked.bin" + tracked.write_bytes(b"\0tracked-before") + git(repo, "add", "tracked.bin") + git(repo, "commit", "-qm", "initial") + limit = self.helper["MAX_BUNDLE_TEXT_BYTES"] + untracked = repo / "generated.bin" + untracked.write_bytes(b"\0" + b"a" * (limit + 16)) + before = self.helper["source_tree_snapshot"](repo) + + tracked.write_bytes(b"\0tracked-after!") + self.assertNotEqual( + self.helper["source_tree_snapshot"](repo), + before, + ) + tracked.write_bytes(b"\0tracked-before") + self.assertEqual( + self.helper["source_tree_snapshot"](repo), + before, + ) + + with untracked.open("r+b") as stream: + stream.seek(-1, os.SEEK_END) + stream.write(b"b") + self.assertNotEqual( + self.helper["source_tree_snapshot"](repo), + before, + ) + + def test_source_tree_snapshot_includes_index_state(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + source = repo / "source.txt" + source.write_text("before\n", encoding="utf-8") + git(repo, "add", "source.txt") + git(repo, "commit", "-qm", "initial") + before = self.helper["source_tree_snapshot"](repo) + + source.write_text("staged\n", encoding="utf-8") + git(repo, "add", "source.txt") + source.write_text("before\n", encoding="utf-8") + self.assertNotEqual( + self.helper["source_tree_snapshot"](repo), + before, + ) + + def test_source_tree_snapshot_includes_tracked_submodule_contents(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + child = root / "child" + child.mkdir() + git(child, "init", "-q") + source = child / "source.txt" + source.write_text("before\n", encoding="utf-8") + git(child, "add", "source.txt") + git(child, "commit", "-qm", "initial") + + repo = init_repo(root) + git( + repo, + "-c", + "protocol.file.allow=always", + "submodule", + "add", + "-q", + str(child), + "vendor/dependency", + ) + git(repo, "commit", "-qam", "add submodule") + before = self.helper["source_tree_snapshot"](repo) + + (repo / "vendor/dependency/source.txt").write_text( + "after\n", + encoding="utf-8", + ) + self.assertNotEqual( + self.helper["source_tree_snapshot"](repo), + before, + ) + + def test_trusted_maintainer_testbox_preserves_only_credentials(self) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + isolated_home = root / "test-home" + host_home = root / "host-home" + rustup_home = host_home / ".rustup" + rustup_home.mkdir(parents=True) + blacksmith_home = host_home / ".blacksmith" + blacksmith_home.mkdir() + blacksmith_credentials = blacksmith_home / "credentials" + blacksmith_credentials.write_bytes(b"test-blacksmith-credentials") + (blacksmith_home / "unrelated-state").write_text( + "do not copy", + encoding="utf-8", + ) + local_bin = repo / ".venv" / "bin" + local_bin.mkdir(parents=True) + try: + os.environ["PATH"] = f"{local_bin}{os.pathsep}/usr/bin" + os.environ["CI"] = "1" + os.environ["GRADLE_USER_HOME"] = "/host/gradle" + os.environ["HOME"] = str(host_home) + os.environ["JAVA_HOME"] = "/opt/jdk" + os.environ["JAVA_TOOL_OPTIONS"] = "-javaagent:/host/unsafe.jar" + os.environ["NODE_ENV"] = "test" + os.environ["OPENCLAW_TESTBOX"] = "1" + os.environ["PROJECT_FEATURE_MODE"] = "strict" + os.environ["GH_CONFIG_DIR"] = "/host/gh" + os.environ["CLOUDSDK_CONFIG"] = "/host/gcloud" + os.environ["XDG_CONFIG_HOME"] = "/host/xdg" + os.environ["GITHUB_TOKEN"] = "test-token-placeholder" + os.environ["AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE"] = ( + "/host/aws-token" + ) + os.environ["AZURE_FEDERATED_TOKEN_FILE"] = "/host/azure-token" + os.environ["CI_JOB_JWT"] = "header.payload.signature" + os.environ["DOCKER_AUTH_CONFIG"] = '{"auths":{"registry":{}}}' + os.environ["PGPASSFILE"] = "/host/pgpass" + os.environ["PGPASSWORD"] = "short-password" + os.environ["REDISCLI_AUTH"] = "short-password" + os.environ["BASH_FUNC_testcmd%%"] = "() { echo injected; }" + os.environ["SHELLOPTS"] = "xtrace" + os.environ["NODE_OPTIONS"] = "--require=/tmp/unsafe.js" + os.environ["SERVICE_URL"] = ( + "https://review-user:review-password@example.invalid/api" + ) + os.environ["UNRELATED_VALUE"] = "ghp_" + "A" * 24 + + env = self.helper["safe_test_env"](repo, isolated_home) + + self.assertEqual(env["PATH"], os.environ["PATH"]) + self.assertEqual(env["CI"], "1") + self.assertEqual( + env["GRADLE_USER_HOME"], + str((isolated_home / ".gradle").resolve()), + ) + self.assertEqual(env["JAVA_HOME"], "/opt/jdk") + self.assertEqual( + env["JAVA_TOOL_OPTIONS"], + self.helper["quote_java_tool_option"]( + f"-Duser.home={isolated_home.resolve()}" + ), + ) + self.assertEqual(env["NODE_ENV"], "test") + self.assertEqual(env["OPENCLAW_TESTBOX"], "1") + isolated_blacksmith = isolated_home / ".blacksmith" + self.assertEqual( + (isolated_blacksmith / "credentials").read_bytes(), + b"test-blacksmith-credentials", + ) + self.assertFalse( + (isolated_blacksmith / "unrelated-state").exists() + ) + if os.name != "nt": + self.assertEqual( + stat.S_IMODE( + (isolated_blacksmith / "credentials").stat().st_mode + ), + 0o600, + ) + self.assertNotIn("PROJECT_FEATURE_MODE", env) + self.assertEqual(env["HOME"], str(isolated_home.resolve())) + self.assertNotIn("CARGO_HOME", env) + self.assertEqual(env["RUSTUP_HOME"], str(rustup_home.resolve())) + self.assertEqual( + env["XDG_CONFIG_HOME"], + str(isolated_home.resolve() / ".config"), + ) + self.assertNotIn("GH_CONFIG_DIR", env) + self.assertNotIn("CLOUDSDK_CONFIG", env) + self.assertNotIn("GITHUB_TOKEN", env) + self.assertNotIn("AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE", env) + self.assertNotIn("AZURE_FEDERATED_TOKEN_FILE", env) + self.assertNotIn("CI_JOB_JWT", env) + self.assertNotIn("DOCKER_AUTH_CONFIG", env) + self.assertNotIn("PGPASSFILE", env) + self.assertNotIn("PGPASSWORD", env) + self.assertNotIn("REDISCLI_AUTH", env) + self.assertNotIn("BASH_FUNC_testcmd%%", env) + self.assertNotIn("SHELLOPTS", env) + self.assertNotIn("NODE_OPTIONS", env) + self.assertNotIn("SERVICE_URL", env) + self.assertNotIn("UNRELATED_VALUE", env) + + os.environ.pop("HOME") + os.environ["USERPROFILE"] = str(host_home) + windows_env = self.helper["safe_test_env"]( + repo, + root / "windows-test-home", + ) + self.assertNotIn("CARGO_HOME", windows_env) + self.assertEqual( + windows_env["RUSTUP_HOME"], + str(rustup_home.resolve()), + ) + finally: + os.environ.clear() + os.environ.update(old) + + def test_parallel_test_environment_isolates_jvm_user_home(self) -> None: + java = shutil.which("java") + if java is None: + self.skipTest("java is not installed") + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + isolated_home = root / "test home" + env = self.helper["safe_test_env"](repo, isolated_home) + + result = subprocess.run( + [java, "-XshowSettings:properties", "-version"], + text=True, + encoding="utf-8", + errors="replace", + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + env=env, + check=False, + ) + + self.assertEqual(result.returncode, 0, result.stderr) + user_home = next( + ( + line.split("=", 1)[1].strip() + for line in result.stderr.splitlines() + if line.strip().startswith("user.home =") + ), + None, + ) + self.assertEqual(user_home, str(isolated_home.resolve())) + + def test_parallel_test_stderr_relay_hides_only_our_java_banner(self) -> None: + option = self.helper["quote_java_tool_option"]( + "-Duser.home=/tmp/test home" + ) + stream = io.StringIO( + f"Picked up JAVA_TOOL_OPTIONS: {option}\n" + "ordinary stderr\n" + f"Picked up JAVA_TOOL_OPTIONS: {option} -Dextra=true\n" + ) + output = io.StringIO() + + with mock.patch("sys.stderr", output): + self.helper["relay_parallel_test_stderr"](stream, option) + + self.assertEqual( + output.getvalue(), + "ordinary stderr\n" + f"Picked up JAVA_TOOL_OPTIONS: {option} -Dextra=true\n", + ) + + def test_java_tool_option_quote_round_trips_special_paths(self) -> None: + java = shutil.which("java") + if java is None: + self.skipTest("java is not installed") + names = ["space home", "apostrophe's home"] + if os.name != "nt": + names.append('double"quote home') + for name in names: + with self.subTest(name=name), tempfile.TemporaryDirectory() as tempdir: + home = Path(tempdir) / name + home.mkdir() + env = os.environ.copy() + env["JAVA_TOOL_OPTIONS"] = self.helper["quote_java_tool_option"]( + f"-Duser.home={home}" + ) + result = subprocess.run( + [java, "-XshowSettings:properties", "-version"], + text=True, + encoding="utf-8", + errors="replace", + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + env=env, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn(f"user.home = {home}", result.stderr) + + def test_safe_proxy_url_accepts_credential_free_formats(self) -> None: + for value in ( + "http://proxy.example.invalid:8080", + "proxy.example.invalid:8080", + "socks4://proxy.example.invalid", + "socks4a://proxy.example.invalid", + ): + with self.subTest(value=value): + self.assertTrue(self.helper["safe_proxy_url"](value)) + + for value in ( + "http://review-user:review-password@proxy.example.invalid:8080", + "socks5://review-user:review-password@proxy.example.invalid:1080", + ): + with self.subTest(value=value): + self.assertFalse(self.helper["safe_proxy_url"](value)) + + def test_safe_engine_env_rejects_credentialed_proxy(self) -> None: + with tempfile.TemporaryDirectory() as tempdir, mock.patch.dict( + os.environ, + { + "HTTPS_PROXY": ( + "http://review-user:review-password@proxy.example.invalid:8080" + ) + }, + clear=False, + ): + repo = init_repo(Path(tempdir)) + with self.assertRaisesRegex(SystemExit, "credentialed or malformed proxy"): + self.helper["safe_engine_env"](repo, engine="codex") + + def test_safe_temp_root_rejects_reviewed_repo_parent(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + hostile_temp = repo / "tmp" + hostile_temp.mkdir() + + with mock.patch.object( + tempfile, + "gettempdir", + return_value=str(hostile_temp), + ), self.assertRaisesRegex( + SystemExit, + "temporary directory must be outside", + ): + self.helper["safe_temp_root"](repo) + + @unittest.skipIf(os.name == "nt", "POSIX Testbox temp-root behavior") + def test_testbox_parallel_test_temp_root_stays_within_socket_limit(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + long_temp = root / ("macos-temp-root-" + "x" * 96) + long_temp.mkdir() + + with mock.patch.object( + tempfile, + "gettempdir", + return_value=str(long_temp), + ), mock.patch.dict( + os.environ, + {"OPENCLAW_TESTBOX": "1"}, + ): + selected = self.helper["parallel_test_temp_root"](repo) + + self.assertEqual(selected, Path("/tmp").resolve()) + socket_path = ( + selected + / ("autoreview-test-home-" + "x" * 8) + / ".blacksmith" + / "c" + / "6d146d2f25180c1d.sock" + ) + self.assertLess(len(os.fsencode(socket_path)), 104) + + def test_parallel_test_temp_root_keeps_configured_root_without_testbox(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + configured_temp = root / "configured-temp" + configured_temp.mkdir() + + with mock.patch.object( + tempfile, + "gettempdir", + return_value=str(configured_temp), + ), mock.patch.dict( + os.environ, + {"OPENCLAW_TESTBOX": "0"}, + ): + selected = self.helper["parallel_test_temp_root"](repo) + + self.assertEqual(selected, configured_temp.resolve()) + + def test_claude_fable_alias_requires_fable_safe_mode_version(self) -> None: + args = argparse.Namespace( + claude_bin="claude", + fallback_model=None, + model="fable", + ) + version_result = subprocess.CompletedProcess( + ["claude", "--version"], + 0, + "2.1.169 (Claude Code)", + "", + ) + + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + with mock.patch.dict( + self.helper["ensure_claude_isolation_supported"].__globals__, + { + "resolve_command": lambda *_args: "/usr/bin/claude", + "safe_engine_env": lambda *_args, **_kwargs: {}, + "safe_temp_root": lambda _repo: Path(tempdir), + "run": lambda *_args, **_kwargs: version_result, + }, + ), self.assertRaisesRegex( + SystemExit, + "2.1.170", + ): + self.helper["ensure_claude_isolation_supported"](args, repo) + + def test_claude_runs_outside_repo_with_auto_memory_disabled(self) -> None: + args = argparse.Namespace( + claude_allowed_tools=None, + claude_bin="claude", + fallback_model=None, + model=None, + stream_engine_output=False, + thinking=None, + tools=False, + web_search=False, + ) + observed: dict[str, object] = {} + + def fake_run( + _cmd: list[str], + cwd: Path, + **kwargs: object, + ) -> subprocess.CompletedProcess[str]: + observed["cwd"] = cwd + observed["env"] = kwargs["env"] + return subprocess.CompletedProcess([], 0, "{}", "") + + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + with mock.patch.dict( + self.helper["run_claude"].__globals__, + { + "ensure_claude_isolation_supported": lambda *_args: None, + "resolve_command": lambda *_args: "/usr/bin/claude", + "run_with_heartbeat": fake_run, + "safe_engine_env": lambda *_args, **_kwargs: { + "CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1" + }, + }, + ): + self.helper["run_claude"](args, repo, "prompt") + + self.assertFalse( + self.helper["is_within"](observed["cwd"], repo.resolve()) + ) + self.assertEqual( + observed["env"]["CLAUDE_CODE_DISABLE_AUTO_MEMORY"], + "1", + ) + + def test_build_prompt_redacts_secret_like_git_metadata(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + secret = "ghp_" + "A" * 24 + git(repo, "checkout", "-q", "-b", f"feature/{secret}") + + prompt = self.helper["build_prompt"]( + repo, "local", None, "diff", "", "" + ) + self.assertIn(self.helper["REVIEW_SECURITY_REDACTION"], prompt) + self.assertNotIn(secret, prompt) + + git(repo, "checkout", "-q", "-B", "safe-branch") + prompt = self.helper["build_prompt"]( + repo, + "branch", + f"origin/{secret}", + "diff", + "", + "", + ) + self.assertIn(self.helper["REVIEW_SECURITY_REDACTION"], prompt) + self.assertNotIn(secret, prompt) + + def test_codex_env_rejects_executable_dbus_transport(self) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + try: + os.environ["DBUS_SESSION_BUS_ADDRESS"] = ( + "unixexec:path=/tmp/hostile-helper" + ) + env = self.helper["safe_engine_env"](repo, engine="codex") + self.assertNotIn("DBUS_SESSION_BUS_ADDRESS", env) + finally: + os.environ.clear() + os.environ.update(old) + + def test_multi_provider_engines_preserve_provider_auth(self) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir).resolve() + repo = init_repo(root) + try: + os.environ["DEEPSEEK_API_KEY"] = "test-token-placeholder" + os.environ["CEREBRAS_API_KEY"] = "test-token-placeholder" + os.environ["CLOUDFLARE_ACCOUNT_ID"] = "test-account" + os.environ["CLOUDFLARE_API_TOKEN"] = "test-token-placeholder" + os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = ( + str(root / "provider-credentials.json") + ) + os.environ["AWS_ROLE_ARN"] = ( + "arn:aws:iam::123456789012:role/autoreview" + ) + os.environ["AWS_CONTAINER_AUTHORIZATION_TOKEN"] = ( + "test-token-placeholder" + ) + os.environ["AWS_CONTAINER_CREDENTIALS_FULL_URI"] = ( + "http://169.254.170.2/credentials" + ) + os.environ["AWS_WEB_IDENTITY_TOKEN_FILE"] = str( + root / "web-identity", + ) + os.environ["AWS_CONFIG_FILE"] = str(root / "aws-config") + os.environ["AWS_SHARED_CREDENTIALS_FILE"] = str( + root / "aws-credentials", + ) + os.environ["NODE_EXTRA_CA_CERTS"] = str(root / "corporate-ca.pem") + os.environ["SSL_CERT_FILE"] = str(root / "tls-ca.pem") + os.environ["SSL_CERT_DIR"] = str(root / "tls-ca") + os.environ["SNOWFLAKE_ACCOUNT"] = "test-account" + os.environ["SNOWFLAKE_CORTEX_TOKEN"] = "test-token-placeholder" + os.environ["AZURE_RESOURCE_NAME"] = "test-resource" + os.environ["ANTHROPIC_OAUTH_TOKEN"] = "test-token-placeholder" + os.environ["AWS_BEDROCK_FORCE_HTTP1"] = "1" + os.environ["AWS_BEDROCK_SKIP_AUTH"] = "1" + os.environ["AZURE_CLIENT_ID"] = "test-client" + os.environ["AZURE_CLIENT_SECRET"] = "test-token-placeholder" + os.environ["AZURE_TENANT_ID"] = "test-tenant" + os.environ["GCLOUD_PROJECT"] = "test-project" + os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project" + os.environ["CODEX_API_KEY"] = "test-token-placeholder" + os.environ["CODEX_CA_CERTIFICATE"] = str(root / "codex-ca.pem") + os.environ["COPILOT_GITHUB_TOKEN"] = "test-token-placeholder" + os.environ["PI_OFFLINE"] = "1" + os.environ["PI_SKIP_VERSION_CHECK"] = "1" + os.environ["PI_TELEMETRY"] = "0" + os.environ["NPM_TOKEN"] = "test-token-placeholder" + os.environ["SENTRY_API_KEY"] = "test-token-placeholder" + os.environ["SENTRY_AUTH_TOKEN"] = "test-token-placeholder" + os.environ["DIGITALOCEAN_ACCESS_TOKEN"] = "test-token-placeholder" + os.environ["GITLAB_TOKEN"] = "test-token-placeholder" + os.environ["NODE_OPTIONS"] = "--require=/tmp/unsafe.js" + os.environ["GOOGLE_EXTERNAL_ACCOUNT_ALLOW_EXECUTABLES"] = "1" + os.environ["XDG_DATA_HOME"] = str(root / "opencode-auth") + + for engine in ("opencode", "pi"): + with self.subTest(engine=engine): + env = self.helper["safe_engine_env"](repo, engine=engine) + for key in ( + "AWS_ROLE_ARN", + "AWS_CONTAINER_AUTHORIZATION_TOKEN", + "AWS_CONTAINER_CREDENTIALS_FULL_URI", + "AWS_BEDROCK_FORCE_HTTP1", + "AWS_BEDROCK_SKIP_AUTH", + "AWS_CONFIG_FILE", + "AWS_SHARED_CREDENTIALS_FILE", + "AWS_WEB_IDENTITY_TOKEN_FILE", + "CEREBRAS_API_KEY", + "CLOUDFLARE_ACCOUNT_ID", + "CLOUDFLARE_API_TOKEN", + "COPILOT_GITHUB_TOKEN", + "DEEPSEEK_API_KEY", + "GOOGLE_APPLICATION_CREDENTIALS", + "NODE_EXTRA_CA_CERTS", + "SSL_CERT_DIR", + "SSL_CERT_FILE", + "SNOWFLAKE_ACCOUNT", + "SNOWFLAKE_CORTEX_TOKEN", + "AZURE_RESOURCE_NAME", + "ANTHROPIC_OAUTH_TOKEN", + ): + self.assertEqual(env[key], os.environ[key]) + self.assertNotIn("NODE_OPTIONS", env) + self.assertNotIn("NPM_TOKEN", env) + self.assertNotIn("SENTRY_API_KEY", env) + self.assertNotIn("SENTRY_AUTH_TOKEN", env) + self.assertNotIn( + "GOOGLE_EXTERNAL_ACCOUNT_ALLOW_EXECUTABLES", + env, + ) + if engine == "opencode": + self.assertEqual( + env["DIGITALOCEAN_ACCESS_TOKEN"], + os.environ["DIGITALOCEAN_ACCESS_TOKEN"], + ) + self.assertEqual( + env["GITLAB_TOKEN"], + os.environ["GITLAB_TOKEN"], + ) + self.assertEqual( + env["XDG_DATA_HOME"], + str(root / "opencode-auth"), + ) + else: + self.assertNotIn("DIGITALOCEAN_ACCESS_TOKEN", env) + self.assertNotIn("GITLAB_TOKEN", env) + self.assertEqual(env["PI_OFFLINE"], "1") + self.assertEqual(env["PI_SKIP_VERSION_CHECK"], "1") + self.assertEqual(env["PI_TELEMETRY"], "0") + + claude_env = self.helper["safe_engine_env"](repo, engine="claude") + for key in ( + "AZURE_CLIENT_ID", + "AZURE_CLIENT_SECRET", + "AZURE_TENANT_ID", + "GCLOUD_PROJECT", + "GOOGLE_CLOUD_PROJECT", + "AWS_ROLE_ARN", + "AWS_CONFIG_FILE", + "AWS_SHARED_CREDENTIALS_FILE", + "AWS_WEB_IDENTITY_TOKEN_FILE", + "GOOGLE_APPLICATION_CREDENTIALS", + "NODE_EXTRA_CA_CERTS", + "SSL_CERT_DIR", + "SSL_CERT_FILE", + ): + self.assertEqual(claude_env[key], os.environ[key]) + self.assertNotIn("DEEPSEEK_API_KEY", claude_env) + self.assertNotIn("NODE_OPTIONS", claude_env) + codex_env = self.helper["safe_engine_env"](repo, engine="codex") + for key in ( + "CODEX_API_KEY", + "CODEX_CA_CERTIFICATE", + "SSL_CERT_DIR", + "SSL_CERT_FILE", + ): + self.assertEqual(codex_env[key], os.environ[key]) + finally: + os.environ.clear() + os.environ.update(old) + + def test_multi_provider_custom_credentials_require_explicit_safe_names(self) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + try: + os.environ["CORP_LLM_API_KEY"] = "test-token-placeholder" + os.environ["CORP_AUTH_TOKEN"] = "test-token-placeholder" + os.environ["AUTOREVIEW_PROVIDER_ENV_ALLOW"] = ( + "CORP_LLM_API_KEY,CORP_AUTH_TOKEN" + ) + + for engine in ("opencode", "pi"): + env = self.helper["safe_engine_env"](repo, engine=engine) + self.assertEqual( + env["CORP_LLM_API_KEY"], + os.environ["CORP_LLM_API_KEY"], + ) + self.assertEqual( + env["CORP_AUTH_TOKEN"], + os.environ["CORP_AUTH_TOKEN"], + ) + self.assertNotIn("AUTOREVIEW_PROVIDER_ENV_ALLOW", env) + + os.environ["AUTOREVIEW_PROVIDER_ENV_ALLOW"] = "NODE_OPTIONS" + with self.assertRaisesRegex( + SystemExit, + "invalid AUTOREVIEW_PROVIDER_ENV_ALLOW entry", + ): + self.helper["safe_engine_env"](repo, engine="pi") + finally: + os.environ.clear() + os.environ.update(old) + + def test_provider_credential_paths_are_forwarded_as_absolute(self) -> None: + old_env = os.environ.copy() + old_cwd = Path.cwd() + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + try: + os.chdir(repo) + os.environ["AWS_CONFIG_FILE"] = "../shared/aws-config" + os.environ["SSL_CERT_DIR"] = os.pathsep.join( + ("../tls/one", "../tls/two"), + ) + + env = self.helper["safe_engine_env"](repo, engine="pi") + + self.assertEqual( + env["AWS_CONFIG_FILE"], + str((root / "shared" / "aws-config").resolve()), + ) + self.assertEqual( + env["SSL_CERT_DIR"], + os.pathsep.join( + ( + str((root / "tls" / "one").resolve()), + str((root / "tls" / "two").resolve()), + ) + ), + ) + finally: + os.chdir(old_cwd) + os.environ.clear() + os.environ.update(old_env) + + def test_opencode_rejects_repo_local_xdg_auth_store(self) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + try: + os.environ["XDG_DATA_HOME"] = str(repo / ".opencode-data") + os.environ["AWS_CONFIG_FILE"] = str(repo / ".aws-config") + os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = str( + repo / "provider-credentials.json" + ) + os.environ["NODE_EXTRA_CA_CERTS"] = str(repo / "ca.pem") + os.environ["SSL_CERT_FILE"] = str(repo / "tls-ca.pem") + os.environ["SSL_CERT_DIR"] = os.pathsep.join( + (str(repo.parent / "tls-ca"), str(repo / "tls-ca")), + ) + env = self.helper["safe_engine_env"](repo, engine="opencode") + self.assertNotIn("XDG_DATA_HOME", env) + self.assertNotIn("AWS_CONFIG_FILE", env) + self.assertNotIn("GOOGLE_APPLICATION_CREDENTIALS", env) + self.assertNotIn("NODE_EXTRA_CA_CERTS", env) + self.assertNotIn("SSL_CERT_FILE", env) + self.assertNotIn("SSL_CERT_DIR", env) + finally: + os.environ.clear() + os.environ.update(old) + + def test_engines_reject_repo_local_config_roots(self) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + try: + os.environ["CLAUDE_CONFIG_DIR"] = str(repo / ".claude") + os.environ["CODEX_HOME"] = str(repo / ".codex") + os.environ["PI_CODING_AGENT_DIR"] = str(repo / ".pi") + os.environ["CODEX_CA_CERTIFICATE"] = str(repo / "codex-ca.pem") + os.environ["SSL_CERT_FILE"] = str(repo / "tls-ca.pem") + os.environ["HOME"] = str(repo) + os.environ["USERPROFILE"] = str(repo) + claude_env = self.helper["safe_engine_env"](repo, engine="claude") + codex_env = self.helper["safe_engine_env"](repo, engine="codex") + pi_env = self.helper["safe_engine_env"](repo, engine="pi") + self.assertNotIn("CLAUDE_CONFIG_DIR", claude_env) + self.assertNotIn("CODEX_HOME", codex_env) + self.assertNotIn("CODEX_CA_CERTIFICATE", codex_env) + self.assertNotIn("SSL_CERT_FILE", codex_env) + self.assertNotIn("PI_CODING_AGENT_DIR", pi_env) + self.assertNotIn("HOME", claude_env) + self.assertNotIn("USERPROFILE", claude_env) + finally: + os.environ.clear() + os.environ.update(old) + + def test_codex_auth_config_ignores_repo_local_home(self) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + config_dir = repo / ".codex" + config_dir.mkdir() + (config_dir / "config.toml").write_text( + 'forced_login_method = "api"\n', + encoding="utf-8", + ) + try: + os.environ["CODEX_HOME"] = str(config_dir) + self.assertEqual(self.helper["codex_auth_config_flags"](repo), []) + finally: + os.environ.clear() + os.environ.update(old) + + def test_codex_runtime_home_links_only_auth_and_persists_refresh(self) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + source_home = root / "host-home" / ".codex" + runtime_home = root / "runtime" / "codex-home" + source_home.mkdir(parents=True) + source_auth = source_home / "auth.json" + source_auth.write_text( + '{"token":"test-token-placeholder"}', + encoding="utf-8", + ) + (source_home / "config.toml").write_text( + 'cli_auth_credentials_store = "file"\n', + encoding="utf-8", + ) + try: + os.environ["CODEX_HOME"] = str(source_home) + linked = self.helper["prepare_codex_runtime_auth"](repo, runtime_home) + self.assertTrue(linked) + self.assertTrue((runtime_home / "auth.json").is_file()) + self.assertTrue( + os.path.samefile(source_auth, runtime_home / "auth.json") + ) + self.assertFalse((runtime_home / "config.toml").exists()) + self.assertIn( + 'cli_auth_credentials_store="file"', + self.helper["codex_auth_config_flags"]( + repo, + force_file=True, + ), + ) + + (runtime_home / "auth.json").write_text( + '{"token":"test-auth-token"}', + encoding="utf-8", + ) + self.assertEqual( + json.loads(source_auth.read_text(encoding="utf-8"))["token"], + "test-auth-token", + ) + finally: + os.environ.clear() + os.environ.update(old) + + def test_codex_runtime_home_does_not_promote_keyring_fallback_file(self) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + source_home = root / "host-home" / ".codex" + source_home.mkdir(parents=True) + (source_home / "auth.json").write_text( + '{"token":"test-token-placeholder"}', + encoding="utf-8", + ) + (source_home / "config.toml").write_text( + 'cli_auth_credentials_store = "keyring"\n', + encoding="utf-8", + ) + try: + os.environ["CODEX_HOME"] = str(source_home) + self.assertFalse( + self.helper["prepare_codex_runtime_auth"]( + repo, + root / "runtime" / "codex-home", + ) + ) + finally: + os.environ.clear() + os.environ.update(old) + + def test_codex_runtime_home_fails_closed_when_linking_is_unavailable( + self, + ) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + source_home = root / "host-home" / ".codex" + source_home.mkdir(parents=True) + source_auth = source_home / "auth.json" + source_auth.write_text( + '{"token":"test-token-placeholder"}', + encoding="utf-8", + ) + try: + os.environ["CODEX_HOME"] = str(source_home) + with ( + mock.patch("os.link", side_effect=OSError("blocked")), + mock.patch.object( + Path, + "symlink_to", + side_effect=OSError("blocked"), + ), + self.assertRaisesRegex( + SystemExit, + "unable to isolate Codex file authentication", + ), + ): + self.helper["prepare_codex_runtime_auth"]( + repo, + root / "runtime" / "codex-home", + ) + self.assertEqual( + json.loads(source_auth.read_text(encoding="utf-8"))["token"], + "test-token-placeholder", + ) + finally: + os.environ.clear() + os.environ.update(old) + + def test_codex_runtime_home_preserves_auto_keyring_namespace(self) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + source_home = root / "host-home" / ".codex" + runtime_home = root / "runtime" / "codex-home" + source_home.mkdir(parents=True) + (source_home / "auth.json").write_text( + '{"token":"test-token-placeholder"}', + encoding="utf-8", + ) + (source_home / "config.toml").write_text( + 'cli_auth_credentials_store = "auto"\n', + encoding="utf-8", + ) + try: + os.environ["CODEX_HOME"] = str(source_home) + linked = self.helper["prepare_codex_runtime_auth"]( + repo, + runtime_home, + ) + self.assertFalse(linked) + flags = self.helper["codex_auth_config_flags"](repo) + self.assertIn('cli_auth_credentials_store="auto"', flags) + finally: + os.environ.clear() + os.environ.update(old) + + def test_empty_codex_home_uses_external_default(self) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + default_home = root / "host-home" / ".codex" + default_home.mkdir(parents=True) + try: + os.environ["CODEX_HOME"] = "" + with mock.patch.object( + Path, + "home", + return_value=default_home.parent, + ): + self.assertEqual( + self.helper["codex_source_home"](repo), + default_home.resolve(), + ) + finally: + os.environ.clear() + os.environ.update(old) + + def test_empty_codex_home_ignores_missing_default(self) -> None: + old = os.environ.copy() + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + missing_home = root / "missing-home" + try: + os.environ["CODEX_HOME"] = "" + with mock.patch.object( + Path, + "home", + return_value=missing_home, + ): + self.assertIsNone( + self.helper["codex_source_home"](repo) + ) + finally: + os.environ.clear() + os.environ.update(old) + + def test_opencode_web_search_preserves_explicit_exa_opt_in(self) -> None: + old = os.environ.copy() + try: + os.environ["OPENCODE_ENABLE_EXA"] = "1" + enabled = self.helper["opencode_review_env"](True) + disabled = self.helper["opencode_review_env"](False) + self.assertEqual(enabled["OPENCODE_ENABLE_EXA"], "1") + self.assertNotIn("OPENCODE_ENABLE_EXA", disabled) + finally: + os.environ.clear() + os.environ.update(old) + + def test_codex_isolation_restricts_tool_environment(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + runtime_root = root / "runtime" + flags = self.helper["codex_config_isolation_flags"]( + repo, + runtime_root, + ) + + for required in ( + f"sqlite_home={json.dumps(str((runtime_root / 'state').resolve()))}", + f"log_dir={json.dumps(str((runtime_root / 'log').resolve()))}", + "features.shell_snapshot=false", + "features.hooks=false", + "features.plugins=false", + "skills.include_instructions=false", + "skills.config=[]", + 'shell_environment_policy.inherit="core"', + "shell_environment_policy.ignore_default_excludes=false", + "shell_environment_policy.experimental_use_profile=false", + "allow_login_shell=false", + 'default_permissions="autoreview"', + 'permissions.autoreview.filesystem={":minimal"="read",":workspace_roots"="read"}', + ): + self.assertIn(required, flags) + set_flag = next( + flag for flag in flags if flag.startswith("shell_environment_policy.set=") + ) + for key, value in self.helper["codex_tool_git_env"]().items(): + self.assertIn(f"{key}={json.dumps(value)}", set_flag) + + def test_safe_engine_env_excludes_repo_local_path_entries(self) -> None: + old_path = os.environ.get("PATH", "") + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + os.environ["PATH"] = f"{repo}{os.pathsep}{old_path}" + try: + env = self.helper["safe_engine_env"](repo, engine="codex") + finally: + os.environ["PATH"] = old_path + + self.assertNotIn(str(repo.resolve()), env["PATH"].split(os.pathsep)) + + def test_find_command_rejects_explicit_repo_local_executables(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + (repo / "tools").mkdir() + (root / "trusted").mkdir() + repo_bin = self.helper["write_executable"]( + repo / "tools" / "codex", + "#!/bin/sh\nexit 0\n", + ) + external_bin = self.helper["write_executable"]( + root / "trusted" / "codex", + "#!/bin/sh\nexit 0\n", + ) + + self.assertIsNone( + self.helper["find_command"]("tools/codex", repo), + ) + self.assertIsNone( + self.helper["find_command"](str(repo_bin), repo), + ) + self.assertEqual( + self.helper["find_command"](str(external_bin), repo), + str(Path(os.path.abspath(external_bin))), + ) + self.assertEqual( + self.helper["find_command"]("../trusted/codex", repo), + str(Path(os.path.abspath(external_bin))), + ) + + external_link = root / "trusted" / "external-codex" + repo_link = repo / "tools" / "external-codex" + try: + external_link.symlink_to(repo_bin) + repo_link.symlink_to(external_bin) + except OSError as exc: + if os.name == "nt" and getattr(exc, "winerror", None) == 1314: + return + raise + self.assertIsNone( + self.helper["find_command"](str(external_link), repo), + ) + self.assertIsNone( + self.helper["find_command"](str(repo_link), repo), + ) + + def test_validate_report_normalizes_relative_finding_paths(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + report = { + "findings": [ + { + "title": "Finding", + "body": "Body", + "priority": "P1", + "confidence": 0.9, + "category": "bug", + "code_location": {"file_path": r".\src\index.ts", "line": 1}, + } + ], + "overall_correctness": "patch is incorrect", + "overall_explanation": "Explanation", + "overall_confidence": 0.9, + } + + self.helper["validate_report"](report, repo, {"src/index.ts"}, []) + + self.assertEqual(report["findings"][0]["code_location"]["file_path"], "src/index.ts") + + report["findings"][0]["code_location"]["file_path"] = r"src\index.ts" + self.helper["validate_report"](report, repo, {r"src\index.ts"}, []) + self.assertEqual( + report["findings"][0]["code_location"]["file_path"], + r"src\index.ts", + ) + + report["findings"][0]["code_location"]["file_path"] = " " + with self.assertRaisesRegex(SystemExit, "invalid location"): + self.helper["validate_report"](report, repo, {"src/index.ts"}, []) + + for invalid_path in (123, None, True): + with self.subTest(invalid_path=invalid_path): + report["findings"][0]["code_location"] = { + "file_path": invalid_path, + "line": 1, + } + with self.assertRaisesRegex(SystemExit, "invalid location"): + self.helper["validate_report"]( + report, + repo, + {"src/index.ts"}, + [], + ) + + report["findings"][0]["code_location"] = { + "file_path": "src/index.ts", + "line": True, + } + with self.assertRaisesRegex(SystemExit, "invalid location"): + self.helper["validate_report"](report, repo, {"src/index.ts"}, []) + + report["findings"][0]["code_location"] = { + "file_path": "src/index.ts", + "line": 1, + "extra": "ignored", + } + with self.assertRaisesRegex( + SystemExit, + "invalid code_location keys", + ): + self.helper["validate_report"](report, repo, {"src/index.ts"}, []) + + def test_print_report_escapes_terminal_controls(self) -> None: + report = { + "findings": [ + { + "title": "clear\x1b[2Jscreen", + "body": "first line\nsecond\u202eline café\udc9b", + "priority": "P1", + "confidence": 0.9, + "category": "security", + "code_location": { + "file_path": "src/\x9b2Jfile.py", + "line": 1, + }, + } + ], + "overall_correctness": "patch is incorrect", + "overall_explanation": "explanation\x07", + "overall_confidence": 0.9, + } + output = io.StringIO() + + with contextlib.redirect_stdout(output): + self.helper["print_report"](report, label="review\x00label") + + rendered = output.getvalue() + for control in ( + "\x00", + "\x07", + "\x1b", + "\x9b", + "\u202e", + "\udc9b", + ): + self.assertNotIn(control, rendered) + for escaped in ( + r"review\x00label", + r"clear\x1b[2Jscreen", + r"src/\x9b2Jfile.py", + r"second\u202eline café\udc9b", + r"explanation\x07", + ): + self.assertIn(escaped, rendered) + self.assertIn("first line\nsecond", rendered) + + def test_validate_report_escapes_controls_in_errors(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + report = { + "findings": [ + { + "title": "Finding", + "body": "Body", + "priority": "P1\x1b]52;c;VEVTVA==\x07", + "confidence": 0.9, + "category": "security", + "code_location": { + "file_path": "src/index.py", + "line": 1, + }, + } + ], + "overall_correctness": "patch is incorrect", + "overall_explanation": "Explanation", + "overall_confidence": 0.9, + } + + with self.assertRaises(SystemExit) as raised: + self.helper["validate_report"]( + report, + repo, + {"src/index.py"}, + [], + ) + + message = str(raised.exception) + self.assertNotIn("\x1b", message) + self.assertNotIn("\x07", message) + self.assertIn(r"P1\x1b]52;c;VEVTVA==\x07", message) + + def test_safe_engine_env_ignores_inaccessible_path_entries(self) -> None: + old_path = os.environ.get("PATH", "") + with tempfile.TemporaryDirectory() as tempdir: + root = Path(tempdir) + repo = init_repo(root) + blocked = root / "blocked" + os.environ["PATH"] = f"{blocked}{os.pathsep}{old_path}" + original_exists = Path.exists + + def fake_exists(path: Path) -> bool: + if str(path) == str(blocked): + raise PermissionError("access denied") + return original_exists(path) + + try: + with mock.patch.object(Path, "exists", fake_exists): + env = self.helper["safe_engine_env"](repo, engine="codex") + finally: + os.environ["PATH"] = old_path + + self.assertNotIn(str(blocked), env["PATH"].split(os.pathsep)) + + def test_run_with_heartbeat_replaces_undecodable_engine_output(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + result = self.helper["run_with_heartbeat"]( + [ + sys.executable, + "-c", + "import sys; sys.stdout.buffer.write(b'\\x90\\n')", + ], + Path(tempdir), + label="decode-test", + heartbeat_seconds=1, + ) + + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("\ufffd", result.stdout) + + def test_large_repo_relative_evidence_file_is_rejected(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + evidence = repo / "evidence.txt" + evidence.write_text("x" * 600_000, encoding="utf-8") + + with self.assertRaisesRegex(SystemExit, "file too large to scan safely"): + self.helper["validate_evidence_file"]( + repo, + "evidence.txt", + "--dataset", + ) + + def test_copilot_fails_closed_without_repo_only_read_sandbox(self) -> None: + args = argparse.Namespace( + copilot_bin="copilot", + thinking=None, + tools=True, + model=None, + web_search=False, + stream_engine_output=False, + ) + + with tempfile.TemporaryDirectory() as tempdir: + repo = init_repo(Path(tempdir)) + with self.assertRaisesRegex( + SystemExit, + r"ignored repository secrets; use codex, claude, or pi", + ) as error: + self.helper["run_copilot"]( + args, + repo, + "Repository root: .\n\nprompt", + ) + self.assertNotIn("opencode", str(error.exception)) + + def test_claude_inventory_is_bundle_and_web_only(self) -> None: + args = argparse.Namespace( + claude_allowed_tools="WebFetch(domain:docs.example.com),WebSearch", + web_search=True, + ) + + self.assertEqual( + self.helper["claude_allowed_tools"](args), + "WebFetch(domain:docs.example.com),WebSearch", + ) + self.assertEqual( + self.helper["claude_tool_inventory"](args), + "WebFetch,WebSearch", + ) + + args.web_search = False + self.assertEqual( + self.helper["claude_allowed_tools"](args), + "", + ) + + args.claude_allowed_tools = "Read" + with self.assertRaisesRegex(SystemExit, "not read-only"): + self.helper["claude_tool_inventory"](args) + + args.web_search = True + args.claude_allowed_tools = "WebFetch" + with self.assertRaisesRegex(SystemExit, "one explicit domain"): + self.helper["claude_tool_inventory"](args) + + def test_uri_reference_suppression_stays_within_credential_span( + self, + ) -> None: + for content in ( + "DATABASE_URL=https://" + "$TOKEN:@host", + "DATABASE_URL=https://" + "${TOKEN}:@host", + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + self.assertFalse( + self.helper["secret_text_risk"](content + "/path") + ) + self.assertTrue( + self.helper["secret_text_risk"]( + content + + "/pass" + + "word=real-hardcoded-" + + "secret" + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + "TO" + + "KEN=https:" + + "//$USER:@host/actual-hardcoded-" + + "secret-123456" + ) + ) + + def test_secret_detector_keeps_chained_assignment_fallbacks(self) -> None: + for content in ( + "pass" + + 'word = first, second = load_pair() or ("real-hardcoded-' + + 'secret", "x")', + "pass" + + 'word = first, second = ("ordinary-hardcoded-value-12345", "x")', + "db_pass" + + 'word = source, second = load_pair() or ("real-hardcoded-' + + 'secret", "x")', + "pass" + + 'word = first, second = load(), "ordinary-hardcoded-' + + 'value-12345"', + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_stops_at_sibling_argument_fallbacks(self) -> None: + for content in ( + "login(pass" + + 'word=getpass.getpass(), second=load_pair() or (' + + '"ordinary-default-value", "x"))', + '{"pass' + + 'word": getpass.getpass(), "second": load_pair() or (' + + '"ordinary-default-value", "x")}', + "config = {\npass" + + "word: first,\n" + + 'second: load_pair() or ("ordinary-default-value", "x")\n}', + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_secret_detector_handles_many_sibling_assignments(self) -> None: + content = ( + "pass" + + "word = source, " + + ", ".join(f"a{index}=source" for index in range(1500)) + ) + + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_secret_detector_precomputes_many_assignment_positions( + self, + ) -> None: + content = "\n".join( + "to" + "ken = process.env.TOKEN" + for _index in range(2000) + ) + scanner = mock.Mock( + wraps=self.helper["top_level_line_assignment_positions"] + ) + detector = self.helper["secret_text_risk"] + + with mock.patch.dict( + detector.__globals__, + {"top_level_line_assignment_positions": scanner}, + ): + self.assertFalse(detector(content)) + + scanner.assert_called_once() + + def test_secret_detector_bounds_separated_key_matching(self) -> None: + content = "a_" * 20_000 + 'ordinary = "value"' + started = time.monotonic() + + self.assertFalse(self.helper["secret_text_risk"](content)) + + self.assertLess(time.monotonic() - started, 5.0) + + def test_csharp_evidence_masker_is_linear_on_long_lines(self) -> None: + content = "x" * 100_000 + started = time.monotonic() + + self.assertEqual( + self.helper["mask_csharp_evidence_prefix"](content), + content, + ) + + self.assertLess(time.monotonic() - started, 5.0) + + def test_csharp_evidence_masker_bounds_quote_run_scanning(self) -> None: + content = " ".join( + '"' * width + "x" + for width in range(1_000, 500, -1) + ) + started = time.monotonic() + + self.helper["mask_csharp_evidence_prefix"](content) + + self.assertLess(time.monotonic() - started, 5.0) + + def test_csharp_context_scan_is_bounded_across_many_uris(self) -> None: + content = "\n".join( + f'void Run{index}() {{ dsn=$@"https://user:' + f'{{password}}@host/{index}"; }}' + for index in range(512) + ) + started = time.monotonic() + + self.assertFalse(self.helper["secret_text_risk"](content)) + + self.assertLess(time.monotonic() - started, 5.0) + + def test_secret_detector_allows_structured_plus_username(self) -> None: + for content in ( + "https://FirstName.LastName+123@host/repo", + "https://FirstName.LastName-123@host/repo", + "https://alice+MarketingTeam2026@example.com", + "https://user123+MarketingTeam2026@example.com", + "https://First.Name+campaign-2026@example.com", + "https://first_name+campaign.2026@example.com", + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + for content in ( + "https://AbCdEfGh.IjKlMnOp" + + "+QrStUvWxYz012345@api.example/repo", + "https://Ab3dE5f" + + "+Gh7Jk9Lm2Np4Qr6St8Uv0Wx2@host/repo", + "https://service+Abcdefghijklmnop" + + "123456@host/repo", + "https://CorrectHorse" + + "+BatteryStaple2026@host/repo", + "https://FirstnameLastname" + + "+MarketingCampaign2026@example.com", + "https://user:correcthorse" + + "+BatteryStaple2026@host/repo", + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + + def test_secret_detector_scans_many_ordinary_uris_in_linear_time( + self, + ) -> None: + uri_expression = ( + '"x:' + + '//u:%s@h" % p' + ) + content = "\n".join( + f"x{index} = {uri_expression}" + for index in range(4000) + ) + started = time.monotonic() + + self.assertFalse(self.helper["secret_text_risk"](content)) + + self.assertLess(time.monotonic() - started, 8.0) + + def test_csharp_uri_interpolation_requires_csharp_declaration( + self, + ) -> None: + for content in ( + "url=$@" + + '"https:' + + '//user:{prodPasswordSecret12345}@host"', + "url=@$" + + '"https:' + + '//user:{prodPasswordSecret12345}@host"', + "endpoint=$@" + + '"https:' + + '//user:{hunter2secret}@host";', + "dsn=$@" + + '"postgres:' + + '//svc:{password}@db.example/app";', + "url=$@" + + '"https:' + + '//user:{prodPasswordSecret12345}@example.com";', + "(echo $@" + + '"https:' + + '//user:{prodPasswordSecret12345}@host")', + "if $@" + + '"https:' + + '//user:{prodPasswordSecret12345}@host"; then :; fi', + "test value == $@" + + '"https:' + + '//user:{prodPasswordSecret12345}@host";', + "echo using $@" + + '"https:' + + '//user:{prodPasswordSecret12345}@host";', + "export url=$@" + + '"https:' + + '//user:{prodPasswordSecret12345}@host";', + "// namespace N { class C { void M() {\n" + + 'connectionString=$@"https:' + + '//user:{prodPasswordSecret12345}@host";', + "/* namespace N { class C { void M() { */\n" + + 'connectionString=$@"https:' + + '//user:{prodPasswordSecret12345}@host";', + 'function Run() { dsn=$@"https:' + + '//user:{prodPasswordSecret12345}@host"; }', + "cat <<'EOF'\n; class C {\nEOF\n" + + 'url=$@"https:' + + '//user:{prodPasswordSecret12345}@host";', + "cat < $@"postgres:' + + '//svc:{password}@db.example/app";', + 'var dsn = enabled ? $@"postgres:' + + '//svc:{password}@db.example/app" : fallback;', + 'var dsn = prefix + $@"postgres:' + + '//svc:{password}@db.example/app";', + 'var values = new[] { enabled ? $@"postgres:' + + '//svc:{password}@db.example/app" : fallback };', + 'var values = new[] { enabled ? fallback : $@"postgres:' + + '//svc:{password}@db.example/app" };', + 'var values = new[] { value ?? $@"postgres:' + + '//svc:{password}@db.example/app" };', + 'var values = new[] { prefix + $@"postgres:' + + '//svc:{password}@db.example/app" + suffix };', + 'var values = new[] { $@"postgres:' + + '//svc:{password}@db.example/app" };', + 'var values = new[] { $@"postgres:' + + '//svc:{password}@db.example/app"[0] };', + 'var values = new[] { $@"postgres:' + + '//svc:{password}@db.example/app".ToString() };', + 'var values = [$@"postgres:' + + '//svc:{password}@db.example/app"];', + 'var text = $@"postgres:' + + '//svc:{password}@db.example/app".ToString();', + 'var first = $@"postgres:' + + '//svc:{password}@db.example/app"[0];', + 'var required = $@"postgres:' + + '//svc:{password}@db.example/app"!;', + 'using System; if ($@"postgres:' + + '//svc:{password}@db.example/app" == expected) {}', + 'using System; if (dsn == $@"postgres:' + + '//svc:{password}@db.example/app") {}', + 'Log(); dsn = $@"postgres:' + + '//svc:{password}@db.example/app";', + 'Log(); dsn += $@"postgres:' + + '//svc:{password}@db.example/app";', + 'Log(); connect($@"postgres:' + + '//svc:{password}@db.example/app");', + 'int retries = 3; dsn = $@"postgres:' + + '//svc:{password}@db.example/app";', + 'void Run() { dsn = $@"https:' + + '//user:{prodPasswordSecret12345}@host"; }', + 'var ready = true; void Run() { dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'class C { void Run() { dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; } }', + 'Task LoadAsync() { dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'Task<(string User, string Password)> Load() { dsn=$@"postgres:' + + '//svc:{dbPassword}@db.example/app"; }', + 'global::System.String Load() { dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'void Run() { if (ready) { Init(); } dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'string? Load() { dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'byte[] Read() { dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'customtype Load() { dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; }', + '(int Code, string Message) Load() { dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; }', + '(int Code, string Message)? Load() { dsn=$@"postgres:' + + '//svc:{dbPassword}@db.example/app"; }', + 'unsafe byte* Load() { dsn=$@"postgres:' + + '//svc:{dbPassword}@db.example/app"; }', + 'ref string Load() { dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'T Load() { dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; }', + '[Conditional("DEBUG")] void Run() { dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'void Run() { dsn=$@"label ""prod"" https:' + + '//svc:{password}@db.example/app"; }', + 'void Run() { dsn=$@"{Get("x")}https:' + + '//svc:{prodPasswordSecret12345}@db.example/app"; }', + 'class C { void Run() { /*' + + "x" * 9_000 + + '*/ dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; } }', + 'var banner = @"""";\n' + + 'void Run() { dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'var banner = @$"""";\n' + + 'void Run() { dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'var banner = """alpha " beta""";\n' + + 'void Run() { dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'var banner = """text"""";\n' + + 'void Run() { dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'var example = "cat <<\'EOF\'";\n' + + 'void Run() { dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; }', + "// example: cat <<'EOF'\n" + + 'void Run() { dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'var banner = """"alpha """ beta"""";\n' + + 'void Run() { dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'if (enabled) { dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'record Worker { void Run() { dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; } }', + 'sealed class Worker { Worker() { dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; } }', + 'abstract class Worker { Worker() { dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; } }', + '[Serializable] public sealed class Worker { ' + + 'Worker() { dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; } }', + 'record class Worker { Worker() { dsn=$@"postgres:' + + '//svc:{password}@db.example/app"; } }', + 'struct Worker { void Run() { dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; } }', + 'interface Worker { void Run() { dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; } }', + 'class C { public string Dsn { get; set; } = $@"postgres:' + + '//svc:{password}@db.example/app"; }', + 'class C { void Run() { if (ready) { Log(); } dsn = $@"postgres:' + + '//svc:{password}@db.example/app"; } }', + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_csharp_spaced_assignment_requires_plain_reference(self) -> None: + secret_shaped_reference = "".join( + ("prodPassword", "Secret", "12345") + ) + formatted_reference = "".join(("ActualToken", "1234567890")) + self.assertFalse( + self.helper["secret_text_risk"]( + 'url = $@"https:' + + '//user:{password}@example.com";' + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + f'url = $@"https://user:' + f'{{{secret_shaped_reference}}}@example.com";' + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + f'url = $@"https:' + f'//user:{{{formatted_reference}:N}}@host/{{password}}";' + ) + ) + + def test_secret_detector_handles_additional_credential_keys(self) -> None: + for content in ( + "cred" + "ential = real-hardcoded-" + "secret", + "cred" + "entials = real-hardcoded-" + "secret", + "private_" + "key = real-hardcoded-" + "secret", + "github_to" + "ken = ordinary-hardcoded-value-12345", + "db_pass" + "word = ordinary-hardcoded-value-12345", + "stripe_api_" + "key = ordinary-hardcoded-value-12345", + "githubTo" + "ken = ordinary-hardcoded-value-12345", + "dbPass" + "word = ordinary-hardcoded-value-12345", + "awsCred" + "entials = ordinary-hardcoded-value-12345", + "githubAPI" + "Key = ordinary-hardcoded-value-12345", + "myAWSSecretAccess" + + "Key = ordinary-hardcoded-value-12345", + "userIDTo" + "ken = ordinary-hardcoded-value-12345", + "GITHUBTO" + "KEN = ordinary-hardcoded-value-12345", + "DBPASS" + "WORD = ordinary-hardcoded-value-12345", + "githubto" + "ken = ordinary-hardcoded-value-12345", + "dbpass" + 'word = "Summer2026!"', + "stripeapi" + "key = ordinary-hardcoded-value-12345", + "x" * 65 + + "_pass" + + "word = ordinary-hardcoded-value-12345", + "pass" + "word: CorrectHorseBatteryStaple", + "PASS" + "WORD=CorrectHorseBatteryConfig", + "pass" + "word: CorrectHorseBatteryOptions", + "cred" + "entials: CorrectHorseBatteryCredentials", + "# class Fake {\ncred" + + "entials: CorrectHorseBatteryCredentials", + "# class Fake {\npass" + + "word: CorrectHorseBatteryCredentials", + "# const opts = { pass" + + "word: actualToken1234567890", + "echo ok # const opts = { pass" + + "word: actualToken1234567890", + "const opts = { cred" + + "entials: CorrectHorseBatteryStaple };", + ): + with self.subTest(content=content): + self.assertTrue(self.helper["secret_text_risk"](content)) + for content in ( + "cred" + "ential = process.env.CREDENTIAL", + "cred" + "entials = config.credentials", + "safe_" + "credentials = config.credentials", + "safeCred" + "entials = config.credentials", + "credentializer = ordinary-hardcoded-value-12345", + "private_" + 'key = os.environ["PRIVATE_KEY"]', + "type AuthOptions = { cred" + + "entials: RequestCredentials };", + 'const banner = "' + + "x" * 3_000 + + '"; type AuthOptions = { cred' + + "entials: RequestCredentials };", + "const cred" + "entials = options.credentials", + "const opts = { cred" + + "entials: requestCredentials };", + "const quote = /'/;\nconst opts = { cred" + + "entials: requestCredentials };", + "const quote = /'/; const opts = { cred" + + "entials: requestCredentials };", + "const quote = `it's`; const opts = { cred" + + "entials: requestCredentials };", + "const quote = `${`it's`}`; const opts = { cred" + + "entials: requestCredentials };", + 'const note = "unmatched `";\nconst opts = { cred' + + "entials: requestCredentials };", + "// unmatched `\nconst opts = { cred" + + "entials: requestCredentials };", + "/* unmatched ` */ const opts = { cred" + + "entials: requestCredentials };", + "safe_uri_cred" + + "entials = interpolated_empty_password_uri_ranges(\n" + + " text,\n" + + " uri_authorities,\n" + + ")", + ): + with self.subTest(content=content): + self.assertFalse(self.helper["secret_text_risk"](content)) + + def test_secret_detector_allows_fetch_credential_modes(self) -> None: + for mode in ("include", "omit", "same-origin"): + with self.subTest(mode=mode): + self.assertFalse( + self.helper["secret_text_risk"]( + "fetch(url, { cred" + + f'entials: "{mode}" }})' + ) + ) + + def test_secret_detector_allows_punctuationless_password_prompt( + self, + ) -> None: + for prompt in ( + "Enter password", + "Enter the password for the database: ", + "Enter password for GitHub: ", + "Enter password for AWS2024", + "Enter password for MicrosoftDynamics365", + "Enter password for MicrosoftDynamics2024", + "Enter password for Oracle2024", + "Enter password for PostgreSQL: ", + "Enter password for SpringBoot2024", + "Enter password for Windows2024", + "Enter your password:", + "Password:", + ): + with self.subTest(prompt=prompt): + self.assertFalse( + self.helper["secret_text_risk"]( + "pass" + + f'word = getpass.getpass("{prompt}")' + ) + ) + self.assertFalse( + self.helper["secret_text_risk"]( + 'banner = """"quoted"""\n' + + 'password = getpass.getpass("Enter password")' + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + "pass" + + 'word = getpass.getpass("Enter password for ghp_' + + 'ActualToken1234567890")' + ) + ) + for prompt in ( + "Enter password for SummerVacation2026", + "Password for Abcdefghijklmno12345", + ): + with self.subTest(prompt=prompt): + self.assertTrue( + self.helper["secret_text_risk"]( + "pass" + + f'word = getpass.getpass("{prompt}")' + ) + ) + + def test_secret_detector_allows_chained_lookup_references(self) -> None: + lookup = ( + "to" + + 'ken = response.json().get("access_' + + 'token")' + ) + + self.assertFalse(self.helper["secret_text_risk"](lookup)) + self.assertFalse( + self.helper["secret_text_risk"]( + "to" + + 'ken = client().headers.get("Authorization")' + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + lookup + ' or "ordinary-hardcoded-value-12345"' + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + "to" + + 'ken = client.auth().get("ghp_' + + 'ActualToken1234567890")' + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + "to" + + 'ken = response.get("ghp_' + + 'ActualToken1234567890")' + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + "pass" + + 'word = response.get("CorrectHorse' + + 'BatteryStaple")' + ) + ) + self.assertTrue( + self.helper["secret_text_risk"]( + "pass" + + 'word = response.get("CORRECTHORSE' + + 'BATTERYSTAPLE")' + ) + ) + + def test_secret_detector_bounds_chained_receiver_tracking(self) -> None: + content = "to" + "ken = f()" + ".x()" * 20_000 + started = time.monotonic() + + self.assertFalse(self.helper["secret_text_risk"](content)) + + self.assertLess(time.monotonic() - started, 5.0) + + def test_review_patch_allows_safe_multiline_call_hunks(self) -> None: + patch = ( + "diff --git a/safe.py b/safe.py\n" + "--- a/safe.py\n" + "+++ b/safe.py\n" + "@@ -0,0 +1,3 @@\n" + "+" + + "pass" + + "word = getpass.getpass(\n" + '+ "Password: ",\n' + "+)\n" + ) + + self.assertEqual( + self.helper["validate_review_patch"]( + "local unstaged diff", + ["safe.py"], + patch, + ), + patch, + ) + + def test_review_patch_rejects_size_before_secret_scanning(self) -> None: + scanner = mock.Mock() + validator = self.helper["validate_review_patch"] + with mock.patch.dict( + validator.__globals__, + {"require_no_secret_values": scanner}, + ): + with self.assertRaisesRegex(SystemExit, r"20 bytes; limit 10"): + validator( + "local unstaged diff", + ["safe.txt"], + "x\n" * 10, + 10, + ) + + scanner.assert_not_called() + + def test_stream_displays_escape_terminal_controls(self) -> None: + control = chr(27) + "]52;c;VEVTVA==" + chr(7) + codex = self.helper["CodexStreamDisplay"]() + claude = self.helper["ClaudeStreamDisplay"]() + codex_message = json.dumps( + { + "type": "item.completed", + "item": { + "type": "agent_message", + "text": control, + }, + } + ) + + for displayed in ( + codex("stdout", codex_message + "\n"), + codex("stderr", control + "\n"), + claude("stderr", control + "\n"), + ): + self.assertIsNotNone(displayed) + assert displayed is not None + self.assertNotIn(chr(27), displayed) + self.assertNotIn(chr(7), displayed) + self.assertIn(r"\x1b", displayed) + self.assertIn(r"\x07", displayed) + self.assertTrue(displayed.endswith("\n")) + + def test_run_with_stream_escapes_terminal_output_only(self) -> None: + control = chr(27) + "]52;c;VEVTVA==" + chr(7) + script = ( + "import sys;" + "value=chr(27)+']52;c;VEVTVA=='+chr(7);" + "sys.stdout.write(value+'\\n');" + "sys.stderr.write(value+'\\n')" + ) + stdout = io.StringIO() + stderr = io.StringIO() + + with ( + contextlib.redirect_stdout(stdout), + contextlib.redirect_stderr(stderr), + ): + result = self.helper["run_with_stream"]( + [sys.executable, "-c", script], + Path.cwd(), + input_text=None, + label="stream-test", + heartbeat_seconds=60, + stream_display=None, + resolve_root=Path.cwd(), + ) + + self.assertIn(control, result.stdout) + self.assertIn(control, result.stderr) + for displayed in (stdout.getvalue(), stderr.getvalue()): + self.assertNotIn(chr(27), displayed) + self.assertNotIn(chr(7), displayed) + self.assertIn(r"\x1b", displayed) + self.assertIn(r"\x07", displayed) + self.assertTrue(displayed.endswith("\n")) + + def test_self_test_shortcut_runs_deterministic_checks(self) -> None: + command = [str(SCRIPT), "--self-test"] + if os.name == "nt": + command = [sys.executable, str(SCRIPT), "--self-test"] + result = subprocess.run( + command, + check=False, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("autoreview engine isolation self-test: ok", result.stdout) + + +if __name__ == "__main__": + unittest.main()