Skip to content

feat(completeness): add --incomplete-policy to control exit behavior on incomplete scan data - #1019

Merged
sonukapoor merged 4 commits into
OWASP:mainfrom
luojiyin1987:feat/incomplete-scan-policy
Aug 29, 2026
Merged

feat(completeness): add --incomplete-policy to control exit behavior on incomplete scan data#1019
sonukapoor merged 4 commits into
OWASP:mainfrom
luojiyin1987:feat/incomplete-scan-policy

Conversation

@luojiyin1987

@luojiyin1987 luojiyin1987 commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add --incomplete-policy warn|error (default: warn) to control exit code when scan detection data is incomplete.

Behavior

Scenario warn error
Scan complete normal normal
Remediation gap warning, normal exit warning, normal exit
Detection gap warning, normal --fail-on EXIT_ERROR (3)

error only targets detection-impact incomplete data. Remediation gaps are never affected by this flag.

Changes

  • src/types.tsIncompletePolicy type, field on ParsedOptions
  • src/cli/args.ts — default warn, parse --incomplete-policy warn|error
  • src/cli/help.ts — help text + example
  • src/scan/completeness.tsshouldFailForIncompleteScan() helper
  • src/index.ts — single-folder exit code
  • src/scan/multi-folder-scan.ts — multi-folder exit code, shared aggregated completeness
  • action.ymlincomplete-policy input (empty default for old CLI compat)
  • tests/helpers.test.ts — updated expectations

Design decisions

  • Ratchet unaffected. --ratchet detection gap check runs independently. incompletePolicy does not override it.
  • EXIT_ERROR (3) over EXIT_FINDINGS (1). Incomplete scan is more severe than findings.
  • Action backward compat. Empty default means old CLI versions don't see unknown args.

Closes #898

…on incomplete scan data

Add `--incomplete-policy warn|error` (default: warn). When set to error,
EXIT_ERROR (3) is returned on detection-impact incomplete scan data.
Remediation gaps and ratchet behavior are unaffected.

Closes OWASP#1018
- Arg parsing: default warn, parse warn/error, =form, invalid throws
- Single-folder: detection+warn, detection+error, remediation+error, ratchet+warn
- Multi-folder: detection+warn, detection+error, remediation+error, multi-folder detection, ratchet+warn
- Fix action.yml: pass incomplete-policy through Apply security fixes step
- Docs: add --incomplete-policy to CLI reference and workflow integration
@luojiyin1987

Copy link
Copy Markdown
Collaborator Author

Addressed all three review items:

P1 — Behavioral tests (+16 tests, 1446 → 1462):

  • tests/helpers.test.ts: 7 tests — default warn, parse warn/error, = form, invalid values throw
  • tests/cli-integration.test.ts: 4 tests — detection+warn → exit 0, detection+error → EXIT_ERROR, remediation+error → exit 0, ratchet+detection+warn → still EXIT_ERROR
  • tests/multi-folder-scan.test.ts: 5 tests — detection+warn → exit 0, detection+error → EXIT_ERROR, remediation+error → exit 0, multi-folder detection → EXIT_ERROR, ratchet+detection+warn → still EXIT_ERROR

P2 — Action fix step: Added INPUT_INCOMPLETE_POLICY env + conditional arg passing in the Apply security fixes step, so incomplete-policy: error applies to both the scan and the fix rescan.

P3 — Docs:

  • website/docs/cli-reference.md: Added --incomplete-policy row in CI / Automation options table
  • website/docs/workflow-integration.md: Added Fail closed on incomplete detection data subsection with CLI and GitHub Actions examples

@sonukapoor

Copy link
Copy Markdown
Collaborator

Hey @luojiyin1987, thanks for this - the completeness work you've done has been great. There's an existing PR (#941) for the same feature that has priority since it came first. I've asked that contributor to address the outstanding changes within the next 48 hours. If they don't respond we'll close #941 and come back to yours. Will keep you posted!

@luojiyin1987

Copy link
Copy Markdown
Collaborator Author

No worries at all — I'm not in a rush. Happy to wait and see how #941 goes. Thanks for keeping me posted! @sonukapoor

@sonukapoor sonukapoor left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work on this one. I went through all the changed files and the logic is correct end-to-end.

A few things I verified explicitly: shouldFailForIncompleteScan() returns false immediately for anything that isn't "error", and it correctly delegates to getCompletenessImpact().hasDetectionGap - so remediation gaps are genuinely unaffected regardless of the policy setting. The ratchet path in index.ts exits well before the shouldFailForIncompleteScan check, so those two code paths are truly independent. Multi-folder aggregation via aggregateMultiFolderCompleteness() correctly sums detection diagnostics across all folders before the policy check fires, so a detection gap in any one subfolder propagates to the exit code.

Test coverage is solid - you hit all the meaningful combinations: detection gap + warn (exit 0), detection gap + error (exit 3), remediation gap + error (still exit 0), multi-folder detection gap + error (exit 3), and ratchet + detection gap + warn (exit 3 regardless of policy). Left a couple of small notes inline - neither is a blocker, just things worth a quick look.

Comment thread src/index.ts
Comment thread action.yml
@sonukapoor
sonukapoor merged commit a6a79f2 into OWASP:main Aug 29, 2026
9 checks passed
@sonukapoor

Copy link
Copy Markdown
Collaborator

Merged - thank you @luojiyin1987! The --incomplete-policy work has been great throughout the whole completeness series.

@luojiyin1987
luojiyin1987 deleted the feat/incomplete-scan-policy branch August 29, 2026 15:21
@sonukapoor sonukapoor mentioned this pull request Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] add a configurable failure policy for incomplete scans

2 participants