Skip to content

fix: require local-check artifacts to exist on disk - #34

Open
Frank-zhu0404 wants to merge 1 commit into
cloudflare:mainfrom
Frank-zhu0404:fix/issue-21-coverage-artifact-exists
Open

Frank-zhu0404 wants to merge 1 commit into
cloudflare:mainfrom
Frank-zhu0404:fix/issue-21-coverage-artifact-exists

Conversation

@Frank-zhu0404

Copy link
Copy Markdown

Summary

Closes #21.

validate-coverage-ledger.cjs previously only checked that local_checks[].artifact paths were lexically owned (agents/<agent_id>/artifacts/...). A covered / blocked / candidate unit could therefore pass while pointing at evidence that was never retained.

This change makes the CLI resolve each local-check artifact against the ledger directory (<output-dir>/coverage-ledger.json) and require that it:

  • exists on disk
  • is a regular file (lstat, not a directory / FIFO / etc.)
  • is not a symlink
  • stays inside the owning agent's real artifacts directory after realpath

The same checks apply to archived attempts[].local_checks[]. Programmatic validateDocument(ledger) without a base directory stays schema-only so existing in-memory unit tests keep working; pass baseDir (as the CLI does) to enable on-disk checks.

Test plan

  • node --test skills/security-audit/validate-coverage-ledger.test.cjs (37 pass)
  • node --test skills/security-audit/validate-findings.test.cjs (34 pass)
  • Manual repro from Coverage validator accepts nonexistent local-check artifacts #21 now exits nonzero with:
    $[0].local_checks[0].artifact: local check artifact does not exist
  • New coverage for missing artifact, directory artifact, symlink artifact, archived missing artifact, retained regular file, and schema-only vs baseDir behavior

Validate that local_checks[].artifact paths resolve to retained regular
files under the owning agent's artifacts directory when the CLI runs
against a coverage ledger. Closes cloudflare#21.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coverage validator accepts nonexistent local-check artifacts

1 participant