Skip to content

Validate retained coverage artifacts and fix platform-aware tests - #48

Open
ZhiHe-ma wants to merge 2 commits into
cloudflare:mainfrom
ZhiHe-ma:fix/retained-artifact-validation
Open

ZhiHe-ma wants to merge 2 commits into
cloudflare:mainfrom
ZhiHe-ma:fix/retained-artifact-validation

Conversation

@ZhiHe-ma

@ZhiHe-ma ZhiHe-ma commented Sep 20, 2026

Copy link
Copy Markdown

A coverage unit could pass validation even when its local-check artifact did not exist. The coverage CLI now verifies retained files for both live checks and archived attempts, so missing evidence cannot support a successful coverage validation.

Fixes #21.

Changes

  • Resolve artifact paths relative to the ledger directory, with --output-dir for exported ledgers.
  • Require existing, single-link regular files under each check owner's artifact tree; reject symlinks/junctions, non-directory parents, and special files without reading artifact contents.
  • Preserve the structural-only library API; validateDocument(ledger, { outputDir }) enables filesystem checks.
  • Correct seven findings CLI tests that assumed OS support for protected file opening, and explicitly test fail-closed behavior on unsupported platforms.
  • Add a Node.js 22/24 CI matrix across Linux, macOS, and Windows, plus workflow documentation and an implementation/validation record.

Validation

node --test skills/security-audit/validate-findings.test.cjs skills/security-audit/validate-coverage-ledger.test.cjs

  • Windows / Node.js 24.19.0: 59 passed, 23 platform-dependent skips, 0 failures.
  • Linux / official Node.js 24 Alpine container: 80 passed, 2 unsupported-platform-only skips, 0 failures. The container had networking disabled, a read-only source mount, temporary writable storage, and resource limits.
  • All 82 tests passed in at least one applicable environment across these runs.
  • Skill structure validation, workflow YAML checks, JavaScript syntax checks, and git diff --check passed.
  • macOS and Node.js 22 are configured in CI but were not run locally.

AI Assistance Disclosure

AI tools were used to assist with implementation review, test development, documentation, and reasoning about edge cases. All submitted changes, security assumptions, test results, and validation claims were reviewed and verified by the contributor. AI-generated output was not treated as independent evidence of correctness or as a substitute for the validation described above.

The artifact check assumes a trusted, stable, parent-owned output tree. It verifies file presence/type, not evidence contents, and does not replace race-safe artifact promotion or independent finding verification. Native Windows CLI file opening continues to fail closed when the required OS protections are unavailable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coverage validator accepts nonexistent local-check artifacts

1 participant