Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions .github/workflows/review-dispatch.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
name: Trusted review dispatch
on:
workflow_dispatch:
inputs:
pr:
description: GitHub PR URL to inspect as data
required: true
type: string
level:
description: Source review only in this hosted-runner profile
required: true
default: L1
type: choice
options: [L1, L2]
model:
description: Explicit Codex model for L2; no automatic upgrade
required: true
default: gpt-6.1-sol
type: string
permissions:
contents: read
jobs:
review:
runs-on: ubuntu-latest
# The workflow must be dispatched from trusted main, never a candidate ref.
if: github.ref == 'refs/heads/main'
timeout-minutes: 15
environment: code-qa-review
steps:
- uses: actions/checkout@v4
with:
ref: main
persist-credentials: false
- uses: actions/setup-node@v4
with:
node-version: 22
- run: npm test && npm run lint
- name: Install trusted reviewer CLI
if: inputs.level == 'L2'
run: npm install --global @openai/codex@0.165.0
- name: Review pinned source without checking out or executing PR code
env:
GH_TOKEN: ${{ github.token }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
REVIEW_PR: ${{ inputs.pr }}
REVIEW_LEVEL: ${{ inputs.level }}
REVIEW_MODEL: ${{ inputs.model }}
shell: bash
run: |
set -euo pipefail
if [[ "$REVIEW_LEVEL" == L2 ]]; then
test -n "$OPENAI_API_KEY"
printf '%s' "$OPENAI_API_KEY" | codex login --with-api-key
node review/cli.mjs review --pr "$REVIEW_PR" --level L2 --model "$REVIEW_MODEL" --reasoning-effort low --review-seconds 120 --config examples/ci/source-review.json --out "$RUNNER_TEMP/code-qa-runs"
else
node review/cli.mjs review --pr "$REVIEW_PR" --level L1 --config examples/ci/source-review.json --out "$RUNNER_TEMP/code-qa-runs"
fi
- name: Retain reports (publication remains a separate action)
if: always()
uses: actions/upload-artifact@v4
with:
name: code-qa-review-${{ github.run_id }}
path: |
${{ runner.temp }}/code-qa-runs/*/result.json
${{ runner.temp }}/code-qa-runs/*/report.md
${{ runner.temp }}/code-qa-runs/*/history.json
if-no-files-found: warn
15 changes: 15 additions & 0 deletions docs/ogc-1560-review-harness/automation-boundary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Automation boundary

The trusted coordinator collects pinned source, runs declared checks in disposable workers and validates reports. The reviewer receives only the source/evidence broker; it has no repair, publication, merge, shell or network tool. Repository instructions are data. Git reads use a fresh bare repository and a minimal environment. Runtime/test workers receive no model or publication credentials. The provider CLI is trusted software and retains its own authentication; the broker boundary is not a claim that arbitrary provider binaries are OS-confined.

Codex 0.165.0 and the installed Claude CLI were exercised with the same common report protocol. The tests verify forbidden path/command/source requests and changed evidence. Provider/tool availability is checked during execution; failure remains partial or error. Claude rendered-image support has not passed conformance, so that adapter cannot claim L5.

Managed results retain their original provider, model, revision, capabilities and omissions. Imported test claims cannot become local execution evidence. Publication needs explicit operator-selected and verified finding IDs, current PR base/head and source evidence; duplicate markers apply only to the same commit. Publication is a COMMENT review, never approval or merge. Public-fork event automation is not enabled. The manual dispatch requires trusted main and operator-configured environment protection/secrets.

References validated against current official documentation on 30 October 2026:

- [Codex MCP configuration](https://learn.chatgpt.com/docs/extend/mcp): stdio servers, required startup and read-only approval policy.
- [Claude CLI](https://code.claude.com/docs/en/cli-reference): restricted tools, explicit MCP configuration, structured output and noninteractive permission policy.
- [GitHub workflow security](https://docs.github.com/en/actions/reference/security/secure-use): keep untrusted candidate execution away from privileged workflow credentials.

Remaining acceptance is reported in the roadmap: reviewed configuration for each deployment, approved product-design demonstration, independent quality labels/adjudication and broader adversarial evaluation. These are not silently counted as passed by unit tests.
14 changes: 14 additions & 0 deletions docs/ogc-1560-review-harness/execution-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,17 @@ L4 uses `--scenarios` (behavior, category, applicable, reason, checkId) plus loa
L5 additionally uses `--workflows` (id, requirement, checkId, designId or visualInapplicableReason) and `--designs` (id, file, sha256, source, revision, authority, approvalStatus). Designs may be draft for comparison, but only approved references support acceptance. A scenario declares artifacts by relative path and kind: http, readback, trace, screenshot, video, measurement, runtime or text. Fresh artifacts are copied out, bound to the candidate and checked for transport integrity. Screenshots and design images are attached to the reviewer. Missing media/readback, wrong loaded artifacts or visible deviations remain explicit in the report.

The OpenELIS example `examples/openelis/label-preset-workflow.mjs` uses synthetic state in the disposable worker. Run the browser container as the worker UID/GID so subsequent checks can remove old evidence. The example records browser errors, screenshots, persisted readback and a trace, and restores the original preset. Setup exceptions are JUnit errors, not successful mutation detections. See `validation-v06.json` for the actual run and its limits. Artifacts are limited to 20 MiB each and 40 MiB per check; oversized evidence is reported as missing, never silently accepted.

## Review isolation and providers

Codex and Claude use the same source/evidence broker. It exposes only pinned Git diff, show, tree, literal grep and history operations plus supplied artifact reads. Paths and revisions are validated; bare Git symlink objects are data, not followed links. Pagination permits further inspection without a total source quota. Built-in shell/file/network/write tools, inherited plugins and other MCP servers are disabled. Trusted CLI authentication stays in the provider process; the model has no credential-reading tool. The broker invokes Git with a minimal environment and no system/global configuration. This is a capability boundary, not a claim that the provider executable itself is untrusted or OS-isolated. Dedicated disposable test workers remain separate.

Use `--adapter claude --model haiku --reasoning-effort low` for an inexpensive Claude source review, or choose another available model explicitly. Codex defaults remain a development-cost profile. Claude image conformance is not yet claimed: L5 fails capability validation for that adapter before starting work. No silent substitution or provider fallback occurs. Provider cancellation and timeout discard the assessment; unavailable inspection stays partial.

## External results and publication

`code-qa import --plan plan.json --external external.json --out imported.json` accepts a version-1 envelope with origin `{provider,model,runId}`, capabilities, omissions and the common reviewer report. It verifies the exact head and source quotations. External test assertions are not local execution evidence; provenance and omissions survive import.

`code-qa publish --result result.json --finding ID --verified ID` previews selected actionable findings. `--verified` is the operator's attestation, not a model-generated field. Add `--authorize-publication` only after reviewing that concrete preview and obtaining publication authorization. The publisher checks the live base/head, rechecks cited evidence, suppresses current-head duplicates, and posts a COMMENT review tied to the reviewed commit. It never approves or merges. GitHub can change immediately after the final check; the explicit commit ID preserves historical identity if that race occurs. Re-review dispositions remain private; this command does not resolve other reviewers' threads.

The manual `Trusted review dispatch` workflow is restricted to trusted main, a protected `code-qa-review` environment, read-only GitHub permissions and source-only L1/L2. Configure that environment's reviewers and API secret before use. It never checks out candidate source or runs its code with secrets. L3–L5 use the local CLI with isolated execution; this hosted profile does not pretend to provision Lima. Public-fork event triggers are deliberately absent. The dispatcher is not live until this stack is reviewed and merged by its owner.
1 change: 1 addition & 0 deletions examples/ci/source-review.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"environment":"none","checks":[],"probes":[],"experiments":[],"sources":[],"scenarios":[],"workflows":[],"designs":[],"maxRounds":0}
18 changes: 15 additions & 3 deletions review/cli.mjs
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
#!/usr/bin/env node
import {importReport,publication} from './exchange.mjs';
import { parseArgs } from 'node:util';
import { readFile, mkdir, writeFile } from 'node:fs/promises';
import { resolve, dirname, join } from 'node:path';
Expand All @@ -15,6 +16,8 @@ import { capabilityGaps } from './reviewers/index.mjs';

const help = `code-qa v0.4
levels
import --plan FILE --external FILE --out FILE
publish --result FILE --finding ID --verified ID [repeat IDs] [--authorize-publication]
plan --pr https://github.com/OWNER/REPO/pull/NUMBER --level L1..L5 [options] --out plan.json
review --pr URL --level L1..L5 [options] [--out /durable/runs]
plan/review --repo PATH --base REF --head REF [options]
Expand All @@ -26,7 +29,7 @@ Options:
--path PATH (repeatable) --context FILE (repeatable)
--requirements FILE --config FILE --environment none|openelis|container --image IMAGE
--decisions FILE --history FILE --scenarios FILE --workflows FILE --designs FILE --runtime-check ID --sources FILE --history-depth N --probes FILE --experiments FILE --max-rounds 0..2
--profile development --adapter codex --model MODEL_ID --billing-mode subscription|api|unknown
--profile development --adapter codex|claude --model MODEL_ID --billing-mode subscription|api|unknown
--review-seconds N --reasoning-effort low|medium|high
--startup-seconds N --checks FILE
Config/check files are explicit trusted operator inputs, not PR instructions.
Expand All @@ -39,12 +42,12 @@ The development profile uses one inexpensive Sol session by default; --max-round
No automatic retry or model upgrade. Provider-specific model availability still applies.
--config overrides the repository's .code-qa.json read from the base revision.
Legacy --rigor mechanical|standard aliases map to L1/L3.
Reports are local; nothing is repaired, posted, pushed or merged.
Reviews stay local. Only explicit publish --authorize-publication posts selected, operator-verified findings; no repair, push or merge.
Exit 0: execution complete (may include findings); 2: partial/error; 3: stale/cancelled.
`;
try {
const { values: v, positionals } = parseArgs({ allowPositionals: true, strict: true, options: {
help: { type:'boolean' }, pr:{type:'string'}, repo:{type:'string'}, base:{type:'string'}, head:{type:'string'},
external:{type:'string'},result:{type:'string'},finding:{type:'string',multiple:true},verified:{type:'string',multiple:true},'authorize-publication':{type:'boolean'},help: { type:'boolean' }, pr:{type:'string'}, repo:{type:'string'}, base:{type:'string'}, head:{type:'string'},
out:{type:'string'}, plan:{type:'string'}, run:{type:'string'}, level:{type:'string'}, lens:{type:'string'},
application:{type:'string'}, question:{type:'string'}, purpose:{type:'string'}, requirements:{type:'string'},
path:{type:'string',multiple:true}, context:{type:'string',multiple:true},
Expand All @@ -56,6 +59,15 @@ try {
if (v.help || !action) { console.log(help); process.exit(0); }
if (positionals.length !== 1) throw new Error('Expected one command');
if (action === 'levels') { console.log(JSON.stringify(LEVELS,null,2)); process.exit(0); }
if(action==='import') {
if(!v.plan||!v.external||!v.out)throw new Error('import requires --plan, --external and --out');
const result=await importReport(JSON.parse(await readFile(v.external,'utf8')),JSON.parse(await readFile(v.plan,'utf8')));
await writeFile(resolve(v.out),JSON.stringify(result,null,2)+'\n',{flag:'wx',mode:0o600});console.log('Imported with external origin; execution remains unverified');process.exit(0);
}
if(action==='publish') {
if(!v.result)throw new Error('publish requires --result');
console.log(JSON.stringify(await publication(JSON.parse(await readFile(v.result,'utf8')),{ids:v.finding,verifiedIds:v.verified,authorize:!!v['authorize-publication']}),null,2));process.exit(0);
}
let plan;
if (['plan','review'].includes(action)) {
resolveLevel(v); // Reject unknown levels before network or any model use.
Expand Down
46 changes: 46 additions & 0 deletions review/exchange.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
import {readFile,writeFile,mkdir} from 'node:fs/promises';
import {join,resolve} from 'node:path';
import {createHash} from 'node:crypto';
import {parsePR,readPR} from './github.mjs';
import {packetFor,captureSourceEvidence,isFresh} from './snapshot.mjs';
import {validateReviewer} from './report.mjs';
import {identifyFindings} from './history.mjs';
import {command,requireSuccess} from './process.mjs';

// External assertions retain their origin; importing does not turn them into
// locally executed tests or operator-verified findings.
export async function importReport(envelope,plan) {
if(envelope.schemaVersion!==1 || !envelope.origin?.provider || !envelope.origin?.runId || !envelope.origin?.model || !Array.isArray(envelope.capabilities) || !Array.isArray(envelope.omissions))throw new Error('External report needs origin, model, run ID, capabilities and omissions');
if(envelope.report?.head!==plan.snapshot.head)throw new Error('External report targets a different head');
if(!await isFresh(plan.snapshot))throw new Error('Import plan is stale');
const packet=await packetFor(plan);await captureSourceEvidence(envelope.report,packet);
// No external check claims are trusted as local execution evidence.
validateReviewer(envelope.report,packet,[]);
return {schemaVersion:1,origin:envelope.origin,capabilities:envelope.capabilities,omissions:envelope.omissions,
report:envelope.report,plan,importedAt:new Date().toISOString(),execution:'external-unverified',
findings:identifyFindings(envelope.report,plan.pullRequest?.repository??plan.snapshot.repo,plan.snapshot.head)};
}
export async function publication(run,{ids,verifiedIds,authorize=false,execute=command}={}) {
if(!run.plan?.pullRequest)throw new Error('Publication needs a GitHub PR identity');
const identity=parsePR(run.plan.pullRequest.url);
if(!Array.isArray(ids)||!ids.length || new Set(ids).size!==ids.length)throw new Error('Select unique finding IDs');
if(ids.some(id=>!verifiedIds?.includes(id)))throw new Error('Publication needs explicit operator verification of each selected finding');
const findings=(run.findings??[]).filter(f=>ids.includes(f.id));
if(findings.length!==ids.length || findings.some(f=>f.disposition!=='actionable'))throw new Error('Only selected actionable findings may be published');
const head=run.plan.snapshot.head;
const checkFresh=async()=>{const current=await readPR(identity,execute);if(current.state!=='OPEN'||current.headRefOid!==head||current.baseRefOid!==run.plan.snapshot.base)throw new Error('PR changed; reassess before publishing');};
await checkFresh();
const packet=await packetFor(run.plan);
const report={...(run.reviewer??run.report),findings:findings.map(({id,originalHead,...f})=>f)};
await captureSourceEvidence(report,packet);validateReviewer(report,packet,run.checks??[]);
const priorResult=requireSuccess(await execute(['gh','api',`repos/${identity.repository}/pulls/${identity.number}/reviews`,'--paginate','--slurp']),'Read prior reviews');
const prior=JSON.parse(priorResult.stdout).flat();
const marker=f=>`<!-- code-qa:${head}:${f.id} -->`;
const pending=findings.filter(f=>!prior.some(r=>r.commit_id===head&&r.body?.includes(marker(f))));
const body=pending.map(f=>`${marker(f)}\n**${f.severity}: ${f.title}**\n\n${f.consequence}\n\nOwner: ${f.owner}.\n\nEvidence: ${f.evidence.map(e=>e.kind==='source'?`${e.path}:${e.line} (${e.side}) — ${e.quote}`:`${e.checkId}: ${e.quote}`).join('\n')}\n\nCounterevidence considered: ${f.counterevidence}`).join('\n\n---\n\n');
const preview={head,pr:identity.url,selected:ids,pending:pending.map(f=>f.id),body,authorized:authorize};
if(!authorize||!pending.length)return {...preview,status:pending.length?'preview':'duplicate'};
await checkFresh();
const response=requireSuccess(await execute(['gh','api',`repos/${identity.repository}/pulls/${identity.number}/reviews`,'--method','POST','--input','-'],{input:JSON.stringify({commit_id:head,event:'COMMENT',body})}),'Publish review');
return {...preview,status:'published',review:JSON.parse(response.stdout)};
}
1 change: 1 addition & 0 deletions review/plan.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ export async function createPlan(options) {
const profile = options.profile ?? 'development';
if (!Object.hasOwn(PROFILES, profile)) throw new Error('Unknown execution profile');
const defaults = PROFILES[profile];
if(ai && options.adapter==='claude' && !options.model)throw new Error('Select a Claude model explicitly; the development profile default is Codex-specific');
const plan = {
schemaVersion: 4, profile, requiredEvidence: [...preset.evidence], presetVersion: PRESET_VERSION, id: randomUUID(), createdAt: new Date().toISOString(),
snapshot: await snapshot(options.repo, options.base, options.head),
Expand Down
3 changes: 3 additions & 0 deletions review/report.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,9 @@ export function renderReport(run) {
`Environment: ${run.plan.environment}. Reviewer: ${run.plan.adapter}.`, '', '## Required evidence', '', ...run.plan.requiredEvidence.map(x => '- ' + x), '', '## Checks', ''];
for (const check of run.checks) lines.push(`- ${check.id}: ${check.status}${check.executed !== undefined ? ` (${check.executed} ${check.measure ?? 'assertions'})` : ''}`);
if(run.probes?.length) lines.push('', '## Test-strength probes', '', ...run.probes.map(p=>'- '+p.id+': '+p.outcome+(p.reason?' — '+p.reason:'')));
if(run.runtimeAssessment) lines.push('', '## Loaded runtime', '', run.runtimeAssessment.status, ...run.runtimeAssessment.reasons.map(r=>'- '+r));
if(run.artifacts?.length) lines.push('', '## Retained evidence', '', ...run.artifacts.map(a=>'- '+a.id+' ('+a.kind+'): '+a.path));
if(run.acceptance) lines.push('', '## Product acceptance', '', ...run.acceptance.deviations.map(d=>'- Deviation: '+d),...run.acceptance.gaps.map(g=>'- Unverified: '+g));
if (run.reviewer) {
if(run.reviewer.behaviors?.length) lines.push('', '## Behavior and test coverage', '', ...run.reviewer.behaviors.map(b=>'- '+b.behavior+' ['+(b.scope ?? 'in-scope')+']: '+b.layer+'; test '+(b.test || 'missing')+'; assertion '+b.assertion+(b.gap?' — Gap: '+b.gap:'')));
lines.push('', '## Reviewer assessment', '', run.reviewer.summary);
Expand Down
Loading
Loading