diff --git a/.commitrail/changes/hosted-smol-project-execution.md b/.commitrail/changes/hosted-smol-project-execution.md new file mode 100644 index 000000000..7b369a25f --- /dev/null +++ b/.commitrail/changes/hosted-smol-project-execution.md @@ -0,0 +1,426 @@ +# Hosted Smol project execution + +- Initiative: `None` +- Durable disposition: `Planned` +- Intended merge outcome: Replace the hosted Docker/runsc project controller with the existing native Smol node for typed trusted intake, ordered non-model PRE checks and authorized model-enabled POST checks. + +## Intent + +Give the hosted release one explicit Smol execution boundary without a fallback runtime or a second controller. Preserve the existing ART, CHECKERS, MODEL, AUTH and TASK ownership of material, grants, paid effects, immutable evidence and governed outcomes. + +## Current behavior + +`deploy/hosted/compose.yml` starts a networkless controller container with the host Docker socket. `preflight.py` requires runsc and Docker image-cache inspection. Trusted intake already uses a native Smol service through a separate typed capability. API and worker mount absolute scratch/socket paths; project execution uses the existing typed `ExternalCheckerExecutionAdapter` and factory. + +At the initial baseline `fcd32057b`, main rejected Smol model grants. Main `6b55fd450` now includes PR #677 and its guest-native model/trial transport, predecessor material, child VM custody and resource enforcement. This hosted change consumes that implementation and retains its independently scoped qualification evidence. + +PR #690 at `f3f818265`, incorporated in main `fcd32057b`, owns provider selection and replacement credential overlays. This change preserves that existing dependency rather than creating another provider-selection path. Provider source behavior remains outside this change. + +PRE model access is unsupported: `configured_model_profile_selector` rejects model selectors outside `post_submit`; `ModelAwareExternalExecution` and `model_execution_facts` require durable POST lineage. Selected PRE currently invokes the ordinary execution adapter without a model grant. This change preserves that rejection and does not claim PRE model support. + +## Bounded change + +### Allowed + +- `deploy/hosted/compose.py`, `compose.yml`, selected provider overlays, `compose.trials.yml`, `preflight.py`, `checker-service.example.json`, `release.env.example` and `README.md`: replace the controller topology, validate native-node identity/readiness and preserve credential isolation. +- `external_checkers/deploy/workstream-checker.service` and `external_checkers/README.md`: align the existing native service configuration and document operator-managed provisioning and shutdown. +- `scripts/verify_hosted_release_images.py`, `scripts/test_hosted_release_images.py`, `scripts/test_hosted_release_config.py` and existing focused deployment tests: qualify hosted backend images separately from native-node readiness, and exercise the rendered topology and meaningful refusal paths. +- Existing `experiments/pilot04_external_checker_runtime/` public journey helpers and their focused `scripts/` tests: attach to the actual hosted API and durable workers, retain scoped fault injection and verify native acknowledgements without substituting an in-process API or recorded worker. +- `.github/workflows/release-images.yml`: run the new hosted attachment/model qualification refusal tests in its existing frozen backend environment; include their paths without weakening or replacing existing image/runtime proof. +- Owned disposable EC2 qualification resources, source-pinned release builds and isolated database/release records and newly owned content-addressed keys within the existing approved AWS namespace: exercise the rendered hosted release without modifying shared services or production state. Reuse the existing simulated RS256 issuer fixture and label its identity proof accurately. +- `README.md`, `docs/decision_0014_external_service_adapter_convention.md`, `docs/architecture_checker_framework.md` and affected `docs/roadmap_status.md` claims: reconcile the intended hosted boundary and explicit evidence limits. +- This combined change record. + +### Not allowed + +- No runtime implementation changes in the PR #677 worktree, duplicate guest transport, fallback controller or provider implementation changes. +- No new adapter convention, service locator, backend business policy, schema migration, auth/payment authority, PRE model entitlement or paid retry. +- No shared service restart, `/etc/workstream` change, product deployment, cloud image publication, global cleanup or merge. Activation is limited to the owned disposable qualification database/release and the existing approved artifact namespace; it does not activate the real pilot project. +- Do not weaken existing credential, identity, material, deadline, cleanup or evidence checks to make a deployment fixture pass. + +## Design and decisions + +1. Use the existing delegated systemd/cgroup Smol service as the sole hosted node. Both typed intake and project execution clients use `/var/lib/workstream/checker-runtime/checker.sock`; their closed protocols and pinned image identities remain distinct. +2. Remove the Compose checker controller and its Docker socket/group grants. API and worker retain the same absolute scratch path and read-only node socket-directory mount. The native process runs as the same numeric UID/GID as the ART grant issuer. +3. Keep model credentials exclusively in the existing trusted model proxy. The native node forwards only invocation-bound guest capability traffic through the PR #677 transport; the project image receives neither global sockets nor credentials. +4. Replace runsc/image-inspection readiness with the existing typed project-execution health capability plus typed trusted-intake health. Validate configured Smol binary/guest/archive identities, native delegation and socket metadata without introducing a deployment controller. Configuration-only mode remains explicitly weaker than live readiness. +5. One trusted `WORKSTREAM_DEPLOYMENT_ROOT` selector derives identical absolute data/socket/model/parser paths on host and clients, defaulting to `/var/lib/workstream`. The root must be canonical, private and owned by the exact service identity; config, provider credentials and executable/cache assets stay outside it. `WORKSTREAM_CHECKER_SERVICE_BINARY` selects one absolute native service artifact under its required digest. This supports independently owned deployments without a test-only controller or validation bypass. +6. Operator provisioning and service lifecycle remain explicit. Compose startup validates the already-running native node; Compose shutdown drains producers/workers but does not stop an unrelated systemd unit. Shared nodes remain untouched; qualification lifecycle is limited to the newly owned deployment. + +### Live model qualification authority + +The user authorized proceeding with US$10 for the required live model test after +being told that an earlier setup charge remains unresolved. Enforce a separate +hard 10,000,000,000-nanodollar ceiling for this qualification through the existing +project budget and reservation ledger before dispatch. Preserve the earlier +uncertain effect separately; do not mark it free, retry it, reset its accounting, +or count it as reconciled. Record actual settled and outstanding reserved amounts +for the new test. Stop paid dispatch on ambiguous delivery or exhausted budget; +only the existing authorized recovery rules may resolve an effect. + +## Acceptance criteria + +- [x] Rendered hosted services contain no checker controller, Docker socket grant or runsc dependency; backend clients use one host Smol socket for the two existing typed capabilities. +- [x] Hosted configuration rejects Docker/runsc selection, missing or substituted Smol binaries/cache assets, unsafe identities, unbounded capacity and incorrect native socket/delegation. +- [x] Readiness uses registered typed adapters, validates exact runtime/image identities and fails closed before launch when the native node is unavailable. +- [x] Provider overlays retain PR #690 credential selection and expose credentials only to the trusted proxy; no provider source behavior changes. +- [x] Ordered non-model PRE remains supported; PRE model selectors remain unsupported and documented. Model-enabled POST is explicitly dependent on PR #677 and PR #690. +- [x] Focused contract tests cover valid composition and identity/credential/runtime substitutions. Relevant existing deployment and adapter checks continue to pass. +- [x] Documentation distinguishes prepared configuration and fixture proof from actual public API/worker, real model, hosted Flow/AWS and interruption qualification. The full public journey remains root-owned evidence. + +## Risk and review routing + +- Risk class: `L1` +- Required reviewers: architecture/reuse, security, QA/test delta, CI integrity and documentation, routed to the changed boundaries. +- Human review focus: the single native endpoint, preserved absolute paths/UIDs, provider-secret isolation, dependency order and the explicitly unsupported PRE model branch. + +## Evidence + +| Claim | Command or proof | Result | Remaining uncertainty | +|---|---|---|---| +| Current deployment and typed ownership | Hosted Compose/preflight, existing systemd unit and actual public API/Celery/AWS proof at `6c483ca61` | Both policy branches passed | Fixture identity, not production Flow | +| Provider overlap | PR #690 at `f3f818265`; main `fcd32057b`; scoped OpenRouter continuation at `5ab96251f` | Existing provider owner preserved; one real response settled | Pinned-rate ledger, not provider-invoice reconciliation | +| Smol callback transport | PR #677 controlled composition/direct proofs and three active nested interruption cases; hosted live POST below | Scoped native and hosted proofs passed | No PRE model entitlement or full-project quality claim | +| Rendered topology and refusal paths | `python3 -m unittest scripts.test_hosted_release_config scripts.test_hosted_release_images` | 70 passed, exit 0 | Contract/image fixtures; no native VM or provider execution | +| Generated readiness client | Execute the actual generated script using pinned backend Python, real Settings, registered typed factory and a local framed Unix health server | Passed; exact typed health request observed | The responder is a local protocol fixture, not a Smol node | +| Source hygiene | Ruff, Markdown links, stale wording and `git diff --check` | Passed | Does not replace runtime or independent review | +| Public lifecycle and real model behavior | Public baseline at `6c483ca61`; same-effect live continuation at `5ab96251f` | Human/automatic baseline branches and live model acceptance passed | Synthetic engineering guide/submission; production identity, setup rehearsal and full project remain separate | + +## Review findings + +The first actual hosted baseline reached the correct failed/passed POST pair and +`review_pending`, then exposed an invalid proof assumption: production +`SafeJsonFormatter` deliberately omits Celery's free-form messages/results, so +the attachment's raw-log duplicate matcher can never observe completion. Repair +only the existing attachment, both duplicate-delivery callers and their focused +tests: subscribe to a private bounded Celery event queue before publishing one +explicit task UUID, observe that exact worker/task's received and successful +duplicate refusal, then retain the existing immutable database/account checks. +Enable task events only for the owned worker and restore its prior state; remove +the obsolete log matcher. No product logging, worker configuration, result +backend, task implementation or paid retry changes are allowed. Verify real +event routing with focused positive/negative controls, rerun the hosted baseline +with retained failure evidence, and independently review the observer before +the live model call. + +The replacement subscribes before dispatch and waits for a real heartbeat from +the exact worker, because Redis installs fanout subscriptions during event +draining. It uses separate, finite event and control/publish connections; a real +Redis/Celery control reproduced a missing restoration acknowledgement when those +consumers shared one connection. The corrected control passed with the production +closed formatter, no raw success messages, one exact succeeded task event, and +both owned processes reaped. This synthetic task qualifies the observer mechanism, +not the real OUTBOX or public journey. The combined 29 attachment/model tests +passed, including wrong task/worker/result, reversed ordering, failure, absolute +deadline, prior event state and restoration failure. An in-memory broker fixture +race was removed by finishing its heartbeat producer before queue deletion; +concurrent real heartbeats remain covered by the Redis control. Ruff and independent +source review passed. The failed hosted attempt remains retained; the next +baseline uses a fresh owned database and Redis logical database before any paid +claim. + +The second hosted baseline observed the actual duplicate OUTBOX delivery and +verified unchanged retained outcome, then failed while assembling its report: +the hosted branch still referenced the local result-backend task variable. The +report now retains the actual task UUID returned by the scoped event observer; +the local branch retains its result-backend UUID. Its observation label is also +corrected to `scoped_worker_events`. The complete hosted branch, including report +assembly, must pass on fresh owned fixtures; neither partial run is a baseline +PASS and neither made a paid model call. Product services and images are unchanged. + +The corrected public hosted baseline passed at `6c483ca61`, including both policy +branches and exact duplicate refusal. Its model qualification then stopped before +materialization or any model effect: the planned 600-second checker needs 720 +seconds including the existing ART and cleanup allowances, but the expiry fixture +configured only 360 seconds. The deadline owner correctly refused execution. +The bounded proof repair adds a read-only runtime-window check before claiming a +paid project, compares actual hosted and driver settings, and derives observation +time from the stored policy and existing owner margins. Allowed files remain the +two hosted proof modules, their focused tests and this record. No policy, budget, +claim, Submission or original expiry receipt may be reset or rewritten. Restore +the normal operational ART window only for future leases and use the existing +public Operator recovery for the same failed run/project/account. No new product +retry path or generic resume framework is authorized by this repair. + +The repair also shares the existing outcome observer with an explicit recovered +dispatch, checking its original project, task and Submission before observation. +The paid-custody, governed-outcome and replay assertions remain unchanged. All 33 +combined attachment/model tests passed; the new pre-claim check fails against +the previous entrypoint. Root and independent runtime review passed the ordering, +locked-policy budget, float-settings conversion and preserved validation tail. +Ruff and diff checks passed. This proof-only repair does not change production +backend, native binaries or checker images; it does not establish a live provider +result or the still-pending active nested recovery cases. + +The first read-only continuation preflight then caught a proof-only assumption: +the hosted release does not override the checker socket timeout, so indexing +that optional key failed before opening the claim or dispatching recovery. The +guard now uses the canonical Settings field default when absent and still +requires the driver's resolved setting to agree. A missing-key regression uses +the actual Settings defaults and rejects a divergent driver. The original claim, +account and failed run are unchanged; no provider effect was created. +All 34 combined tests passed; loading the previous guard into the new test +reproduced the exact missing-key error. Independent review and Ruff passed. +The operations guide also explains drained scratch-root replacement when limits +change, preserving the old root and refreshing container and systemd mounts; +independent review confirmed the canonical ART marker and lifecycle ordering. + +The same-account recovery reached a prepared model effect, but stopped before +request-evidence binding, budget reservation or provider dispatch. Inspection +identified a missing bootstrap prerequisite: the canonical journey service list +omits `workstream.artifact.checker_output`, required by both checker and setup +model evidence storage. Extend `backend/scripts/provision_local_service_actors.py` +and the focused `backend/tests` checker-output custody regression, using existing +public service-actor provisioning and unchanged authority. Prove denial with no +writer, then verified storage after canonical provisioning on PostgreSQL/MinIO. +No permission, schema, ledger, policy or retained-effect reset is allowed. Any +continuation must use the existing recovery owners and original project account. +The proof must also recognize the owner's supported prepared-effect recovery: +one newly authorized successor grant can reference the same paid slot/effect as +its failed predecessor. Retain the exact predecessor grant/call snapshot before +retry, verify its immutable identity and direct failed-run lineage afterward, +and continue requiring exactly one paid slot and effect. The initial path still +requires one grant; recovery permits exactly that retained predecessor plus the +current grant, with the same request hash and no additional spend opportunity. +The existing model proof module and its focused tests own this bounded check; +production grant, retry and ledger operations remain unchanged. +Both real custody regression variants (Local and MinIO, with PostgreSQL) passed; +the old bootstrap list, substituted only in memory, fails at request-evidence +storage with the expected authority error. Isolated database, role and MinIO +cleanup passed. All 35 combined hosted attachment/model tests passed, and the +previous one-grant proof guard rejects the new supported-recovery test. Root +review passed the canonical bootstrap change; independent review passed the +bounded prior-grant proof. No live provider outcome is established by these +regressions. The failed hosted recovery retains one prepared effect, no ART +request evidence and a zero-reserved/zero-settled account for later exact recovery. + +Root plan review of `e07f28b4a`: PASS WITH LOW RISKS for implementation only. The review required exact numeric identity, native/Compose path parity in both directions, native health before API/worker release and explicit drain ordering. Runtime signoff remains separate. The root subsequently approved one trusted deployment-root selector and explicit pinned native binary path because fixed production paths prevented an isolated rendered-topology proof without touching shared resources. Exact UID/GID and live health requirements remain unchanged. + +Cold-start inspection found a schema/activation/proxy/node cycle: the proxy reads the activated ART namespace, the native node requires its model socket, and API normally applies migrations. The fixed `migrate`, AWS `proof` and exact proxy-only `up -d model-proxy` stages therefore validate static custody while releasing no application producer/worker; full `up` still requires both typed native health checks. AWS proof itself uses DB/AWS directly and retains its exact existing credential-separated observations. + +The actual `docker/backend/release-entrypoint.sh` API role only loads its four role secrets and executes the supplied command; it performs no application startup, activation or readiness call. `backend/alembic/env.py` loads settings and model declarations, then runs migrations against PostgreSQL. A focused entrypoint control reaches its supplied command with absent node/proxy sockets and no model-provider files. This is source/entrypoint evidence, not a fresh-database migration or activation drill. Operator instructions explicitly require root/approved sudo for private input traversal and Docker access; runtime UID/GID `10001:10001` receives no daemon privilege. + +Controller-only tests were replaced by native custody/readiness and second-controller refusal proof; API/worker mount, secret isolation, parser and producer/worker drain guards are retained. The generated readiness script was exercised through real Settings and the registered typed factory: the material root requires the matching ART scratch setting, which is now supplied from the same derived path. This proof uses a local framed Unix responder and makes no VM or paid-provider claim. + +Exact-head implementation review follows this clean tested checkpoint; no passing implementation verdict is claimed here. + +The native-hosted image verifier proves both backend SDK images without requiring +the separate ordinary Docker controller artifact. Its closed evidence explicitly +names that scope; full native preflight still owns service/guest/cache readiness. +The Docker artifact proof remains in CI for the existing local agent-loop owner. +The existing wrapper exposes only exact worker stop/start for the public replay +proof. Stop retains the configured warm-drain period; start revalidates private +inputs and both native capabilities before reconciling that one worker container. +Smol HOME limits use filesystem bytes, matching the pinned runtime socket layout. + +## Reconciliation + +- Current-source reconciliation: start from main `ae788e6a8`; preserve the pinned PR #690 dependency and reconcile main `fcd32057b`, including its current CLI/roadmap changes; consume PR #677 runtime transport without copying it. The main reconciliation has no source conflicts and changes no hosted executable code. +- Next usable boundary: consume the qualified native/hosted composition in the separately owned real-project approval and contribution journey. +- Remaining risks: production Flow identity, substantive model-grading quality, the full project journey and unsupported PRE model custody remain explicit. + +### Public hosted attachment custody + +The public journey driver now attaches to the rendered API, Celery worker, Beat, +parser and optional proxy and verifies their exact image, command, environment, +mounts, ports and isolation. It uses a labelled simulated RS256 issuer and +manager-authored fixture policies; it does not qualify setup-agent behavior or +live model-enabled POST. Duplicate delivery is observed through the production +worker's exact task UUID and immutable database/outbox facts. Original intake +expiry is measured against PostgreSQL time without rewriting its deadline. +The existing preparation deadline is exposed consistently to all hosted roles; +its default remains 3400 seconds and the existing Settings constraints remain. + +Independent review reproduced an undeclared writable `/app` tmpfs passing the +initial container validator. The corrected validator matches rendered tmpfs +targets and options, rejects that substitution, and also inspects the actual +parser with its declared capabilities and resource limits. The regression fails +against the earlier validator. Nineteen focused attachment/Smol-proof tests pass; +the root independently inspected the repair and reran its five custody cases. +These are fixture and source checks, not a completed hosted journey. + +The existing AWS role grants only the approved `workstream/artifacts/sha256/` +object prefix. Qualification reuses that immutable namespace without IAM changes, +using a separate database/release and nonce-bound challenge objects. Existing +same-digest objects remain untouched. Record newly created durable object keys +and their retention separately from disposable container/VM cleanup; do not +claim deletion of the shared namespace or bypass its immutability policy. + +### Bounded live model POST qualification + +The public hosted model qualification extends the same inspected deployment and +public guide/submission helpers. One exclusive private proof-state file owns one +isolated project and its immutable US$10 lifetime account; a rerun must inspect +that state, not create another paid project or reset its counters. The existing +Terminal Systems `solver_brief_and_extension_disclosure` image rule supplies a +one-call, model-only POST. Manager-authored engineering source and rubric bindings +remain explicit; this is not a setup-agent or real pilot-project rehearsal. + +The OpenRouter profile pins `openai/gpt-5.6-terra`, full-context reservation and +conservative tier/cache rates, one call with finite bytes/tokens/deadline, and no +provider tools. Credentials remain solely in the trusted proxy. The user confirmed +that no optional OpenRouter plugins were enabled. The existing Operator profile, +budget and registry routes, separate policy approvals, public claim/start and ZIP +intake lead to a real Submission and durable Celery dispatch. Actual model output +is evaluated without manufactured findings: a pass follows shared automatic +acceptance, a valid work failure follows revision, and ambiguous/infrastructure +outcomes stop further paid qualification. + +Exact public replay and duplicate outbox delivery must preserve checker effects, +model slots, ART request/response evidence and account counters. Retain settled +and outstanding reserved amounts within the total cap; conservative ledger +charges are not represented as an external provider invoice. Any later live +model-plus-trial proof must reuse this same project/account and its remaining +headroom. No separate US$10 allowance or automatic paid retry is introduced. + +### Hosted fault-supervisor qualification + +The hosted proof separates root-owned operator controls from a fixed +`10001:10001` runtime sibling. The existing node configuration remains mode +`0400` on its original inode and is restored without changing ownership or +permissions. Writes reject file substitution, aliases, extra links and foreign +ancestors. A root-owned sticky ancestor is allowed only above the validated +private directory. Runtime inventory commands drop root and supplementary groups; +the runtime cannot read the operator controls or acquire operator authority. + +The root independently reviewed the three-file change at `5e28b089e`. +Twenty-five focused supervisor/attachment tests passed, including refusal of a +FIFO, foreign ancestry and substituted configuration. An isolated EC2 custody +probe confirmed the identity split and exact configuration restoration; that +probe preceded the final nonblocking-open addition, which has focused test proof. +The canonical fresh hosted migration also passed after correcting the unused +fixture secrets to the required padded Base64 encoding. Product validation was +unchanged. These results do not establish the pending public hosted journey or +live model result. + +The five-file model qualification at `ea49554b7` passed 37 focused tests. Root +review verified the public profile/budget/registry owners, exact model effect +inspection DTO, paid-slot joins, fixed deployment claim and full lifetime budget. +The exclusive claim is synced with its parent directory before public mutations; +changing the input path cannot allocate another paid project. Both evaluation +and completion duplicates are observed through real Celery task identities, with +unchanged model, grant, slot, account and governed-outcome facts required. No +live provider call is established by these tests. + +Actual systemd startup then refused `User=10001` with `217/USER` because the +host had no account/group entries for that identity. Bounded controls reproduced +the refusal both without and with numeric KVM supplementary-group selection. +Host preparation now explicitly requires a dedicated locked no-login/no-home +account and primary group at the fixed IDs, with collision refusal and no Docker +or sudo grant. The isolated node provisions only its newly owned service identity; +existing accounts, services and `/etc/workstream` are preserved. This fixes the +missing prerequisite without changing node privileges or execution behavior. + +Fresh baseline setup also lacked a usable registry ordering: it expected +registered capabilities before bootstrapping the Operator allowed to register +them. At `9db6d68b1`, the existing setup grants public Operator authority and then +registers exactly the supplied PRE/POST capability identities through the public +registry. Their image comes from the independently validated native cache; typed +receipts must preserve the full spec digest, operation and actor. Root review and +14 focused attachment tests passed, including refusal before mutation for a +missing/ambiguous image and refusal of substituted replies. No hidden owner +operation or second bootstrap was added. The release-image workflow now runs +the attachment and model-qualification guard tests in its frozen backend +environment, without changing existing deployment proof or timeout bounds. + +Independent review passed the added CI step and its exact path triggers. Its +first hosted run (`40373193281`) then reproduced an order-sensitive test fixture: +the stalled-admission guard set a broker environment value after settings had +already been cached. The test now scopes that value and clears settings before +worker import and during cleanup; its no-verifier-dispatch assertion is intact. +The exact combined two-module command with the broker environment unset failed +one of 26 tests before `f406310f7` and passed all 26 afterward. Root review, Ruff +and diff checks passed. Product settings and release images are unchanged. + + +### Live hosted model continuation + +Exact deployment source `5ab96251fc08e0c8cd3e8128bc9fb4f4fc8ed2bf` +(tree `3b4f8be16fe3ca7915add5fc24a08f01f1c5dd2c`) rebuilt both backend SDK +images and their fixture-CA derivatives. The unchanged native-hosted image +verifier passed (evidence SHA-256 +`05b5dea9a8567f5b0c8331b4fdff4b536ff9574af2bcb9970dd9e314bcae966b`). +Release `20c965fa8cf3d2a45e30ae9bdc78b105db8d62de6231fd8cf0ec525447f0300f` +passed canonical AWS and native/attachment readiness checks. Native service, +guest, launcher and project image pins remained unchanged from the qualified +runtime. Compared with baseline `6c483ca61`, backend changes are exactly the +canonical service bootstrap and its real custody regression; application, +Alembic, dependency and image-recipe trees are identical. Later runtime merges +change only the recovery observer and its documentation. + +The canonical public service bootstrap provisioned the missing existing model +output writer. It changed neither the original project account nor its prepared +effect. Read-only inspection passed with receipt SHA-256 +`e9d0a6852eeeae6fe297c0581880b048437dcce9496c3265fc7b62cf49e6b60d`. +One explicit public Operator retry then advanced generation 2 to generation 3 +on the same immutable Submission. It reused the same paid slot, prepared effect +and exact request hash; the old grant and complete earlier failure history were +retained. No new project, budget allocation, Submission or paid-call identity +was created for the continuation. + +The real OpenRouter response graded the disclosed synthetic +`solver_brief_and_extension_disclosure` fixture as passed. The durable worker +and completion delivery reached `accepted`, retaining exactly one Submission, +one FinalAcceptance and one submitter ContributionRecord, with zero Reviews and +zero awards under the unpaid policy. Private request and response ART bindings +and public redacted effect inspection were verified. The ledger settled +16,527,000 nanodollars (US$0.016527), within the authorized US$10 ceiling, and +released the outstanding reservation to zero. This is the ledger's conservative +charge at locked rates, not provider-invoice reconciliation. The earlier +uncertain setup charge remains separately unresolved. +Independent arithmetic confirmed 2,319 uncached input tokens at 5,000 +nanodollars and 274 output tokens at 18,000 nanodollars equal the recorded +16,527,000-nanodollar settlement; cached input was zero. The selected provider +and model were `openrouter` and `openai/gpt-5.6-terra`. + +Exact public Submission replay and two real duplicate Celery deliveries retained +identical account, effect, grant, slot, outbox and governed-outcome facts. The +native owned-resource inventory was empty and the foreign control remained +present. Full continuation receipt SHA-256: +`bd96a3b0b2b8609cc7f0c298aae0d86721ca5ee4346a613b8e030aa963cd1fd1`, retained at +`/var/tmp/workstream-hosted-smol-20261026/assets/node-control/model-prepared-continuation-receipt.json`. +The root independently rehashed the 15 preparation artifacts and inspected this +receipt; a separate read-only PostgreSQL snapshot confirmed the same completed +effect, current run and zero-reserved account. + +The original baseline and failed attempts remain immutable. A private startup +observer used the wrong parser role name and failed after actual startup; +separate exact-role inspection confirmed all five healthy roles without +restarting them or weakening product readiness. The live proof uses simulated +RS256 identity and one synthetic rubric. It does not establish production Flow +identity, a live setup-agent rehearsal, full trial execution in that paid call, +substantive rubric quality, an approved revision journey or the complete Terminal +Systems project owned by #555. PRE model selectors remain unsupported. + +The reviewed success-only cleanup then stopped producers, drained the worker, +verified native drain, finalized the supervisor while the proxy remained live, +and stopped the native node and its clients. Cleanup receipt SHA-256 +`a0d171fbeee72c7f81d1adf8fffa90936d43e52641497afc029ff22babc6035c` +and independent final inventory SHA-256 +`11095053e9b24add353b1cae1aa06fc0e82f94b916fd33eac06b87036ee5f5aa` +confirm stopped owned services/timer, empty VM/lease/cgroup/relay and ART scratch +inventories, and no outstanding reservation. The fixed claim retained inode +4790559 and final SHA-256 +`92ae55c5ec91940b64ff934f2b143c24b95796e48d6b613a966150442761a116`. +Database, object-store evidence, original failure history and shared services +were preserved. No global cleanup or additional model call occurred. + +### Asset diagnostic review correction + +CodeRabbit identified that `_verify_smol_asset` caught its own `PreflightError` +through the generic `RuntimeError` handler, replacing specific refusal reasons +with an unavailable-file message. Preserve explicit path, custody, deadline and +digest/identity diagnostics; continue normalizing underlying filesystem/runtime +errors and refusing launch. Scope is the existing preflight, its hosted-config +tests and this record. No asset acceptance, authorization, runtime or provider +behavior changes. Verify real temporary files for each diagnostic, ordinary +missing-file normalization, the existing readiness tests and a pre-fix control. +The merge from main `6b55fd450` has no source delta beyond the already consumed +PR #677. Roadmap capabilities and remaining gates are unchanged by this +diagnostic-only correction. + +Validation: the new diagnostic regression failed against the original handler +for all four explicit refusals, while its missing-file control passed. After +the correction, all 71 hosted configuration/image tests passed. Ruff check, +formatting and diff whitespace checks passed. No paid call or deployment was +needed: the only executable change preserves already-raised error messages. diff --git a/.github/workflows/release-images.yml b/.github/workflows/release-images.yml index 9cdb76e5b..b2c4b78ca 100644 --- a/.github/workflows/release-images.yml +++ b/.github/workflows/release-images.yml @@ -15,6 +15,8 @@ on: - 'deploy/hosted/**' - 'scripts/*hosted_release*.py' - 'scripts/test_public_smol_proof.py' + - 'scripts/test_public_hosted_attachment.py' + - 'scripts/test_public_hosted_model_post.py' - 'experiments/pilot04_external_checker_runtime/**' - 'scripts/test_local_guide_runtime.py' - 'scripts/test_guide_document_api_drill.py' @@ -76,6 +78,14 @@ jobs: scripts.test_local_guide_runtime scripts.test_guide_document_api_drill \ scripts.test_public_smol_proof test -z "$(git status --porcelain)" + - name: Verify hosted public journey and model qualification guards + timeout-minutes: 5 + shell: bash + run: | + set -euo pipefail + PYTHONPATH=backend backend/.venv/bin/python -m unittest -v \ + scripts.test_public_hosted_attachment scripts.test_public_hosted_model_post + test -z "$(git status --porcelain)" - name: Build exact release images without cache or publication timeout-minutes: 42 shell: bash diff --git a/README.md b/README.md index 2059dd0cc..644e06937 100644 --- a/README.md +++ b/README.md @@ -98,12 +98,17 @@ These parts work in a local test run: The local test run used recorded AI model output. It does not prove project setup with a live AI model. +An isolated hosted API/Celery/Smol/AWS run also passed both review-policy +branches. A separate live OpenRouter POST reached automatic acceptance with +recorded model usage and replay protection. These runs used fixture identity +and engineering submissions; they do not prove the full real-project journey. + These parts are not complete: - Live human review and revision after a human decision. - Reviewer contribution records, because live human review is not complete. - Proof of project setup with a live AI model. -- Operation on a hosted server. +- The full hosted project journey with production Flow identity. - Payment delivery. The [v0.1 Roadmap And Capability Status](docs/roadmap_status.md) gives the diff --git a/backend/scripts/provision_local_service_actors.py b/backend/scripts/provision_local_service_actors.py index 5dfda9c30..33dea1ed8 100644 --- a/backend/scripts/provision_local_service_actors.py +++ b/backend/scripts/provision_local_service_actors.py @@ -32,6 +32,7 @@ "workstream.artifact.verifier", "workstream.artifact.scheduler", "workstream.artifact.binding", + "workstream.artifact.checker_output", "workstream.artifact.guide_reader", "workstream.artifact.materializer", "workstream.project.setup", diff --git a/backend/tests/test_checker_output_custody.py b/backend/tests/test_checker_output_custody.py index e246e5d02..c8ca7fa54 100644 --- a/backend/tests/test_checker_output_custody.py +++ b/backend/tests/test_checker_output_custody.py @@ -47,6 +47,139 @@ def content_sha256(value: bytes) -> str: return "sha256:" + hashlib.sha256(value).hexdigest() +@pytest.mark.parametrize("provider", ["local", "minio"]) +async def test_canonical_public_provisioning_enables_exact_model_request_custody( + tmp_path, isolated_database_env, admin_access, provider, +) -> None: + """The journey bootstrap must provision the writer before any model request I/O.""" + from scripts.provision_local_service_actors import ( + JOURNEY_SERVICE_IDENTITIES, + idempotency_key, + subject_for, + ) + from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine + + from app.adapters.artifacts import create_artifact_store_bootstrap + from app.interfaces.artifacts import ArtifactStoreNamespaceClaim + from app.modules.actors.models import ActorProfile + from app.modules.artifacts.preparation import ( + ArtifactPreparationService, + ArtifactScratchManager, + ) + from app.modules.artifacts.service import artifact_storage_namespace_spec + from app.modules.checkers.api.output_custody import ( + CheckerOutputReservation, + CheckerOutputSelector, + CheckerOutputSlot, + ) + from tests.artifact_store_helpers import artifact_preparation_limits + from tests.checker_output_admission_helpers import seed_checker_output_relationships + from tests.checker_output_custody_helpers import ( + ObservedStore, + _output_storage_settings, + _prepare_model_attempt, + _provision_model_policy, + _store_model_request_evidence, + ) + from tests.checkers_model_calls.model_usage_boundary_support import BoundaryPorts + + if provider == "minio": + from tests.test_s3_artifact_store import provision_minio_bucket + + await provision_minio_bucket.__wrapped__() + engine = create_async_engine(isolated_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + settings = _output_storage_settings(tmp_path, provider, None) + bootstrap = create_artifact_store_bootstrap(settings, activation_guard=None) + namespace = artifact_storage_namespace_spec(settings, bootstrap) + store = ObservedStore(bootstrap.initialize_after_namespace_claim( + ArtifactStoreNamespaceClaim( + bootstrap.identity, bootstrap.namespace_identity, namespace.namespace_fingerprint, + ) + )) + manager = ArtifactScratchManager( + root=tmp_path / "request-scratch", limits=artifact_preparation_limits(), + ) + try: + policy, profile = await _provision_model_policy(factory, namespace, admin_access) + data = b'{"prompt":"publicly provisioned request custody"}' + async with factory() as session: + _, _, run_id = await seed_checker_output_relationships( + session, namespace, policy_bundle=policy, + ) + execution, attempt, _ = await _prepare_model_attempt( + session, run_id, content_sha256(data), model_profile=profile, + model_ports=BoundaryPorts(), + ) + await session.commit() + selector = CheckerOutputSelector( + paid_slot=attempt.slot_key, task_id=execution.task_id, + model_call_slot_id=attempt.slot_id, model_call_attempt_id=attempt.attempt_id, + slot_key="model_request", content_sha256=content_sha256(data), execution=execution, + ) + reservation = CheckerOutputReservation( + selector=selector, + slot=CheckerOutputSlot(key="model_request", maximum_bytes=profile.spec.maximum_request_bytes), + ) + identity = ServiceIdentity.ARTIFACT_CHECKER_OUTPUT.value + async with factory() as session: + assert await session.scalar(select(ActorProfile.id).where( + ActorProfile.service_identity == identity, + )) is None + arguments = dict( + factory=factory, preparation=ArtifactPreparationService(manager), store=store, + namespace=namespace, settings=settings, reservation=reservation, request_bytes=data, + ) + with pytest.raises(CheckerOutputUnavailable, match="authority_unavailable"): + await _store_model_request_evidence(**arguments) + assert (store.puts, store.opens) == (0, 0) + async with factory() as session: + for table in (ArtifactPutAttempt, ArtifactOperationReceipt): + assert await session.scalar(select(func.count()).select_from(table).where( + table.model_call_attempt_id == str(attempt.attempt_id), + )) == 0 + assert await session.scalar(select(func.count()).select_from(ArtifactBinding).where( + ArtifactBinding.resource_type == "checker_model_call", + ArtifactBinding.resource_id == str(attempt.attempt_id), + )) == 0 + effect = await session.get(ModelUsageEffect, str(attempt.attempt_id)) + assert effect.state == "prepared" and effect.request_binding_id is None + assert effect.reserved_nanodollars is None + + # Select through the actual canonical bootstrap list; removing the identity + # must leave this real storage call refused, rather than pass a constant test. + responses = [] + for service_identity in JOURNEY_SERVICE_IDENTITIES: + if service_identity == identity: + subject = subject_for(service_identity, "public-request-custody") + for _ in range(2): + response = await admin_access.signed.client.post( + "/api/v1/service-actors", + headers=admin_access.admin.headers | { + "Idempotency-Key": idempotency_key(service_identity, subject), + }, + json={"service_identity": service_identity, "subject": subject, + "reason": "Provision canonical model evidence writer"}, + ) + assert response.status_code == 201, response.text + responses.append(response.json()) + evidence = await _store_model_request_evidence(**arguments) + assert len(responses) == 2 and responses[0] == responses[1] + assert (store.puts, store.opens) == (1, 1) + assert evidence.content_sha256 == selector.content_sha256 + async with factory() as session: + binding = await session.get(ArtifactBinding, str(evidence.binding_id)) + assert (binding.resource_type, binding.resource_id, binding.logical_role) == ( + "checker_model_call", str(attempt.attempt_id), "model_request", + ) + assert binding.put_attempt_id == str(evidence.put_attempt_id) + assert binding.verification_receipt_id == str(evidence.verification_receipt_id) + finally: + manager.close() + bootstrap.close() + await engine.dispose() + + @pytest.mark.parametrize("provider", ["local", "minio"]) async def test_real_store_verification_and_binding( tmp_path, diff --git a/deploy/hosted/README.md b/deploy/hosted/README.md index de552cdce..193076ccc 100644 --- a/deploy/hosted/README.md +++ b/deploy/hosted/README.md @@ -1,42 +1,72 @@ # Hosted Release Configuration -This directory contains the first bounded release composition for the Workstream -API, durable Celery worker, Beat scheduler, fixed guide-document parser and -existing Rust external-checker service. It targets one dedicated Linux runner -host with Docker and gVisor `runsc` for project execution, plus a separately -provisioned KVM/Smol node for trusted ZIP intake. PostgreSQL, Redis, AWS S3, Flow identity and the optional OTLP collector -are external services managed by the approved environment. - -The files prepare a release; they do not provision or deploy one. A successful -render, image build or preflight is not AWS, Flow, recovery, performance or -hosted-isolation proof. Parent issue #555 owns the connected deployed evidence. +This directory prepares the Workstream API, durable Celery worker, Beat, +fixed guide-document parser and optional trusted model proxy on one dedicated +Linux amd64 host. Project PRE/POST and trusted ZIP intake use the existing native +Smol systemd node. Compose does not start a checker controller or grant access +to its Docker socket. PostgreSQL, Redis, AWS S3, Flow identity and the optional +OTLP collector remain environment-managed dependencies. + +This configuration depends on the guest-native model/trial transport from +[PR #677](https://github.com/Flow-Research/workstream/pull/677) and preserves the +provider selection from [PR #690](https://github.com/Flow-Research/workstream/pull/690). +A render, image build or preflight does not establish hosted operation or a real +model call. Parent issue #555 owns the connected public API/worker evidence. ## Trust boundary -The checker service controls the host Docker daemon through -`/var/run/docker.sock`. That is host-level authority. gVisor isolates the -untrusted checker containers launched by the service; it does not protect the -host from compromise of the trusted checker-service controller. Use a dedicated -runner host and a trusted daemon. Do not place unrelated workloads or credentials -on that host. +The native node uses KVM and its delegated cpu/memory/pids cgroup, under the same +numeric UID/GID `10001:10001` as the ART grant issuer. It owns fresh VM execution, +exact leases and cleanup. Neither the node nor project images receive provider +credentials. The optional Compose model proxy alone owns those credentials; +the native node connects to its host-visible private socket and forwards only +invocation-bound capability traffic through the Smol guest transport. -The checker controller has no network, runs as a fixed non-root UID/GID, drops -all Linux capabilities and mounts verified material read-only. Its supplemental -group grants only access to the Docker socket. The API and worker use the same -absolute host paths as the controller because the host daemon, rather than the -controller container's mount namespace, resolves checker bind sources: +The API and worker mount the node socket directory read-only. Both typed +capabilities use the same host endpoint, while retaining distinct closed +protocols and pinned images: ```text /var/lib/workstream/scratch /var/lib/workstream/checker-runtime/checker.sock -/var/lib/workstream/intake-runtime/checker.sock +/var/lib/workstream/model-runtime/model.sock ``` -Keep these absolute paths consistent across the host and clients. The intake -node owns its private socket directory; API and Celery mount it read-only. It -needs no Docker socket, provider credentials or project model credentials. -Trusted ZIP inspection and default checks run in its pinned guest image; project -PRE/POST still use the separately configured execution capability. +`WORKSTREAM_DEPLOYMENT_ROOT` defaults to `/var/lib/workstream`. An independently +owned deployment may select one canonical absolute private root; all scratch, +checker/model/parser socket paths below it are derived together, with identical +absolute source/target paths in the containers. The root must be mode `0700`, +owned by `10001:10001`, and contain no config, provider secret or executable/cache +asset. Symlink roots, ancestor aliases and data/asset overlap fail preflight. +`WORKSTREAM_CHECKER_SERVICE_BINARY` defaults to the unit's installed path and +selects one absolute artifact under `WORKSTREAM_CHECKER_SERVICE_BINARY_SHA256`. + +For an isolated proof, use a newly owned `/var/tmp/workstream-hosted-smol-*` +prefix with a private `data` subdirectory and config/assets as separate siblings. +Give UID `10001` traversal access to the owned prefix; do not rely on another +user's private home ancestry. Configure the existing scoped unit's `ExecStart` +and cgroup path for that exact binary/config and preserve its ownership. This +uses the same release topology and preflight, without shared `/etc/workstream` +or `/var/lib/workstream` mutation. No source here deploys or restarts that unit. + +These absolute paths and identities must match across native configuration, +API/worker containers and the proxy. ART material lives under the same scratch +root on host and clients; the native node independently verifies each opaque +ART grant before staging. Model proxy sockets and ART material are never assumed +to exist only inside a Docker namespace. Project guests receive guest-local +capability sockets, never the global host socket or a provider key. + +Ordered non-model PRE and approved model-enabled POST retain their existing +owners. PRE model selectors remain unsupported: PRE has no supported paid-model +lineage/grant operation, and policy validation rejects those selectors. This +configuration does not introduce PRE model authority. + +The [hosted qualification record](../../.commitrail/changes/hosted-smol-project-execution.md#live-hosted-model-continuation) +records the actual API/Celery/AWS baseline and one live OpenRouter POST through +this wiring. The model continuation preserves its original paid effect, reaches +governed acceptance and proves replay without another charge. It uses fixture +identity and a synthetic single-rubric submission; production identity and full +project grading remain separate qualifications. Guide parsing uses a separate private Unix socket at `/var/lib/workstream/guide-runtime/guide.sock`. The parser uses the immutable @@ -48,7 +78,7 @@ a distinct fixed identity (10003 or 10004), no supplementary groups and no capabilities. The broker and children verify their process protections before processing documents; startup fails if the required identity boundary is absent. The broker owns the socket directory and socket. Only the guide worker mounts that directory as -a read-only client. API, Beat, proof jobs and the checker controller do not +a read-only client. API, Beat, proof jobs and the native node do not receive it. The parser process validates directory and socket ownership before binding. The health check drops to client UID/GID 10001 with no capabilities before performing a framed protocol round trip against the fixed parser build identity. @@ -70,11 +100,9 @@ docker build --build-arg WORKSTREAM_PROJECT_AGENT_SDK_ENVIRONMENT=agents \ docker build --build-arg WORKSTREAM_PROJECT_AGENT_SDK_ENVIRONMENT=omnicore \ --file docker/backend/Dockerfile.release \ --tag registry.example/workstream/backend-omnicore: . -docker build --file docker/checker/Dockerfile.release \ - --tag registry.example/workstream/checker-service: . ``` -Both Dockerfiles pin their base images. Each backend build requires one exact SDK +Backend and intake image definitions pin their base images. Each backend build requires one exact SDK environment and installs only its committed `agents` or `omnicore` resolution, without the development extra. The image records that choice in a root-owned read-only marker. Select the worker image that matches the configured setup SDK. @@ -86,21 +114,43 @@ credential. `manager_authored` is accepted in either worker image with the exact `none/none/none` setup tuple; `WORKSTREAM_MODEL_PROXY_ENABLED` independently decides whether approved POST images can use the proxy. The backend image also installs the official Roles Anywhere helper 1.8.5 at its fixed path, with -architecture-specific hashes checked during the build and image proof. The -checker-service build compiles the committed Rust lock and copies the pinned -Docker client. A local build tag is build evidence only; the release file must +architecture-specific hashes checked during the build and image proof. Build the native service and static guest from the same reviewed source using the +[node instructions](../../external_checkers/README.md#smol-microvm-engine). Record the +installed service digest in `WORKSTREAM_CHECKER_SERVICE_BINARY_SHA256`; the +private node config pins Smol, the guest and every cached archive/config. A local build tag is build evidence only; the release file must contain the registry-resolved digest after publication. The backend build's uv package-download cache mount exists only on the builder; it is not retained in the image. Runtime image inspection uses `uv --no-cache pip check` and bounded temporary storage. -The scoped [release-image workflow](../../.github/workflows/release-images.yml) +For native hosted backend-image proof, use the existing verifier with the +explicit `native-hosted` scope from a clean checkout of the exact source: + +```sh +python3 scripts/verify_hosted_release_images.py --scope native-hosted \ + --expected-head --container-owner - \ + --backend-agents-image \ + --backend-omnicore-image \ + --evidence /private/backend-image-evidence.json +``` + +This scope proves both backend SDK images without requiring a checker controller +OCI image. It rejects `--checker-image` and records a closed backend-only +evidence shape. Installed native service, Smol, guest and cache identities remain +the full preflight's responsibility. Backend-image proof establishes no native +readiness or hosted journey result. + +The [release-image workflow](../../.github/workflows/release-images.yml) runs for release definitions and copied backend/checker source changes, building the exact PR head without cache, publication or deployment credentials. It first requires the canonical AWS Settings owner, then checks the copied application/migration files, lockfiles, secret loader, production dependency closure and default image commands. Bounded networkless containers verify the -Docker CLI and the checker's expected missing-configuration failure; they do not +Docker CLI and the checker's expected missing-configuration failure for the +separate controller artifact still used by the opt-in +[local agent-loop overlay](../../docker-compose.agent-loop.yml). The default +`all-release-images` verifier scope retains that artifact's regression proof; +it is not a native hosted deployment prerequisite. These checks do not start Workstream or run a project checker. Its fixed JSON artifact records source identity and local image IDs, not publishable repository digests. Full hosted operation remains separate evidence. The proof builds and inspects both SDK @@ -109,48 +159,57 @@ that no smoke container owned by that run remains after inspection or timeout. ## Prepare the host -Install and configure Docker and gVisor using the approved host process. Confirm -that Docker reports a `runsc` runtime. Installation and daemon configuration are -outside this repository configuration. When model access is enabled, the outer -checker runtime must include exactly `--host-uds=open` so its invocation-local -Unix socket works under gVisor; full preflight checks Docker's reported runtime -arguments. This permits the mounted socket, not network access or a global proxy -mount inside the image. A separate trial runtime remains configured with host -Unix sockets disabled. +Install Docker for the backend containers and the pinned native Smol bundle +for Linux amd64 with `/dev/kvm`, cgroup v2 and systemd 254 or later. Use the +[existing unit](../../external_checkers/deploy/workstream-checker.service), which +runs as `10001:10001`, places its controller in a delegated subgroup and preserves +independent guardians with `KillMode=process`. Review its explicit deployment +CPU/memory/PID/storage capacity against intake, checker and sequential trial +peak demand; this configuration does not increase registered checker limits. + +Provision a dedicated host service account and group at UID/GID `10001:10001` +before starting the unit. Numeric file ownership alone is insufficient: systemd +must resolve this account, otherwise it can fail with `217/USER`. Inspect +`getent passwd 10001` and `getent group 10001` first. If either ID belongs to an +unrelated identity, stop and resolve that host allocation; do not modify or reuse +the unrelated account. When both IDs and the name `workstream-checker` are unused: + +```sh +sudo groupadd --gid 10001 workstream-checker +sudo useradd --system --uid 10001 --gid 10001 --no-create-home \ + --home-dir /nonexistent --shell /usr/sbin/nologin workstream-checker +``` + +Keep the account password locked and without a login or home directory. Grant no +sudo or Docker access. The unit's `SupplementaryGroups=kvm` supplies KVM access; +ensure that host group owns `/dev/kvm`. Existing dedicated service identities +must have the same UID, primary GID and restrictions. -The hosted profile fixes backend processes at UID/GID 10001 and the parser broker -at UID 10002 with the shared client group 10001 after its identity bootstrap. -Find the Docker socket GID and -create the fixed private directories with these exact identities: +Create canonical private roots on durable host storage: ```sh +sudo install -d -m 0700 -o 10001 -g 10001 /var/lib/workstream sudo install -d -m 0700 -o 10001 -g 10001 /var/lib/workstream/scratch sudo install -d -m 0700 -o 10001 -g 10001 /var/lib/workstream/checker-runtime -sudo install -d -m 0700 -o 10001 -g 10001 /var/lib/workstream/intake-runtime sudo install -d -m 0700 -o 10001 -g 10001 /var/lib/workstream/model-runtime sudo install -d -m 0700 -o 10001 -g 10001 /var/lib/workstream/model-relays +sudo install -d -m 0700 -o 10001 -g 10001 /var/lib/workstream-checker/smol +sudo install -d -m 0700 -o 10001 -g 10001 /var/lib/workstream-checker/images sudo install -d -m 0750 -o 10002 -g 10001 /var/lib/workstream/guide-runtime -stat -c '%g' /var/run/docker.sock ``` -The checker and model runtime directories must be on durable host storage, not `/run`, -tmpfs or ephemeral container storage. Its private `.workstream-checker-owner` -file identifies containers recoverable after a service or host restart. Only the -checker service has a writable mount; API and worker socket mounts are read-only -so clients can connect but cannot replace recovery custody. Preserve the file -with the directory; never clone it into another live deployment. Before -changing or deleting this directory, drain checker work and verify cleanup of -its exact owned containers. If custody was lost, reconcile independently verified -container IDs; a new owner will not adopt old containers. See -[container recovery ownership](../../external_checkers/README.md#container-recovery-ownership). - -Provision the [Smol node and pinned trusted image](../../external_checkers/README.md#trusted-submission-intake) -under UID/GID 10001 before launch. Its socket must be mode `0600` inside the -`0700` intake directory. Preserve its owner and lease files across restarts. Set -`WORKSTREAM_ARTIFACT_INTAKE_IMAGE_DIGEST` and -`WORKSTREAM_ARTIFACT_INTAKE_CONFIG_DIGEST` to the exact cached manifest/config; -project registrations cannot select that privileged purpose. Account for its -2 GiB guest memory plus VMM overhead when setting the delegated node budget. +Preserve the node socket parent's persistent owner/lease custody and all trial +root markers across restarts. Never clone them into another live deployment or +replace them with `/run`, tmpfs or container storage. Do not delete retained +inputs, product evidence, shared roots or unknown runtime resources during +recovery. Unknown ownership or cleanup refuses admission. + +Provision project images and the pinned trusted intake image using the existing +[Smol cache command](../../external_checkers/README.md#smol-microvm-engine). +The native `trusted_intake` configuration selects the intake image; project +registrations cannot select that privileged purpose. Its image/config pins must +match the two `WORKSTREAM_ARTIFACT_INTAKE_*_DIGEST` values. Both typed clients +connect through the one mode-`0600` socket in the mode-`0700` checker directory. Copy `release.env.example` to an operator-owned non-secret environment file and replace every example identity. The file contains paths and public configuration, @@ -179,13 +238,18 @@ startup, and an unrelated secret is neither mounted nor loaded. The loader independently requires the running identity and mode `0400`, performs no shell evaluation and never prints a value. Do not put AWS static keys in these files. -With the proxy enabled, the trusted checker service also mounts the global -model socket directory read-only. Its separate writable -`/var/lib/workstream/model-relays` directory has the same absolute path on the -Docker host and in the service, owned by the runtime UID/GID with mode `0700`. -The service waits for the proxy to become healthy and mounts only the current -invocation's relay into a project image. Project images receive neither the -global socket nor provider credentials. API, Beat and proof jobs receive +Before setup or POST model calls, provision the existing service actors through +the public administrative API, including `workstream.artifact.checker_output`. +This principal admits private model request/response evidence; configuring the +proxy and provider credentials does not provision it. The canonical journey +service bootstrap includes it for both setup and checker model use. + +With the proxy enabled, the native checker service reads the global +model socket from its canonical host path. Its separate writable +`/var/lib/workstream/model-relays` directory is private to the native service, owned by the runtime UID/GID with mode `0700`. +Start the proxy before the native node. The service creates an invocation relay +and the Smol transport exposes only its guest-local capability endpoint. Project +images receive neither the global socket nor provider credentials. API, Beat and proof jobs receive neither model directory. The two provider file paths are trusted release configuration. The proxy loads only configured dedicated files as `OPENROUTER_API_KEY` or `OPENAI_API_KEY`; this is @@ -274,26 +338,69 @@ canonical application owner validates eligibility at startup. Replace the exampl drill; satisfying the arithmetic does not prove live latency, AWS identity or activation. -Copy `checker-service.example.json` to the configured private path, mode `0400` -and the same runtime owner. Replace the example cache with every approved -single-platform checker image supplied by the project checker/registry work. -The example includes the exact `model_proxy_socket` and `model_relay_root` -selectors for a proxy-enabled release. Remove both keys when -`WORKSTREAM_MODEL_PROXY_ENABLED=false`; preflight rejects partial, substituted -or disabled-proxy selectors. Rust consumes these paths from the private JSON -configuration, not environment variables. -For each entry, preserve the exact repository, platform manifest digest/media -type/byte count, Linux architecture and Docker config image ID. Pull every -`repository@sha256:...` identity before preflight. Full preflight checks that -point-in-time local cache identity. The checked-in runner independently rejects -a tag, missing image, substituted config ID, platform mismatch or absent runsc -for each execution. +`WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS` explicitly carries the +existing ART processing deadline to every backend role and proof job. The +template preserves its 3,400-second default. With the unchanged ZIP inspection, +cleanup margin and scratch TTL defaults, it must be finite and strictly between +300 and 3,600 seconds: ZIP inspection must fit inside processing, and processing +plus the 300-second cleanup margin must expire before the 3,900-second scratch +reservation TTL. Canonical Settings validates the same relationships at startup. +This is an operational deadline, not a project/checker resource cap. Choose a +value that accommodates the selected workload; a shorter owned qualification +may observe real expiry sooner, without resetting its original deadline or +changing registered checker limits. + +Copy `checker-service.example.json` to the configured private mode-`0400` +path. The unit reads `/etc/workstream/checker-service.json`. Replace all example +binary, guest, archive, platform-manifest and config digests with independently +verified values. The default-behavior digest must match the built source's +canonical default checker. Runtime execution uses preloaded archives and never +pulls, resolves a tag or falls back to Docker/runsc. + +For a proxy-disabled release remove both `model_proxy_socket` and +`model_relay_root`; preflight rejects partial or substituted selectors. For a +proxy-enabled release retain their exact host paths. The running service reads +its configuration from JSON, not the release environment. + +Approved POST trial selectors additionally require the optional paired +`WORKSTREAM_CHECKER_TRIAL_AUTHORITY_MANIFEST_PATH` and +`WORKSTREAM_CHECKER_TRIAL_AUTHORITY_MANIFEST_DIGEST`. They activate +`compose.trials.yml`, which mounts that exact host file read-only only into the +model proxy. Set the native JSON's `trial_authority_manifest_path` and +`trial_authority_manifest_digest` to the same host source and digest, and supply +its existing `trial_runtime` object: private relay/scratch/child-home/cache roots, +pinned builder config ID, launcher/filesystem-tool paths and digests, deployment +owner, candidate archive ceiling and exact runtime-configuration digest. Provision +the builder cache entry and approved offline cache from the qualified runtime +release. Do not substitute a project-selected profile or copy the runtime into +this deployment layer. The native runtime and backend typed manifest loader +validate the authority catalogue; preflight checks their shared byte identity +and custody. API, worker and Beat do not read this catalogue or receive its mount. + +Smol 1.31.4's longest Unix socket adds 48 filesystem bytes below its HOME. The +107-byte socket limit therefore permits a global `smol.home` of at most 59 +bytes. A trial child adds `/ws-<32 hex digits>.home` first, so +`trial_runtime.child_home_root` must be at most 18 bytes. These are filesystem +byte limits, including multibyte characters. Use a separately owned short +private directory such as `/var/tmp/t-<6 hex>` for the trial child root; retain +its runtime owner marker and exact custody. Do not shorten invocation identities +or derive this root from a long deployment prefix. Preflight rejects excess +length before a VM is created. ## Validate without disclosure -Run preflight from the trusted host administrator account. Use `sudo` when the -operator account cannot read the runtime-owned `0400` checker configuration or -access the Docker socket; the script never opens application secret files. +The directory commands above describe the default production root. For another +owned root, create its derived children with the same identities and modes, and +replace native JSON socket/material/model paths with those exact derived paths. +Keep source/cache/config/credential files outside that data root. + +Run full preflight and the Compose wrapper as the trusted root administrator or +through approved `sudo`. That operator needs authority to traverse the +`10001:10001`-owned `0700` deployment root, inspect private `0400` inputs and +access Docker. This also applies to a newly owned `/var/tmp` proof root. The +native node and health containers run as `10001:10001`; do not grant that runtime +identity Docker group membership or daemon access to make operator checks work. +Preflight never opens application secret files. The configuration-only check renders Compose and validates fixed topology, digest identities, Flow/AWS profile and the hosted checker selectors plus @@ -303,20 +410,21 @@ including the canonical AWS evidence fields described above. This check does not service: ```sh -python3 deploy/hosted/preflight.py \ +sudo python3 deploy/hosted/preflight.py \ --env-file /etc/workstream/release.env \ --configuration-only ``` The full preflight additionally checks secret/config metadata without reading -secret contents, validates the private directories and Docker-socket group, and -requires locally registered runsc plus every exact cached image at that point in -time. It also checks the private intake socket and calls the production typed -intake health adapter from the pinned backend image with no secrets or network. +secret contents, validates private directories and the one node socket owner/mode, and +hashes the installed native service, Smol, guest and cached archive/config assets. +It checks delegated cpu/memory/pids controls and calls the existing typed project +execution and trusted intake health adapters from the pinned backend image with +no credentials or network. Configuration-only mode performs none of those live checks. Wrong runtime, image/config or default behavior rejects launch: ```sh -python3 deploy/hosted/preflight.py --env-file /etc/workstream/release.env +sudo python3 deploy/hosted/preflight.py --env-file /etc/workstream/release.env ``` Preflight cannot attest the active AWS identity, absence of extra IAM @@ -333,35 +441,52 @@ Use the checked-in wrapper for every Compose operation. It parses and validates the approved environment file, renders the fixed topology, and gives Docker Compose only that parsed allowlist plus fixed process path, operator home and local Docker-socket values. Ambient shell values cannot replace an image, Flow -URL, secret path, Compose selector or Docker daemon selector. Every `up` also -reloads the current checker configuration and repeats the checker-selector, -private-host-input, Docker-runtime/cache and typed intake readiness checks before starting anything. -Skipping the separate preflight does not skip these launch checks. `pull`, `ps` -and `stop` remain available without launch-only checks so an invalid host can -still be inspected or stopped. Run `up` from the same trusted administrator -account used for full preflight. Keep the private configuration and host inputs -unchanged between validation and launch and while the service runs; these -point-in-time checks do not constrain a trusted host administrator: +URL, secret path, Compose selector or Docker daemon selector. Normal `up` +reloads the node configuration and requires static custody plus both typed native +health checks before releasing API/worker services. `pull`, `ps` and fixed `stop` +remain usable while a node is unavailable. + +Cold startup has explicit prerequisite stages to avoid a schema/activation/proxy/ +node dependency cycle. `migrate` runs only the existing API image's Alembic command +with no dependencies. The fixed AWS `proof` sequence and exactly +`up -d model-proxy` validate static topology and private inputs but require no live +node and release no API, worker or scheduler. The proof command retains its exact +credential-separated AWS observations and coordinator; it does not use intake or +project execution. The API image's release entrypoint loads its API role secrets +and executes the supplied Alembic command without application startup or service +health checks. Alembic loads settings and model metadata, then connects directly +to PostgreSQL; activation and proxy/node readiness follow migration. Run these +reviewed operator steps: ```sh -python3 deploy/hosted/compose.py --env-file /etc/workstream/release.env pull -python3 deploy/hosted/compose.py --env-file /etc/workstream/release.env \ +sudo python3 deploy/hosted/compose.py --env-file /etc/workstream/release.env pull +sudo python3 deploy/hosted/compose.py --env-file /etc/workstream/release.env migrate +sudo python3 deploy/hosted/compose.py --env-file /etc/workstream/release.env \ + proof --group-nonce +sudo python3 deploy/hosted/compose.py --env-file /etc/workstream/release.env \ + up -d model-proxy +# Wait for the proxy health check, then start the reviewed native unit explicitly. +sudo systemctl start workstream-checker.service +sudo python3 deploy/hosted/preflight.py --env-file /etc/workstream/release.env +sudo python3 deploy/hosted/compose.py --env-file /etc/workstream/release.env \ up --detach --wait -python3 deploy/hosted/compose.py --env-file /etc/workstream/release.env ps +sudo python3 deploy/hosted/compose.py --env-file /etc/workstream/release.env ps ``` -The API applies forward migrations before serving. The guide parser and an -enabled trusted model proxy must pass their protocol/socket health checks before -the worker starts. Worker startup uses the existing prefork lifecycle. Beat runs the -existing recovery and cleanup -schedules. The checker health check requires its private socket and Docker's -registered runsc runtime. It does not attest cache identity; full preflight -observes the cache once, and the canonical runner verifies the exact image again -for each execution. API health checks a lightweight endpoint; the worker and -Beat health checks verify their local Celery processes. These checks do not -replace a public authenticated journey or dependency-specific provider probes. -Each process stops after five consecutive failed restarts so an invalid release -cannot remain in an unbounded crash loop. +A proxy-disabled release omits its proxy stage. Schema migration and any required +fresh AWS activation still precede API startup. Keep reviewed files unchanged +between validation and launch. These checks do not constrain a trusted host +administrator and do not attest the running node's complete source custody; +qualification separately records its actual process/binary identities. + +The guide parser and enabled model proxy must pass their existing health checks +before the worker starts. Beat retains existing recovery schedules. API health +is a lightweight endpoint; Celery health checks its processes. Typed native +readiness verifies runtime/identity/cache-count and pinned intake/default facts; +exact image selection is independently checked during every execution. None of +these checks replaces a public authenticated journey, provider proof, resource +qualification or restart/cleanup evidence. Each Compose service has its existing +finite restart policy; the native unit retains its own bounded systemd policy. The API binds to host loopback for an environment-owned TLS reverse proxy. Do not expose the checker socket, Docker socket, PostgreSQL, Redis or OTLP collector @@ -382,7 +507,8 @@ outage does not change product results. Configuration alone is not that proof. The wrapper's `stop` operation stops Beat and API first so they cannot produce new work, gives the prefork worker its configured 7,320-second graceful window, -then drains the model proxy before stopping the parser and checker controller. +then drains the model proxy before stopping the parser. It never stops the +native systemd node. Keeping all socket services alive while the worker drains lets cancellation and terminal persistence finish against their original dependencies. Guide inference has a 7,260-second process backstop; @@ -392,12 +518,43 @@ execution bound. Celery task hard limits and durable database/outbox custody remain owned by the application. ```sh -python3 deploy/hosted/compose.py --env-file /etc/workstream/release.env stop +sudo python3 deploy/hosted/compose.py --env-file /etc/workstream/release.env stop +``` + +For an intentional worker-only interruption, the existing wrapper also accepts +exactly `stop worker` followed by `start worker`. Stop preserves API, Beat, proxy, +parser and native node while allowing the configured 7,320-second worker drain; +its CLI deadline adds 60 seconds for management. This is not a full deployment +drain: producers can continue creating durable work. Start repeats the full +private-input and typed native readiness checks, then reconciles only the worker +with a 60-second health wait and a 120-second CLI deadline. It does not start +dependencies. A failed or timed-out command requires state inspection; it never +automatically force-kills, retries or stops another service. + +```sh +sudo python3 deploy/hosted/compose.py --env-file /etc/workstream/release.env stop worker +sudo python3 deploy/hosted/compose.py --env-file /etc/workstream/release.env start worker ``` For a restart or upgrade, preserve scratch until cleanup/recovery has classified all retained work, update only reviewed digest identities/configuration, rerun -full preflight, pull, and start with `up --detach --wait`. Exercise an interrupted +the cold/warm dependency sequence above, and start with `up --detach --wait`. +The scratch ownership marker binds the configured limits, including the total +preparation deadline. Changing those limits requires a drained scratch pool; +do not edit its marker to make a new release start. Stop all processes mounting +that pool, verify that no reservations or workspaces remain, and retain the old +directory intact before provisioning a fresh private directory at the canonical +path. Let the normal ART manager initialize it, recreate the affected container +and systemd mounts, and verify that API, worker, proxy and native node all see +the same new directory. +If retained work remains, complete its recovery under the original configuration +before changing the pool. + +Before restarting or stopping the native unit, first drain producers and the +worker through the wrapper, retain the proxy until node calls settle, and +independently confirm that this node has no active owned leases, VMs or cgroups. +Preserve guardian recovery and use the exact owner-scoped inventory; never prune +shared resources or stop another deployment. Exercise an interrupted job and verify the existing exact retry/reconciliation operations rather than editing rows or replaying unknown work blindly. @@ -453,7 +610,7 @@ change. Before #583 or parent #555 can claim a hosted release, retain evidence for real Flow human/service identities, AWS custody and negative identity cases, selected -registry/LIVE controls, actual project checker images, gVisor isolation, +registry/LIVE controls, actual project checker images, Smol isolation, representative CPU/memory/PID/disk/time measurements, missing-cache and limit classification, collector outage, drain/restart, supported recovery and isolated backup restore. Bind that evidence to exact source, service-image, checker-image, diff --git a/deploy/hosted/checker-service.example.json b/deploy/hosted/checker-service.example.json index 718061488..090e18b93 100644 --- a/deploy/hosted/checker-service.example.json +++ b/deploy/hosted/checker-service.example.json @@ -3,9 +3,8 @@ "material_root": "/var/lib/workstream/scratch", "model_proxy_socket": "/var/lib/workstream/model-runtime/model.sock", "model_relay_root": "/var/lib/workstream/model-relays", - "docker_binary": "/usr/local/bin/docker", - "runtime": "runsc", - "isolation_mode": "gvisor", + "runtime": "smolvm", + "isolation_mode": "smol-microvm", "operating_system": "linux", "architecture": "amd64", "sandbox_uid": 10001, @@ -18,7 +17,40 @@ "platform_manifest_byte_count": 1, "operating_system": "linux", "architecture": "amd64", - "config_image_id": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd" + "config_image_id": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", + "smol": { + "archive": "/var/lib/workstream-checker/images/project-checker.tar", + "archive_sha256": "sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee", + "image_config": "/var/lib/workstream-checker/images/project-checker.config.json" + } + }, + { + "repository": "registry.example/workstream/trusted-intake", + "platform_manifest_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "platform_manifest_media_type": "application/vnd.oci.image.manifest.v1+json", + "platform_manifest_byte_count": 1, + "operating_system": "linux", + "architecture": "amd64", + "config_image_id": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "smol": { + "archive": "/var/lib/workstream-checker/images/trusted-intake.tar", + "archive_sha256": "sha256:3333333333333333333333333333333333333333333333333333333333333333", + "image_config": "/var/lib/workstream-checker/images/trusted-intake.config.json" + } } - ] + ], + "smol": { + "binary": "/opt/workstream/smol/smol", + "binary_sha256": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "home": "/var/lib/workstream-checker/smol", + "guest_binary": "/opt/workstream/bin/workstream-checker-guest", + "guest_sha256": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "cgroup_root": "/sys/fs/cgroup/system.slice/workstream-checker.service", + "maximum_parallel": 2, + "maximum_storage_bytes": 8589934592 + }, + "trusted_intake": { + "platform_manifest_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "default_behavior_sha256": "sha256:af2c604f792c4456d3b222036c772161b1503eace65ff90c3d3a23d9d348c28f" + } } diff --git a/deploy/hosted/compose.inference.yml b/deploy/hosted/compose.inference.yml index 01c7ac5a9..8321a8485 100644 --- a/deploy/hosted/compose.inference.yml +++ b/deploy/hosted/compose.inference.yml @@ -1,24 +1,11 @@ services: - checker-service: - volumes: - - type: bind - source: /var/lib/workstream/model-runtime - target: /var/lib/workstream/model-runtime - read_only: true - - type: bind - source: /var/lib/workstream/model-relays - target: /var/lib/workstream/model-relays - depends_on: - model-proxy: - condition: service_healthy - worker: environment: - WORKSTREAM_CHECKER_MODEL_PROXY_SOCKET: /var/lib/workstream/model-runtime/model.sock + WORKSTREAM_CHECKER_MODEL_PROXY_SOCKET: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/model-runtime/model.sock volumes: - type: bind - source: /var/lib/workstream/model-runtime - target: /var/lib/workstream/model-runtime + source: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/model-runtime + target: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/model-runtime read_only: true depends_on: model-proxy: diff --git a/deploy/hosted/compose.py b/deploy/hosted/compose.py index 6c8595fb9..e3b854616 100644 --- a/deploy/hosted/compose.py +++ b/deploy/hosted/compose.py @@ -17,7 +17,7 @@ run_shutdown, validate_checker_config, validate_compose, - validate_docker_runtime, + validate_smol_runtime, validate_trusted_intake, validate_environment, validate_private_host_inputs, @@ -38,15 +38,32 @@ def main() -> int: values = parse_environment(args.env_file) validate_environment(values) validate_compose(render_compose(args.env_file, values), values) - launch = args.compose_arguments and args.compose_arguments[0] in {"up", "proof"} - if launch: + worker_start = args.compose_arguments == ["start", "worker"] + launch = worker_start or ( + args.compose_arguments and args.compose_arguments[0] in {"up", "proof"} + ) + proxy_bootstrap = args.compose_arguments in ( + ["up", "-d", "model-proxy"], + ["up", "--detach", "model-proxy"], + ) + migration = args.compose_arguments == ["migrate"] + proof_bootstrap = bool( + args.compose_arguments and args.compose_arguments[0] == "proof" + ) + if proxy_bootstrap and values["WORKSTREAM_MODEL_PROXY_ENABLED"] != "true": + raise PreflightError("model proxy bootstrap requires an enabled proxy") + if launch or migration: checker_config = load_checker_config( Path(values["WORKSTREAM_CHECKER_CONFIG_FILE"]) ) validate_checker_config(checker_config, values) - validate_private_host_inputs(values) - validate_docker_runtime(checker_config, values) - validate_trusted_intake(values) + if proxy_bootstrap or migration or proof_bootstrap: + # No API, worker or producer is released in these fixed stages. + validate_private_host_inputs(values, require_node=False) + else: + validate_private_host_inputs(values) + validate_smol_runtime(checker_config, values) + validate_trusted_intake(values) if args.compose_arguments and args.compose_arguments[0] == "proof": if ( len(args.compose_arguments) != 3 @@ -59,9 +76,11 @@ def main() -> int: raise PreflightError("proof group nonce is invalid") from exc return run_proof_group(args.env_file, values, group_nonce) if args.compose_arguments and args.compose_arguments[0] == "stop": + if args.compose_arguments == ["stop", "worker"]: + return run_compose(args.env_file, values, args.compose_arguments) if args.compose_arguments != ["stop"]: raise PreflightError( - "stop uses the fixed producer, worker, parser/checker sequence" + "stop requires the fixed shutdown sequence or exactly one worker" ) return run_shutdown(args.env_file, values) return run_compose(args.env_file, values, args.compose_arguments) diff --git a/deploy/hosted/compose.trials.yml b/deploy/hosted/compose.trials.yml new file mode 100644 index 000000000..28584f899 --- /dev/null +++ b/deploy/hosted/compose.trials.yml @@ -0,0 +1,11 @@ +# Consumes the trial authority/handler supplied by the Smol runtime dependency. +services: + model-proxy: + environment: + WORKSTREAM_CHECKER_TRIAL_AUTHORITY_MANIFEST_PATH: /etc/workstream/trial-authority.json + WORKSTREAM_CHECKER_TRIAL_AUTHORITY_MANIFEST_DIGEST: ${WORKSTREAM_CHECKER_TRIAL_AUTHORITY_MANIFEST_DIGEST:?set the pinned trial authority digest} + volumes: + - type: bind + source: ${WORKSTREAM_CHECKER_TRIAL_AUTHORITY_MANIFEST_PATH:?set the private trial authority manifest path} + target: /etc/workstream/trial-authority.json + read_only: true diff --git a/deploy/hosted/compose.yml b/deploy/hosted/compose.yml index a925ccd98..238973a7f 100644 --- a/deploy/hosted/compose.yml +++ b/deploy/hosted/compose.yml @@ -35,8 +35,9 @@ x-backend-environment: &backend-environment WORKSTREAM_ARTIFACT_S3_RUNTIME_ROLE_ARN: ${WORKSTREAM_ARTIFACT_S3_RUNTIME_ROLE_ARN:?set the exact runtime role ARN} WORKSTREAM_ARTIFACT_S3_RUNTIME_TRUST_POLICY_SHA256: ${WORKSTREAM_ARTIFACT_S3_RUNTIME_TRUST_POLICY_SHA256:?set the exact runtime trust policy digest} WORKSTREAM_ARTIFACT_S3_WRITE_TIMEOUT_SECONDS: ${WORKSTREAM_ARTIFACT_S3_WRITE_TIMEOUT_SECONDS:?set the measured S3 write timeout} - WORKSTREAM_ARTIFACT_SCRATCH_ROOT: /var/lib/workstream/scratch - WORKSTREAM_ARTIFACT_INTAKE_SERVICE_SOCKET: /var/lib/workstream/intake-runtime/checker.sock + WORKSTREAM_ARTIFACT_SCRATCH_ROOT: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/scratch + WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS: ${WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS:?set the artifact preparation deadline} + WORKSTREAM_ARTIFACT_INTAKE_SERVICE_SOCKET: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/checker-runtime/checker.sock WORKSTREAM_ARTIFACT_INTAKE_IMAGE_DIGEST: ${WORKSTREAM_ARTIFACT_INTAKE_IMAGE_DIGEST:?set the pinned trusted intake image digest} WORKSTREAM_ARTIFACT_INTAKE_CONFIG_DIGEST: ${WORKSTREAM_ARTIFACT_INTAKE_CONFIG_DIGEST:?set the pinned trusted intake config digest} WORKSTREAM_ARTIFACT_STORE_BACKEND: s3_compatible @@ -45,8 +46,8 @@ x-backend-environment: &backend-environment WORKSTREAM_AUTH_PROVIDER: flow WORKSTREAM_DEBUG: "false" WORKSTREAM_ENVIRONMENT: production - WORKSTREAM_EXTERNAL_CHECKER_MATERIAL_ROOT: /var/lib/workstream/scratch - WORKSTREAM_EXTERNAL_CHECKER_SERVICE_SOCKET: /var/lib/workstream/checker-runtime/checker.sock + WORKSTREAM_EXTERNAL_CHECKER_MATERIAL_ROOT: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/scratch + WORKSTREAM_EXTERNAL_CHECKER_SERVICE_SOCKET: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/checker-runtime/checker.sock WORKSTREAM_OBSERVABILITY_LOG_LEVEL: ${WORKSTREAM_OBSERVABILITY_LOG_LEVEL:-INFO} WORKSTREAM_OBSERVABILITY_OTLP_ENDPOINT: ${WORKSTREAM_OBSERVABILITY_OTLP_ENDPOINT:-} WORKSTREAM_OBSERVABILITY_TRACE_SAMPLE_RATIO: ${WORKSTREAM_OBSERVABILITY_TRACE_SAMPLE_RATIO:-0.1} @@ -110,13 +111,13 @@ services: user: "0:0" environment: WORKSTREAM_RELEASE_ROLE: parser - WORKSTREAM_GUIDE_DOCUMENT_PARSER_SOCKET: /var/lib/workstream/guide-runtime/guide.sock + WORKSTREAM_GUIDE_DOCUMENT_PARSER_SOCKET: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/guide-runtime/guide.sock command: - python - -m - app.adapters.guide_document_tools.service - --socket - - /var/lib/workstream/guide-runtime/guide.sock + - ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/guide-runtime/guide.sock network_mode: none read_only: true cap_drop: [ALL] @@ -132,51 +133,15 @@ services: - /tmp:mode=0700,uid=10002,gid=10001,size=268435456,noexec,nosuid,nodev volumes: - type: bind - source: /var/lib/workstream/guide-runtime - target: /var/lib/workstream/guide-runtime + source: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/guide-runtime + target: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/guide-runtime healthcheck: - test: [CMD, python, -m, app.adapters.guide_document_tools.service, --healthcheck, --socket, /var/lib/workstream/guide-runtime/guide.sock] + test: [CMD, python, -m, app.adapters.guide_document_tools.service, --healthcheck, --socket, "${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/guide-runtime/guide.sock"] interval: 10s timeout: 5s retries: 6 start_period: 10s - checker-service: - image: ${WORKSTREAM_CHECKER_SERVICE_IMAGE:?set a digest-qualified checker-service image} - pull_policy: always - init: true - user: "${WORKSTREAM_RUNTIME_UID}:${WORKSTREAM_RUNTIME_GID}" - group_add: ["${WORKSTREAM_DOCKER_SOCKET_GID:?set the Docker socket group ID}"] - command: [/etc/workstream/checker-service.json] - network_mode: none - read_only: true - cap_drop: [ALL] - security_opt: [no-new-privileges:true] - restart: on-failure:5 - tmpfs: - - /tmp:mode=0700,uid=${WORKSTREAM_RUNTIME_UID},gid=${WORKSTREAM_RUNTIME_GID} - volumes: - - type: bind - source: ${WORKSTREAM_CHECKER_CONFIG_FILE:?set the private checker config path} - target: /etc/workstream/checker-service.json - read_only: true - - type: bind - source: /var/lib/workstream/scratch - target: /var/lib/workstream/scratch - read_only: true - - type: bind - source: /var/lib/workstream/checker-runtime - target: /var/lib/workstream/checker-runtime - - type: bind - source: /var/run/docker.sock - target: /var/run/docker.sock - healthcheck: - test: [CMD-SHELL, "test -S /var/lib/workstream/checker-runtime/checker.sock && docker info --format '{{json .Runtimes.runsc}}' | grep -vq '^null$'"] - interval: 15s - timeout: 10s - retries: 4 - start_period: 10s - api: <<: *backend-runtime environment: @@ -215,15 +180,11 @@ services: - "127.0.0.1:${WORKSTREAM_API_HOST_PORT:-8000}:8000" volumes: - type: bind - source: /var/lib/workstream/scratch - target: /var/lib/workstream/scratch + source: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/scratch + target: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/scratch - type: bind - source: /var/lib/workstream/checker-runtime - target: /var/lib/workstream/checker-runtime - read_only: true - - type: bind - source: /var/lib/workstream/intake-runtime - target: /var/lib/workstream/intake-runtime + source: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/checker-runtime + target: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/checker-runtime read_only: true - type: bind source: ${WORKSTREAM_ARTIFACT_S3_ROLES_ANYWHERE_RUNTIME_CERTIFICATE_HOST_FILE:?set the private runtime certificate path} @@ -233,9 +194,6 @@ services: source: ${WORKSTREAM_ARTIFACT_S3_ROLES_ANYWHERE_RUNTIME_PRIVATE_KEY_HOST_FILE:?set the private runtime key path} target: /run/workstream-roles-anywhere/private-key.pem read_only: true - depends_on: - checker-service: - condition: service_healthy restart: on-failure:5 stop_grace_period: 30s healthcheck: @@ -252,7 +210,7 @@ services: WORKSTREAM_RELEASE_ROLE: worker WORKSTREAM_DATABASE_URL_FILE: /run/secrets/database_url WORKSTREAM_CELERY_BROKER_URL_FILE: /run/secrets/celery_broker_url - WORKSTREAM_GUIDE_DOCUMENT_PARSER_SOCKET: /var/lib/workstream/guide-runtime/guide.sock + WORKSTREAM_GUIDE_DOCUMENT_PARSER_SOCKET: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/guide-runtime/guide.sock secrets: - source: database_url target: database_url @@ -278,19 +236,15 @@ services: - --loglevel=INFO volumes: - type: bind - source: /var/lib/workstream/scratch - target: /var/lib/workstream/scratch + source: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/scratch + target: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/scratch - type: bind - source: /var/lib/workstream/checker-runtime - target: /var/lib/workstream/checker-runtime + source: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/checker-runtime + target: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/checker-runtime read_only: true - type: bind - source: /var/lib/workstream/intake-runtime - target: /var/lib/workstream/intake-runtime - read_only: true - - type: bind - source: /var/lib/workstream/guide-runtime - target: /var/lib/workstream/guide-runtime + source: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/guide-runtime + target: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/guide-runtime read_only: true - type: bind source: ${WORKSTREAM_ARTIFACT_S3_ROLES_ANYWHERE_RUNTIME_CERTIFICATE_HOST_FILE:?set the private runtime certificate path} @@ -303,8 +257,6 @@ services: depends_on: api: condition: service_healthy - checker-service: - condition: service_healthy guide-document-parser: condition: service_healthy restart: on-failure:5 @@ -324,7 +276,7 @@ services: <<: *backend-environment WORKSTREAM_RELEASE_ROLE: model-proxy WORKSTREAM_DATABASE_URL_FILE: /run/secrets/database_url - WORKSTREAM_CHECKER_MODEL_PROXY_SOCKET: /var/lib/workstream/model-runtime/model.sock + WORKSTREAM_CHECKER_MODEL_PROXY_SOCKET: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/model-runtime/model.sock secrets: - source: database_url target: database_url @@ -334,11 +286,11 @@ services: command: [python, -m, app.workers.checker_model_proxy] volumes: - type: bind - source: /var/lib/workstream/scratch - target: /var/lib/workstream/scratch + source: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/scratch + target: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/scratch - type: bind - source: /var/lib/workstream/model-runtime - target: /var/lib/workstream/model-runtime + source: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/model-runtime + target: ${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/model-runtime - type: bind source: ${WORKSTREAM_ARTIFACT_S3_ROLES_ANYWHERE_RUNTIME_CERTIFICATE_HOST_FILE:?set the private runtime certificate path} target: /run/workstream-roles-anywhere/certificate.pem @@ -350,7 +302,7 @@ services: restart: on-failure:5 stop_grace_period: 7320s healthcheck: - test: [CMD-SHELL, "test -S /var/lib/workstream/model-runtime/model.sock"] + test: [CMD, python, -c, "import pathlib,sys; sys.exit(0 if pathlib.Path(sys.argv[1]).is_socket() else 1)", "${WORKSTREAM_DEPLOYMENT_ROOT:-/var/lib/workstream}/model-runtime/model.sock"] interval: 10s timeout: 5s retries: 6 diff --git a/deploy/hosted/preflight.py b/deploy/hosted/preflight.py index 8a713a6fd..3fdc13ff9 100644 --- a/deploy/hosted/preflight.py +++ b/deploy/hosted/preflight.py @@ -4,6 +4,8 @@ from __future__ import annotations import argparse +from dataclasses import dataclass +import hashlib from decimal import Decimal, InvalidOperation import json import os @@ -20,19 +22,73 @@ COMPOSE_PATH = Path(__file__).with_name("compose.yml") INFERENCE_COMPOSE_PATH = Path(__file__).with_name("compose.inference.yml") +TRIAL_COMPOSE_PATH = Path(__file__).with_name("compose.trials.yml") +TRIAL_MANIFEST_TARGET = Path("/etc/workstream/trial-authority.json") +TRIAL_KEYS = ( + "WORKSTREAM_CHECKER_TRIAL_AUTHORITY_MANIFEST_PATH", + "WORKSTREAM_CHECKER_TRIAL_AUTHORITY_MANIFEST_DIGEST", +) MODEL_OPENAI_COMPOSE_PATH = Path(__file__).with_name("compose.model-openai.yml") MODEL_OPENROUTER_COMPOSE_PATH = Path(__file__).with_name("compose.model-openrouter.yml") -SCRATCH_ROOT = Path("/var/lib/workstream/scratch") -SOCKET_ROOT = Path("/var/lib/workstream/checker-runtime") -SOCKET_PATH = SOCKET_ROOT / "checker.sock" -INTAKE_SOCKET_ROOT = Path("/var/lib/workstream/intake-runtime") -INTAKE_SOCKET_PATH = INTAKE_SOCKET_ROOT / "checker.sock" -INTAKE_DIGEST_KEYS = ("WORKSTREAM_ARTIFACT_INTAKE_IMAGE_DIGEST", "WORKSTREAM_ARTIFACT_INTAKE_CONFIG_DIGEST") -GUIDE_SOCKET_ROOT = Path("/var/lib/workstream/guide-runtime") -GUIDE_SOCKET_PATH = GUIDE_SOCKET_ROOT / "guide.sock" -MODEL_SOCKET_ROOT = Path("/var/lib/workstream/model-runtime") -MODEL_SOCKET_PATH = MODEL_SOCKET_ROOT / "model.sock" -MODEL_RELAY_ROOT = Path("/var/lib/workstream/model-relays") +DEFAULT_DEPLOYMENT_ROOT = Path("/var/lib/workstream") +DEFAULT_CHECKER_SERVICE_BINARY = Path("/opt/workstream/bin/workstream-checker-service") +INTAKE_DIGEST_KEYS = ( + "WORKSTREAM_ARTIFACT_INTAKE_IMAGE_DIGEST", + "WORKSTREAM_ARTIFACT_INTAKE_CONFIG_DIGEST", +) + + +@dataclass(frozen=True) +class HostedPaths: + root: Path + + @property + def scratch(self) -> Path: + return self.root / "scratch" + + @property + def socket_root(self) -> Path: + return self.root / "checker-runtime" + + @property + def model_root(self) -> Path: + return self.root / "model-runtime" + + @property + def model_relays(self) -> Path: + return self.root / "model-relays" + + @property + def guide_root(self) -> Path: + return self.root / "guide-runtime" + + @property + def socket(self) -> Path: + return self.socket_root / "checker.sock" + + @property + def model_socket(self) -> Path: + return self.model_root / "model.sock" + + @property + def guide_socket(self) -> Path: + return self.guide_root / "guide.sock" + + +def deployment_paths(values: dict[str, str]) -> HostedPaths: + return HostedPaths( + Path(values.get("WORKSTREAM_DEPLOYMENT_ROOT", str(DEFAULT_DEPLOYMENT_ROOT))) + ) + + +def checker_service_binary(values: dict[str, str]) -> Path: + return Path( + values.get( + "WORKSTREAM_CHECKER_SERVICE_BINARY", str(DEFAULT_CHECKER_SERVICE_BINARY) + ) + ) + + GUIDE_PARSER_UID = 10002 GUIDE_PARSER_GID = 10001 DOCKER_SOCKET = Path("/var/run/docker.sock") @@ -171,6 +227,7 @@ "circuit_cooldown_seconds": (1, 600), } TIMEOUT_KEYS = ( + "WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS", "WORKSTREAM_ARTIFACT_S3_CONNECT_TIMEOUT_SECONDS", "WORKSTREAM_ARTIFACT_S3_READ_TIMEOUT_SECONDS", "WORKSTREAM_ARTIFACT_S3_WRITE_TIMEOUT_SECONDS", @@ -218,11 +275,10 @@ "COMPOSE_PROJECT_NAME", "WORKSTREAM_BACKEND_IMAGE", "WORKSTREAM_MODEL_PROXY_IMAGE", - "WORKSTREAM_CHECKER_SERVICE_IMAGE", + "WORKSTREAM_CHECKER_SERVICE_BINARY_SHA256", "WORKSTREAM_MODEL_PROXY_ENABLED", "WORKSTREAM_RUNTIME_UID", "WORKSTREAM_RUNTIME_GID", - "WORKSTREAM_DOCKER_SOCKET_GID", "WORKSTREAM_TOKEN_ISSUER", "WORKSTREAM_TOKEN_AUDIENCE", "WORKSTREAM_TOKEN_JWKS_URL", @@ -244,6 +300,9 @@ *BASE_SECRET_FILE_KEYS, ) OPTIONAL_RELEASE_KEYS = ( + "WORKSTREAM_DEPLOYMENT_ROOT", + "WORKSTREAM_CHECKER_SERVICE_BINARY", + *TRIAL_KEYS, "WORKSTREAM_API_HOST_PORT", "WORKSTREAM_CELERY_WORKER_CONCURRENCY", "WORKSTREAM_OBSERVABILITY_LOG_LEVEL", @@ -318,22 +377,36 @@ def validate_environment(values: dict[str, str]) -> None: raise PreflightError( f"release environment contains an unsupported key: {unknown[0]}" ) + root = deployment_paths(values).root + binary = checker_service_binary(values) + if values.get("WORKSTREAM_DEPLOYMENT_ROOT", str(root)) != str(root) or values.get( + "WORKSTREAM_CHECKER_SERVICE_BINARY", str(binary) + ) != str(binary): + raise PreflightError("deployment paths must use canonical spelling") + for path in (root, binary): + if ( + not path.is_absolute() + or path != path.resolve(strict=False) + or any(character in str(path) for character in (",", "\n", "\r", "\0")) + or len(path.parts) < 3 + ): + raise PreflightError("deployment paths must be canonical absolute paths") + if root == binary or binary.is_relative_to(root) or root.is_relative_to(binary): + raise PreflightError("native service asset overlaps deployment data") for key in REQUIRED_RELEASE_KEYS: _required(values, key) for key in ( "WORKSTREAM_BACKEND_IMAGE", "WORKSTREAM_MODEL_PROXY_IMAGE", - "WORKSTREAM_CHECKER_SERVICE_IMAGE", ): if not IMAGE_REFERENCE.fullmatch(_required(values, key)): raise PreflightError(f"release image must use repository@sha256: {key}") for key in ( "WORKSTREAM_RUNTIME_UID", "WORKSTREAM_RUNTIME_GID", - "WORKSTREAM_DOCKER_SOCKET_GID", ): _positive_id(values, key) - for key in INTAKE_DIGEST_KEYS: + for key in (*INTAKE_DIGEST_KEYS, "WORKSTREAM_CHECKER_SERVICE_BINARY_SHA256"): if not DIGEST.fullmatch(values[key]): raise PreflightError(f"trusted intake requires a pinned digest: {key}") for key in FORBIDDEN_RELEASE_KEYS: @@ -346,6 +419,15 @@ def validate_environment(values: dict[str, str]) -> None: raise PreflightError(f"unsupported release setting: {key}") for key in TIMEOUT_KEYS: _positive_decimal(values, key) + preparation_deadline = _positive_decimal( + values, "WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS" + ) + # The canonical Settings owner keeps ZIP inspection at 300s, cleanup at + # 300s and scratch TTL at 3900s. No other scratch timing is configurable here. + if not 300 < float(preparation_deadline) < 3600: + raise PreflightError( + "artifact preparation deadline must be greater than 300 and less than 3600 seconds" + ) for key in PROJECT_AGENT_LIMIT_KEYS: _positive_id(values, key) if ( @@ -398,6 +480,18 @@ def validate_environment(values: dict[str, str]) -> None: ) elif any(values.get(key) for key in MODEL_PROXY_SECRET_FILE_KEYS): raise PreflightError("disabled model proxy cannot select provider credentials") + trial_path, trial_digest = (values.get(key, "") for key in TRIAL_KEYS) + if bool(trial_path) != bool(trial_digest) or ( + trial_path + and ( + not model_proxy_enabled + or not Path(trial_path).is_absolute() + or not DIGEST.fullmatch(trial_digest) + ) + ): + raise PreflightError( + "trial authority requires the enabled proxy and one pinned manifest pair" + ) try: runtime_options = json.loads(values["WORKSTREAM_PROJECT_AGENT_RUNTIME_OPTIONS"]) except json.JSONDecodeError as exc: @@ -465,27 +559,86 @@ def load_checker_config(path: Path) -> dict[str, Any]: def validate_checker_config(config: dict[str, Any], values: dict[str, str]) -> None: """Bind release selectors and cache identities to the existing runner.""" + paths = deployment_paths(values) expected = { - "socket_path": str(SOCKET_PATH), - "material_root": str(SCRATCH_ROOT), - "docker_binary": "/usr/local/bin/docker", - "runtime": "runsc", - "isolation_mode": "gvisor", + "socket_path": str(paths.socket), + "material_root": str(paths.scratch), + "runtime": "smolvm", + "isolation_mode": "smol-microvm", "operating_system": "linux", "sandbox_uid": _positive_id(values, "WORKSTREAM_RUNTIME_UID"), "sandbox_gid": _positive_id(values, "WORKSTREAM_RUNTIME_GID"), } + smol = config.get("smol") + if not isinstance(smol, dict) or set(smol) != { + "binary", + "binary_sha256", + "home", + "guest_binary", + "guest_sha256", + "cgroup_root", + "maximum_parallel", + "maximum_storage_bytes", + }: + raise PreflightError("native Smol configuration is incomplete") + for key in ("binary", "guest_binary", "home", "cgroup_root"): + if not isinstance(smol[key], str) or not Path(smol[key]).is_absolute(): + raise PreflightError("native Smol path is invalid") + if len(os.fsencode(smol["home"])) > 59: + raise PreflightError("native Smol home exceeds the 59-byte socket path budget") + for key in ("binary_sha256", "guest_sha256"): + if not isinstance(smol[key], str) or not DIGEST.fullmatch(smol[key]): + raise PreflightError("native Smol binary digest is invalid") + for key in ("maximum_parallel", "maximum_storage_bytes"): + if type(smol[key]) is not int or not 0 < smol[key] <= 2**64 - 1: + raise PreflightError("native Smol capacity must be explicitly bounded") + if Path(smol["home"]) == paths.scratch or Path(smol["home"]).is_relative_to( + paths.scratch + ): + raise PreflightError("native Smol home must be separate from ART material") + protected_inputs = [ + checker_service_binary(values), + Path(values["WORKSTREAM_CHECKER_CONFIG_FILE"]), + Path(smol["binary"]), + Path(smol["guest_binary"]), + *( + Path(values[key]) + for key in (*_secret_file_keys(values), *ROLES_ANYWHERE_HOST_FILE_KEYS) + ), + ] + protected_inputs.extend( + Path(image["smol"][key]) + for image in config.get("cache", []) + if isinstance(image, dict) and isinstance(image.get("smol"), dict) + for key in ("archive", "image_config") + if isinstance(image["smol"].get(key), str) + ) + if values.get(TRIAL_KEYS[0]): + protected_inputs.append(Path(values[TRIAL_KEYS[0]])) + if any( + path.resolve(strict=False).is_relative_to(paths.root) + for path in protected_inputs + ): + raise PreflightError( + "private config, credentials and assets must remain outside deployment data" + ) model_proxy_enabled = values["WORKSTREAM_MODEL_PROXY_ENABLED"] == "true" - expected.update({ - "model_proxy_socket": str(MODEL_SOCKET_PATH) if model_proxy_enabled else None, - "model_relay_root": str(MODEL_RELAY_ROOT) if model_proxy_enabled else None, - }) + expected.update( + { + "model_proxy_socket": str(paths.model_socket) + if model_proxy_enabled + else None, + "model_relay_root": str(paths.model_relays) + if model_proxy_enabled + else None, + } + ) for key, required in expected.items(): if config.get(key) != required: raise PreflightError( f"checker configuration has an unsafe release value: {key}" ) - if config.get("architecture") not in {"amd64", "arm64"}: + if config.get("architecture") != "amd64" or config.get("docker_binary") is not None: raise PreflightError( "checker architecture is unsupported by this release profile" ) @@ -505,6 +658,22 @@ def validate_checker_config(config: dict[str, Any], values: dict[str, str]) -> N raise PreflightError("checker cache manifest digest is invalid") if not isinstance(image_id, str) or not DIGEST.fullmatch(image_id): raise PreflightError("checker cache image ID is invalid") + image = entry.get("smol") + if not isinstance(image, dict) or set(image) != { + "archive", + "archive_sha256", + "image_config", + }: + raise PreflightError("checker cache lacks native Smol assets") + if not isinstance(image["archive_sha256"], str) or not DIGEST.fullmatch( + image["archive_sha256"] + ): + raise PreflightError("checker cache archive digest is invalid") + if any( + not isinstance(image[key], str) or not Path(image[key]).is_absolute() + for key in ("archive", "image_config") + ): + raise PreflightError("checker cache asset path is invalid") if digest in seen: raise PreflightError("checker cache manifest digest is duplicated") seen.add(digest) @@ -513,6 +682,82 @@ def validate_checker_config(config: dict[str, Any], values: dict[str, str]) -> N if entry.get("architecture") != config["architecture"]: raise PreflightError("checker cache architecture differs from the runner") + intake = config.get("trusted_intake") + if not isinstance(intake, dict) or set(intake) != { + "platform_manifest_digest", + "default_behavior_sha256", + }: + raise PreflightError("native node lacks the trusted intake purpose") + selected = [ + entry + for entry in cache + if entry["platform_manifest_digest"] == values[INTAKE_DIGEST_KEYS[0]] + ] + if ( + len(selected) != 1 + or selected[0]["config_image_id"] != values[INTAKE_DIGEST_KEYS[1]] + or intake["platform_manifest_digest"] != values[INTAKE_DIGEST_KEYS[0]] + or not isinstance(intake["default_behavior_sha256"], str) + or not DIGEST.fullmatch(intake["default_behavior_sha256"]) + ): + raise PreflightError( + "trusted intake cache identity differs from release configuration" + ) + + trial_path, trial_digest = (values.get(key) or None for key in TRIAL_KEYS) + if ( + config.get("trial_authority_manifest_path"), + config.get("trial_authority_manifest_digest"), + ) != (trial_path, trial_digest): + raise PreflightError("native trial authority differs from the model proxy") + trial = config.get("trial_runtime") + if trial_path: + path_keys = ( + "relay_root", + "scratch_root", + "child_home_root", + "offline_cache_root", + "trusted_launcher_path", + "filesystem_tool_path", + ) + digest_keys = ( + "outer_runtime_config_digest", + "builder_image_id", + "trusted_launcher_digest", + "filesystem_tool_digest", + ) + if ( + not isinstance(trial, dict) + or set(trial) + != { + *path_keys, + *digest_keys, + "deployment_owner", + "maximum_candidate_archive_bytes", + } + or any( + not isinstance(trial[key], str) or not Path(trial[key]).is_absolute() + for key in path_keys + ) + or any( + not isinstance(trial[key], str) or not DIGEST.fullmatch(trial[key]) + for key in digest_keys + ) + or not isinstance(trial["deployment_owner"], str) + or not re.fullmatch(r"[A-Za-z0-9_-]{1,128}", trial["deployment_owner"]) + or type(trial["maximum_candidate_archive_bytes"]) is not int + or not 0 < trial["maximum_candidate_archive_bytes"] <= 2**64 - 1 + or trial["builder_image_id"] + not in {entry["config_image_id"] for entry in cache} + ): + raise PreflightError("native trial runtime configuration is incomplete") + if len(os.fsencode(trial["child_home_root"])) > 18: + raise PreflightError( + "native trial child home root exceeds the 18-byte socket path budget" + ) + elif trial is not None: + raise PreflightError("native trial runtime lacks a pinned authority manifest") + def _private_metadata(path: Path, *, kind: str, uid: int, gid: int, mode: int) -> None: try: @@ -551,38 +796,50 @@ def _role_secret_file_keys(role: str, values: dict[str, str]) -> tuple[str, ...] return keys -def validate_private_host_inputs(values: dict[str, str]) -> None: +def _validate_native_socket(path: Path, uid: int, gid: int) -> None: + try: + metadata = path.lstat() + except OSError as exc: + raise PreflightError("native node socket is unavailable") from exc + if ( + not stat.S_ISSOCK(metadata.st_mode) + or metadata.st_uid != uid + or metadata.st_gid != gid + or stat.S_IMODE(metadata.st_mode) != 0o600 + or metadata.st_nlink != 1 + ): + raise PreflightError("native node socket identity is unsafe") + + +def validate_private_host_inputs( + values: dict[str, str], *, require_node: bool = True +) -> None: """Inspect metadata only; do not read application secret contents.""" + paths = deployment_paths(values) uid = _positive_id(values, "WORKSTREAM_RUNTIME_UID") gid = _positive_id(values, "WORKSTREAM_RUNTIME_GID") - _private_metadata(SCRATCH_ROOT, kind="directory", uid=uid, gid=gid, mode=0o700) - _private_metadata(SOCKET_ROOT, kind="directory", uid=uid, gid=gid, mode=0o700) - _private_metadata(INTAKE_SOCKET_ROOT, kind="directory", uid=uid, gid=gid, mode=0o700) - try: - intake_socket = INTAKE_SOCKET_PATH.lstat() - except OSError as exc: - raise PreflightError("trusted intake socket is unavailable") from exc - if (not stat.S_ISSOCK(intake_socket.st_mode) - or intake_socket.st_uid != uid or intake_socket.st_gid != gid - or stat.S_IMODE(intake_socket.st_mode) != 0o600 or intake_socket.st_nlink != 1): - raise PreflightError("trusted intake socket identity is unsafe") + _private_metadata(paths.root, kind="directory", uid=uid, gid=gid, mode=0o700) + _private_metadata(paths.scratch, kind="directory", uid=uid, gid=gid, mode=0o700) + _private_metadata(paths.socket_root, kind="directory", uid=uid, gid=gid, mode=0o700) + if require_node: + _validate_native_socket(paths.socket, uid, gid) model_proxy_enabled = values["WORKSTREAM_MODEL_PROXY_ENABLED"] == "true" if model_proxy_enabled: _private_metadata( - MODEL_SOCKET_ROOT, kind="directory", uid=uid, gid=gid, mode=0o700 + paths.model_root, kind="directory", uid=uid, gid=gid, mode=0o700 ) _private_metadata( - MODEL_RELAY_ROOT, kind="directory", uid=uid, gid=gid, mode=0o700 + paths.model_relays, kind="directory", uid=uid, gid=gid, mode=0o700 ) _private_metadata( - GUIDE_SOCKET_ROOT, + paths.guide_root, kind="directory", uid=GUIDE_PARSER_UID, gid=GUIDE_PARSER_GID, mode=0o750, ) try: - guide_socket = GUIDE_SOCKET_PATH.lstat() + guide_socket = paths.guide_socket.lstat() except FileNotFoundError: guide_socket = None except OSError as exc: @@ -598,7 +855,7 @@ def validate_private_host_inputs(values: dict[str, str]) -> None: raise PreflightError("guide parser socket has unsafe retained state") if model_proxy_enabled: try: - model_socket = MODEL_SOCKET_PATH.lstat() + model_socket = paths.model_socket.lstat() except FileNotFoundError: model_socket = None except OSError as exc: @@ -614,6 +871,7 @@ def validate_private_host_inputs(values: dict[str, str]) -> None: raise PreflightError("model proxy socket has unsafe retained state") private_paths = [ Path(values["WORKSTREAM_CHECKER_CONFIG_FILE"]), + *((Path(values[TRIAL_KEYS[0]]),) if values.get(TRIAL_KEYS[0]) else ()), *(Path(values[key]) for key in _secret_file_keys(values)), *(Path(values[key]) for key in ROLES_ANYWHERE_HOST_FILE_KEYS), ] @@ -621,14 +879,13 @@ def validate_private_host_inputs(values: dict[str, str]) -> None: _private_metadata(path, kind="file", uid=uid, gid=gid, mode=0o400) try: shared_roots = [ - SCRATCH_ROOT.resolve(strict=True), - SOCKET_ROOT.resolve(strict=True), - INTAKE_SOCKET_ROOT.resolve(strict=True), - GUIDE_SOCKET_ROOT.resolve(strict=True), + paths.scratch.resolve(strict=True), + paths.socket_root.resolve(strict=True), + paths.guide_root.resolve(strict=True), ] if model_proxy_enabled: - shared_roots.append(MODEL_SOCKET_ROOT.resolve(strict=True)) - shared_roots.append(MODEL_RELAY_ROOT.resolve(strict=True)) + shared_roots.append(paths.model_root.resolve(strict=True)) + shared_roots.append(paths.model_relays.resolve(strict=True)) resolved_paths = [path.resolve(strict=True) for path in private_paths] except (OSError, RuntimeError) as exc: raise PreflightError("private host input identity is unavailable") from exc @@ -652,8 +909,6 @@ def validate_private_host_inputs(values: dict[str, str]) -> None: raise PreflightError("Docker socket is unavailable") from exc if not stat.S_ISSOCK(docker_socket.st_mode): raise PreflightError("Docker control path is not a socket") - if docker_socket.st_gid != _positive_id(values, "WORKSTREAM_DOCKER_SOCKET_GID"): - raise PreflightError("Docker socket group differs from release configuration") def compose_environment(values: dict[str, str]) -> dict[str, str]: @@ -692,6 +947,8 @@ def _compose_file_arguments(values: dict[str, str]) -> list[str]: ("--file", str(MODEL_PROVIDER_COMPOSE_PATHS[provider])) ) arguments.extend(("--profile", "model-proxy")) + if values.get(TRIAL_KEYS[0]): + arguments.extend(["-f", str(TRIAL_COMPOSE_PATH)]) return arguments @@ -722,7 +979,35 @@ def run_compose( environment_path: Path, values: dict[str, str], compose_arguments: list[str] ) -> int: """Run one documented Compose operation with no ambient interpolation.""" - if not compose_arguments or compose_arguments[0] not in { + command_timeout = None + if compose_arguments == ["migrate"]: + compose_arguments = [ + "run", + "--rm", + "--no-deps", + "api", + "python", + "-m", + "alembic", + "upgrade", + "head", + ] + elif compose_arguments == ["stop", "worker"]: + # The fixed grace period matches validate_compose's worker contract. + compose_arguments = ["stop", "--timeout", "7320", "worker"] + command_timeout = 7380 + elif compose_arguments == ["start", "worker"]: + compose_arguments = [ + "up", + "--detach", + "--no-deps", + "--wait", + "--wait-timeout", + "60", + "worker", + ] + command_timeout = 120 + elif not compose_arguments or compose_arguments[0] not in { "pull", "up", "ps", @@ -740,7 +1025,12 @@ def run_compose( ], check=False, env=compose_environment(values), + timeout=command_timeout, ) + except subprocess.TimeoutExpired as exc: + raise PreflightError( + "hosted worker operation exceeded its management deadline" + ) from exc except OSError as exc: raise PreflightError("hosted Compose operation could not start") from exc return completed.returncode @@ -754,7 +1044,7 @@ def run_shutdown(environment_path: Path, values: dict[str, str]) -> int: ] if values["WORKSTREAM_MODEL_PROXY_ENABLED"] == "true": phases.append(["stop", "--timeout", "7320", "model-proxy"]) - phases.append(["stop", "guide-document-parser", "checker-service"]) + phases.append(["stop", "guide-document-parser"]) for arguments in phases: try: completed = subprocess.run( @@ -892,13 +1182,13 @@ def _validate_exact_bind_mounts( def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: """Check the rendered release process and privilege topology.""" + paths = deployment_paths(values) services = rendered.get("services") expected_services = { "api", "worker", "beat", "guide-document-parser", - "checker-service", *PROOF_SERVICES, } model_proxy_enabled = values["WORKSTREAM_MODEL_PROXY_ENABLED"] == "true" @@ -945,7 +1235,6 @@ def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: "api", "worker", "beat", - "checker-service", *PROOF_SERVICES, } if model_proxy_enabled: @@ -1002,37 +1291,22 @@ def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: ), } shared_backend_mounts = { - str(SCRATCH_ROOT): (str(SCRATCH_ROOT), False), - str(SOCKET_ROOT): (str(SOCKET_ROOT), True), - str(INTAKE_SOCKET_ROOT): (str(INTAKE_SOCKET_ROOT), True), + str(paths.scratch): (str(paths.scratch), False), + str(paths.socket_root): (str(paths.socket_root), True), } expected_mounts = { - "checker-service": { - "/etc/workstream/checker-service.json": ( - values["WORKSTREAM_CHECKER_CONFIG_FILE"], - True, - ), - str(SCRATCH_ROOT): (str(SCRATCH_ROOT), True), - str(SOCKET_ROOT): (str(SOCKET_ROOT), False), - str(DOCKER_SOCKET): (str(DOCKER_SOCKET), False), - } | ( - { - str(MODEL_SOCKET_ROOT): (str(MODEL_SOCKET_ROOT), True), - str(MODEL_RELAY_ROOT): (str(MODEL_RELAY_ROOT), False), - } if model_proxy_enabled else {} - ), "api": shared_backend_mounts | runtime_identity_mounts, "worker": shared_backend_mounts - | {str(GUIDE_SOCKET_ROOT): (str(GUIDE_SOCKET_ROOT), True)} + | {str(paths.guide_root): (str(paths.guide_root), True)} | ( - {str(MODEL_SOCKET_ROOT): (str(MODEL_SOCKET_ROOT), True)} + {str(paths.model_root): (str(paths.model_root), True)} if model_proxy_enabled else {} ) | runtime_identity_mounts, "beat": {}, "guide-document-parser": { - str(GUIDE_SOCKET_ROOT): (str(GUIDE_SOCKET_ROOT), False), + str(paths.guide_root): (str(paths.guide_root), False), }, "aws-proof-runtime": runtime_identity_mounts, "aws-proof-readiness": { @@ -1067,10 +1341,15 @@ def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: } if model_proxy_enabled: expected_mounts["model-proxy"] = { - str(SCRATCH_ROOT): (str(SCRATCH_ROOT), False), - str(MODEL_SOCKET_ROOT): (str(MODEL_SOCKET_ROOT), False), + str(paths.scratch): (str(paths.scratch), False), + str(paths.model_root): (str(paths.model_root), False), **runtime_identity_mounts, } + if values.get(TRIAL_KEYS[0]): + expected_mounts["model-proxy"][str(TRIAL_MANIFEST_TARGET)] = ( + values[TRIAL_KEYS[0]], + True, + ) for name, mounts in expected_mounts.items(): _validate_exact_bind_mounts(name, services[name], mounts) expected_secret_files = { @@ -1093,17 +1372,6 @@ def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: raise PreflightError( f"release Compose secret file differs from configuration: {name}" ) - checker = services["checker-service"] - if checker.get("network_mode") != "none" or checker.get("ports"): - raise PreflightError("checker controller must be networkless and unpublished") - checker_groups = checker.get("group_add") - if ( - not isinstance(checker_groups, list) - or len(checker_groups) != 1 - or not isinstance(checker_groups[0], str) - or not re.fullmatch(r"[1-9][0-9]*", checker_groups[0]) - ): - raise PreflightError("checker controller must use only the Docker socket group") parser = services["guide-document-parser"] parser_environment = parser.get("environment") if ( @@ -1136,7 +1404,7 @@ def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: or parser_environment != { "WORKSTREAM_RELEASE_ROLE": "parser", - "WORKSTREAM_GUIDE_DOCUMENT_PARSER_SOCKET": str(GUIDE_SOCKET_PATH), + "WORKSTREAM_GUIDE_DOCUMENT_PARSER_SOCKET": str(paths.guide_socket), } or parser.get("command") != [ @@ -1144,7 +1412,7 @@ def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: "-m", "app.adapters.guide_document_tools.service", "--socket", - str(GUIDE_SOCKET_PATH), + str(paths.guide_socket), ] ): raise PreflightError("guide parser isolation profile is invalid") @@ -1156,7 +1424,7 @@ def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: "app.adapters.guide_document_tools.service", "--healthcheck", "--socket", - str(GUIDE_SOCKET_PATH), + str(paths.guide_socket), ]: raise PreflightError("guide parser healthcheck is invalid") parser_tmpfs = parser.get("tmpfs") @@ -1187,10 +1455,6 @@ def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: "required": True, }: raise PreflightError("worker must wait for a healthy model proxy") - if model_proxy_enabled and services["checker-service"].get("depends_on", {}).get( - "model-proxy" - ) != {"condition": "service_healthy", "required": True}: - raise PreflightError("checker service must wait for a healthy model proxy") worker = services["worker"] worker_command = worker.get("command") if ( @@ -1213,8 +1477,11 @@ def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: or proxy.get("stop_grace_period") not in {"7320s", "2h2m0s"} or proxy.get("healthcheck", {}).get("test") != [ - "CMD-SHELL", - "test -S /var/lib/workstream/model-runtime/model.sock", + "CMD", + "python", + "-c", + "import pathlib,sys; sys.exit(0 if pathlib.Path(sys.argv[1]).is_socket() else 1)", + str(paths.model_socket), ] ): raise PreflightError("model proxy delivery profile is invalid") @@ -1315,6 +1582,7 @@ def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: "WORKSTREAM_RELEASE_ROLE", "WORKSTREAM_GUIDE_DOCUMENT_PARSER_SOCKET", "WORKSTREAM_CHECKER_MODEL_PROXY_SOCKET", + *TRIAL_KEYS, } } ) @@ -1338,10 +1606,10 @@ def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: "WORKSTREAM_ARTIFACT_S3_ROLES_ANYWHERE_PRIVATE_KEY_FILE": str( ROLES_ANYWHERE_PRIVATE_KEY_TARGET ), - "WORKSTREAM_ARTIFACT_SCRATCH_ROOT": str(SCRATCH_ROOT), - "WORKSTREAM_EXTERNAL_CHECKER_MATERIAL_ROOT": str(SCRATCH_ROOT), - "WORKSTREAM_EXTERNAL_CHECKER_SERVICE_SOCKET": str(SOCKET_PATH), - "WORKSTREAM_ARTIFACT_INTAKE_SERVICE_SOCKET": str(INTAKE_SOCKET_PATH), + "WORKSTREAM_ARTIFACT_SCRATCH_ROOT": str(paths.scratch), + "WORKSTREAM_EXTERNAL_CHECKER_MATERIAL_ROOT": str(paths.scratch), + "WORKSTREAM_EXTERNAL_CHECKER_SERVICE_SOCKET": str(paths.socket), + "WORKSTREAM_ARTIFACT_INTAKE_SERVICE_SOCKET": str(paths.socket), **{key: values[key] for key in INTAKE_DIGEST_KEYS}, } for key, value in exact.items(): @@ -1357,26 +1625,47 @@ def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: service_environment = services[name].get("environment") if not isinstance(service_environment, dict): raise PreflightError(f"{name} release environment is unavailable") + deadline_key = "WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS" + if str(service_environment.get(deadline_key, "")) != values[deadline_key]: + raise PreflightError( + f"rendered artifact preparation deadline differs from release: {name}.{deadline_key}" + ) for key in PROJECT_AGENT_CONFIGURATION_KEYS: if str(service_environment.get(key, "")) != values[key]: raise PreflightError( "rendered project agent configuration differs from release: " f"{name}.{key}" ) + for name, service in services.items(): + environment = service.get("environment") or {} + expected_trial = ( + { + TRIAL_KEYS[0]: str(TRIAL_MANIFEST_TARGET), + TRIAL_KEYS[1]: values.get(TRIAL_KEYS[1]), + } + if name == "model-proxy" and values.get(TRIAL_KEYS[0]) + else {} + ) + if { + key: environment[key] for key in TRIAL_KEYS if key in environment + } != expected_trial: + raise PreflightError( + "trial authority is exposed outside its model-proxy owner" + ) worker_environment = environments["worker"] if worker_environment.get("WORKSTREAM_GUIDE_DOCUMENT_PARSER_SOCKET") != str( - GUIDE_SOCKET_PATH + paths.guide_socket ): raise PreflightError("worker lacks the fixed guide parser socket") if model_proxy_enabled and worker_environment.get( "WORKSTREAM_CHECKER_MODEL_PROXY_SOCKET" - ) != str(MODEL_SOCKET_PATH): + ) != str(paths.model_socket): raise PreflightError("worker lacks the fixed model proxy socket") if model_proxy_enabled and environments["model-proxy"].get( "WORKSTREAM_CHECKER_MODEL_PROXY_SOCKET" - ) != str(MODEL_SOCKET_PATH): + ) != str(paths.model_socket): raise PreflightError("model proxy lacks its fixed socket") - for name in ("api", "beat", "checker-service", *PROOF_SERVICES): + for name in ("api", "beat", *PROOF_SERVICES): service_environment = services[name].get("environment") or {} if not isinstance(service_environment, dict): raise PreflightError(f"{name} release environment is invalid") @@ -1429,73 +1718,215 @@ def validate_compose(rendered: dict[str, Any], values: dict[str, str]) -> None: raise PreflightError(f"secret value was interpolated into Compose: {key}") -def validate_docker_runtime(config: dict[str, Any], values: dict[str, str]) -> None: - """Check local runsc registration and exact preloaded image identities.""" - runtimes = _run_json( - ["docker", "info", "--format", "{{json .Runtimes}}"], - failure="Docker runtime inventory is unavailable", - environment=compose_environment(values), +def _verify_smol_asset(path: Path, expected: str, uid: int, deadline: float) -> None: + """Hash pinned regular assets without symlinks or writable substitutions.""" + try: + if path.resolve(strict=True) != path: + raise PreflightError("native Smol asset path is not canonical") + descriptor = os.open( + path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC + ) + with os.fdopen(descriptor, "rb") as asset: + metadata = os.fstat(asset.fileno()) + if ( + not stat.S_ISREG(metadata.st_mode) + or metadata.st_uid not in {0, uid} + or metadata.st_mode & 0o022 + ): + raise PreflightError("native Smol asset custody is unsafe") + digest = hashlib.sha256() + while True: + if time.monotonic() >= deadline: + raise PreflightError( + "native Smol asset verification deadline exceeded" + ) + chunk = asset.read(65536) + if not chunk: + break + digest.update(chunk) + current = path.lstat() + if ( + not os.path.samestat(metadata, current) + or metadata.st_size != current.st_size + or "sha256:" + digest.hexdigest() != expected + ): + raise PreflightError("native Smol asset digest or identity differs") + except PreflightError: + raise + except (OSError, RuntimeError) as exc: + raise PreflightError("native Smol asset is unavailable") from exc + + +def validate_smol_runtime(config: dict[str, Any], values: dict[str, str]) -> None: + """Check pinned native inputs, delegation and the existing typed node health.""" + paths = deployment_paths(values) + uid = _positive_id(values, "WORKSTREAM_RUNTIME_UID") + gid = _positive_id(values, "WORKSTREAM_RUNTIME_GID") + smol = config["smol"] + _private_metadata( + Path(smol["home"]), kind="directory", uid=uid, gid=gid, mode=0o700 ) - if not isinstance(runtimes, dict) or "runsc" not in runtimes: - raise PreflightError("Docker does not report the required runsc runtime") - if values["WORKSTREAM_MODEL_PROXY_ENABLED"] == "true": - runtime = runtimes["runsc"] - arguments = runtime.get("runtimeArgs", []) if isinstance(runtime, dict) else [] - if not isinstance(arguments, list) or [ - argument for argument in arguments - if isinstance(argument, str) and argument.startswith("--host-uds") - ] != ["--host-uds=open"]: - raise PreflightError( - "model-enabled checker runsc requires exactly --host-uds=open" - ) + group = Path(smol["cgroup_root"]) + try: + if ( + group.resolve(strict=True) != group + or group.stat().st_uid != uid + or not group.joinpath("controller").is_dir() + ): + raise PreflightError("native Smol cgroup delegation identity differs") + for name in ("cgroup.controllers", "cgroup.subtree_control"): + with group.joinpath(name).open("r", encoding="ascii") as control: + controllers = control.read(4097) + if len(controllers) > 4096 or not {"cpu", "memory", "pids"} <= set( + controllers.split() + ): + raise PreflightError("native Smol cgroup delegation is incomplete") + except (OSError, UnicodeError) as exc: + raise PreflightError("native Smol cgroup delegation is unavailable") from exc + deadline = time.monotonic() + 30 + _verify_smol_asset( + checker_service_binary(values), + values["WORKSTREAM_CHECKER_SERVICE_BINARY_SHA256"], + uid, + deadline, + ) + for path, digest in [ + (smol["binary"], smol["binary_sha256"]), + (smol["guest_binary"], smol["guest_sha256"]), + ]: + _verify_smol_asset(Path(path), digest, uid, deadline) for entry in config["cache"]: - reference = f"{entry['repository']}@{entry['platform_manifest_digest']}" - inspected = _run_json( - ["docker", "image", "inspect", reference], - failure="one configured checker image is not preloaded", - environment=compose_environment(values), + _verify_smol_asset( + Path(entry["smol"]["archive"]), + entry["smol"]["archive_sha256"], + uid, + deadline, ) - if not isinstance(inspected, list) or len(inspected) != 1: - raise PreflightError("checker image inspection is ambiguous") - image = inspected[0] - if ( - image.get("Id") != entry["config_image_id"] - or image.get("Os") != entry["operating_system"] - or image.get("Architecture") != entry["architecture"] - or reference not in image.get("RepoDigests", []) + _verify_smol_asset( + Path(entry["smol"]["image_config"]), entry["config_image_id"], uid, deadline + ) + if values.get(TRIAL_KEYS[0]): + _verify_smol_asset( + Path(values[TRIAL_KEYS[0]]), values[TRIAL_KEYS[1]], uid, deadline + ) + trial = config["trial_runtime"] + for key in ( + "relay_root", + "scratch_root", + "child_home_root", + "offline_cache_root", ): - raise PreflightError( - "preloaded checker image identity differs from configuration" + _private_metadata( + Path(trial[key]), kind="directory", uid=uid, gid=gid, mode=0o700 ) - + for path_key, digest_key in ( + ("trusted_launcher_path", "trusted_launcher_digest"), + ("filesystem_tool_path", "filesystem_tool_digest"), + ): + _verify_smol_asset(Path(trial[path_key]), trial[digest_key], uid, deadline) + script = ( + "import asyncio,json; from dataclasses import asdict; from pathlib import Path; " + "from app.core.config import Settings; " + "from app.adapters.checkers import configured_external_checker_execution; " + "s=Settings(external_checker_service_socket=Path(" + + repr(str(paths.socket)) + + "),artifact_scratch_root=Path(" + + repr(str(paths.scratch)) + + "),external_checker_material_root=Path(" + + repr(str(paths.scratch)) + + "),external_checker_service_timeout_seconds=5.0); " + "print(json.dumps(asdict(asyncio.run(configured_external_checker_execution(s).health()))))" + ) + health = _run_json( + [ + "docker", + "run", + "--rm", + "--pull=never", + "--network", + "none", + "--read-only", + "--cap-drop=ALL", + "--security-opt=no-new-privileges", + "--pids-limit=64", + "--memory=512m", + "--cpus=1", + "--user", + str(uid) + ":" + str(gid), + "--mount", + f"type=bind,src={paths.socket_root},dst={paths.socket_root},readonly", + "--entrypoint", + "python", + values["WORKSTREAM_BACKEND_IMAGE"], + "-c", + script, + ], + failure="native Smol project readiness failed", + environment=compose_environment(values), + ) + if ( + not isinstance(health, dict) + or health.get("runtime") != "smolvm" + or health.get("isolation_mode") != "smol-microvm" + or health.get("operating_system") != config["operating_system"] + or health.get("architecture") != config["architecture"] + or health.get("sandbox_uid") != uid + or health.get("sandbox_gid") != gid + or health.get("cached_platform_manifests") != len(config["cache"]) + ): + raise PreflightError("native Smol project readiness identity differs") def validate_trusted_intake(values: dict[str, str]) -> None: """Use the release image's typed ART adapter without mounting credentials.""" + paths = deployment_paths(values) script = ( "import asyncio,json; from dataclasses import asdict; from pathlib import Path; " "from app.core.config import Settings; " "from app.adapters.artifacts import trusted_submission_intake; " "s=Settings(artifact_intake_service_socket=Path(" - + repr(str(INTAKE_SOCKET_PATH)) + "),artifact_intake_image_digest=" - + repr(values[INTAKE_DIGEST_KEYS[0]]) + ",artifact_intake_config_digest=" - + repr(values[INTAKE_DIGEST_KEYS[1]]) + "); " + + repr(str(paths.socket)) + + "),artifact_intake_image_digest=" + + repr(values[INTAKE_DIGEST_KEYS[0]]) + + ",artifact_intake_config_digest=" + + repr(values[INTAKE_DIGEST_KEYS[1]]) + + "); " "print(json.dumps(asdict(asyncio.run(trusted_submission_intake(s).health()))))" ) health = _run_json( - ["docker", "run", "--rm", "--pull=never", "--network", "none", "--read-only", - "--cap-drop=ALL", "--security-opt=no-new-privileges", "--pids-limit=64", - "--memory=512m", "--cpus=1", "--user", - values["WORKSTREAM_RUNTIME_UID"] + ":" + values["WORKSTREAM_RUNTIME_GID"], - "--mount", f"type=bind,src={INTAKE_SOCKET_ROOT},dst={INTAKE_SOCKET_ROOT},readonly", - "--entrypoint", "python", values["WORKSTREAM_BACKEND_IMAGE"], "-c", script], + [ + "docker", + "run", + "--rm", + "--pull=never", + "--network", + "none", + "--read-only", + "--cap-drop=ALL", + "--security-opt=no-new-privileges", + "--pids-limit=64", + "--memory=512m", + "--cpus=1", + "--user", + values["WORKSTREAM_RUNTIME_UID"] + ":" + values["WORKSTREAM_RUNTIME_GID"], + "--mount", + f"type=bind,src={paths.socket_root},dst={paths.socket_root},readonly", + "--entrypoint", + "python", + values["WORKSTREAM_BACKEND_IMAGE"], + "-c", + script, + ], failure="trusted Smol intake readiness failed", environment=compose_environment(values), ) - if (not isinstance(health, dict) - or (health.get("runtime"), health.get("isolation_mode")) != ("smolvm", "smol-microvm") - or health.get("image_digest") != values[INTAKE_DIGEST_KEYS[0]] - or health.get("config_digest") != values[INTAKE_DIGEST_KEYS[1]]): + if ( + not isinstance(health, dict) + or (health.get("runtime"), health.get("isolation_mode")) + != ("smolvm", "smol-microvm") + or health.get("image_digest") != values[INTAKE_DIGEST_KEYS[0]] + or health.get("config_digest") != values[INTAKE_DIGEST_KEYS[1]] + ): raise PreflightError("trusted intake readiness identity differs") @@ -1505,7 +1936,7 @@ def parse_args() -> argparse.Namespace: parser.add_argument( "--configuration-only", action="store_true", - help="skip host files, Docker runsc and image-cache checks", + help="skip native host inputs, Smol readiness and image-cache checks", ) return parser.parse_args() @@ -1522,16 +1953,16 @@ def main() -> int: validate_compose(render_compose(args.env_file, values), values) if args.configuration_only: print( - "configuration valid; host identity, runsc/cache, cloud, health and recovery checks skipped" + "configuration valid; host identity, Smol/cache, cloud, health and recovery checks skipped" ) return 0 validate_private_host_inputs(values) - validate_docker_runtime(checker_config, values) + validate_smol_runtime(checker_config, values) validate_trusted_intake(values) except PreflightError as exc: print(f"hosted release preflight failed: {exc}", file=sys.stderr) return 1 - print("hosted release configuration and local runsc/cache prerequisites are valid") + print("hosted release configuration and native Smol prerequisites are valid") return 0 diff --git a/deploy/hosted/release.env.example b/deploy/hosted/release.env.example index b52cf47a4..cace61dba 100644 --- a/deploy/hosted/release.env.example +++ b/deploy/hosted/release.env.example @@ -3,12 +3,13 @@ COMPOSE_PROJECT_NAME=workstream-release WORKSTREAM_BACKEND_IMAGE=registry.example/workstream/backend@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa WORKSTREAM_MODEL_PROXY_IMAGE=registry.example/workstream/backend-agents@sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd +WORKSTREAM_CHECKER_SERVICE_BINARY_SHA256=sha256:9999999999999999999999999999999999999999999999999999999999999999 WORKSTREAM_MODEL_PROXY_ENABLED=true -WORKSTREAM_CHECKER_SERVICE_IMAGE=registry.example/workstream/checker-service@sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb +WORKSTREAM_DEPLOYMENT_ROOT=/var/lib/workstream +WORKSTREAM_CHECKER_SERVICE_BINARY=/opt/workstream/bin/workstream-checker-service WORKSTREAM_RUNTIME_UID=10001 WORKSTREAM_RUNTIME_GID=10001 -WORKSTREAM_DOCKER_SOCKET_GID=999 WORKSTREAM_API_HOST_PORT=8000 WORKSTREAM_CELERY_WORKER_CONCURRENCY=2 @@ -100,6 +101,11 @@ WORKSTREAM_ARTIFACT_S3_OPERATION_TOTAL_TIMEOUT_SECONDS=300 WORKSTREAM_ARTIFACT_COMPLETE_READ_DEADLINE_SECONDS=300 WORKSTREAM_ARTIFACT_TERMINAL_PERSISTENCE_MARGIN_SECONDS=120 +# Existing ART processing deadline; keep ZIP inspection (300s) below it and +# deadline + cleanup margin (300s) below scratch reservation TTL (3900s). +# This operational setting does not change registered checker/project limits. +WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS=3400 + # The collector origin is optional. Production-like Settings require HTTPS. WORKSTREAM_OBSERVABILITY_OTLP_ENDPOINT= WORKSTREAM_OBSERVABILITY_TRACE_SAMPLE_RATIO=0.1 @@ -108,7 +114,13 @@ WORKSTREAM_OBSERVABILITY_LOG_LEVEL=INFO WORKSTREAM_CHECKER_CONFIG_FILE=/etc/workstream/checker-service.json # Provision the trusted Smol intake node before launching API/Celery. Its private -# socket is /var/lib/workstream/intake-runtime/checker.sock, owned by 10001:10001. +# socket is /var/lib/workstream/checker-runtime/checker.sock, owned by 10001:10001. # Replace both placeholders with identities verified from the built intake image. -WORKSTREAM_ARTIFACT_INTAKE_IMAGE_DIGEST=sha256:0000000000000000000000000000000000000000000000000000000000000000 -WORKSTREAM_ARTIFACT_INTAKE_CONFIG_DIGEST=sha256:0000000000000000000000000000000000000000000000000000000000000000 +WORKSTREAM_ARTIFACT_INTAKE_IMAGE_DIGEST=sha256:1111111111111111111111111111111111111111111111111111111111111111 +WORKSTREAM_ARTIFACT_INTAKE_CONFIG_DIGEST=sha256:2222222222222222222222222222222222222222222222222222222222222222 + +# Optional only for approved POST trial selectors (requires the Smol trial runtime). +# Configure both here and pin the same host path/digest in the native service. +# Ordinary POST model calls do not require trial authority. +# WORKSTREAM_CHECKER_TRIAL_AUTHORITY_MANIFEST_PATH=/etc/workstream/trial-authority.json +# WORKSTREAM_CHECKER_TRIAL_AUTHORITY_MANIFEST_DIGEST=sha256:replace-with-canonical-manifest-digest diff --git a/docs/architecture_checker_framework.md b/docs/architecture_checker_framework.md index 3b9361fc6..c2bc0aafd 100644 --- a/docs/architecture_checker_framework.md +++ b/docs/architecture_checker_framework.md @@ -307,10 +307,14 @@ closed deployment pairs are `runsc/gvisor`, `runc/docker-dev`, and `smolvm/smol-microvm`; project requests cannot select an engine. Smol uses a fresh KVM VM, a static trusted bootstrap, workload and host resource limits, bounded concurrency and independent exact-lease cleanup guardians. Its node requires -Linux amd64, KVM and delegated cgroup v2. The existing hosted Docker composition -still requires `runsc`; replacing that hosted project-execution wiring is a -separate integration boundary. Trusted ZIP/default checking already uses the -Smol intake capability. See the [node contract](../external_checkers/README.md#smol-microvm-engine). +Linux amd64, KVM and delegated cgroup v2. The hosted release composition uses the existing native Smol systemd node for +two typed capabilities at one private endpoint: trusted intake and project +execution. API/Celery worker containers share its absolute ART material path and numeric +identity and mount its socket directory read-only; Compose starts no checker +controller or Docker/runsc fallback. Guest-native model/trial callbacks retain +their existing invocation custody and trusted proxy ownership. PRE model +selectors remain unsupported. Hosted configuration and readiness checks do not +establish a real public journey or model call. See the [node contract](../external_checkers/README.md#smol-microvm-engine). Project-specific requirements are implemented by registered external images. The shared service accepts the normalized result contract; it does not interpret diff --git a/docs/decision_0014_external_service_adapter_convention.md b/docs/decision_0014_external_service_adapter_convention.md index 3bbf61b46..08abd96c6 100644 --- a/docs/decision_0014_external_service_adapter_convention.md +++ b/docs/decision_0014_external_service_adapter_convention.md @@ -64,16 +64,20 @@ authority or construct a transport. The adapter factory is instance-local and ex hidden CHECKERS composition. Durable PRE/POST owners retain request, lease and result custody, while ART owns verified material and callback-scoped grants. Production request/completion registration does not expose public intake. -Local integration uses runc/docker-dev; hosted gVisor proof remains separate. +Local Docker integration uses runc/docker-dev. Hosted release configuration +selects the existing native Smol node through the same typed port; actual hosted +qualification remains separate. -The Rust service alone owns container-engine access. Python sends an opaque, +The Rust service alone owns sandbox-engine access. Python sends an opaque, request-bound callback grant issued from ART's existing pre-admission scratch lifetime. The service resolves it beneath the configured private scratch root, recomputes exact file and directory facts, and mounts only the sealed workspace read-only. Its trusted cache binds one OCI platform-manifest digest to repository, OS, architecture and observed config image ID; it never pulls or resolves tags. -Hosted health fails closed without `runsc`. The explicit `docker-dev` mode is a -local mechanism qualification and cannot report hosted gVisor isolation. +Hosted readiness requires the configured native Smol identity, delegation and +pinned cached inputs; the release does not start a Docker/runsc fallback. The +explicit `docker-dev` mode remains local mechanism qualification, outside the +selected hosted topology. For an approved POST capability that needs to build or execute submitted work, the same service may expose one invocation-local Unix socket to the selected diff --git a/docs/roadmap_status.md b/docs/roadmap_status.md index c5c6d0ea7..1ff7336ed 100644 --- a/docs/roadmap_status.md +++ b/docs/roadmap_status.md @@ -41,10 +41,14 @@ through governed outcome on a one-slot Smol node, including cancellation, ordered project PRE, runtime recovery and public Operator POST retry. Locked human review remains pending; locked automatic acceptance creates the submitter ContributionRecord and applicable award without a fabricated Review. Hosted -composition requires pinned intake settings. Hosted Smol composition and its -scoped AWS proof are tracked separately in [PR #692](https://github.com/Flow-Research/workstream/pull/692), -whose owned baseline passed with its disclosed fixture identity. Production Flow -identity, live-model grading and full project qualification remain separate. See +composition selects the existing native Smol node for typed intake and project +execution through one private endpoint. The owned hosted AWS baseline passed +with its disclosed fixture identity. A scoped live OpenRouter POST also passed +through the native guest, trusted proxy, durable worker and governed acceptance, +retaining one paid effect and unchanged accounting on replay. Approved callbacks +require their pinned authority configuration. PRE model selectors remain +unsupported. Production Flow identity, substantive grading quality and the full +project journey remain separate. See [WS-EXEC-001](../.commitrail/initiatives/WS-EXEC-001/OVERVIEW.md). The artifact diagnosis API uses exact, @@ -481,10 +485,13 @@ and finite-retention controls, and an operational end-to-end drill remain release work; diagnostics do not establish lifecycle truth. The first hosted release configuration now separates immutable API/Celery/Beat -and checker-service images from the development stack, requires real Flow -verification and AWS workload identity, and binds the existing checker service -to a dedicated Docker/runsc host with exact private material paths and cached -image identities. It carries the required release digest and exact AWS +from the development stack, requires real Flow verification and AWS workload +identity, and binds project execution and trusted intake to the existing native +Smol node with exact private material paths and pinned cached image identities. +Compose starts no checker controller; credential-owning model proxy and native +node use the same host-visible socket path under one numeric identity. Cold +startup explicitly orders migrations, AWS activation, proxy, native readiness +and API/Celery worker release. It carries the required release digest and exact AWS role/policy/trust/bucket-policy identities into every long-running process environment. The external-host credential integration uses AWS Roles Anywhere with dedicated certificate identities, preserving the same S3 adapter and @@ -572,7 +579,7 @@ cannot be reused as post-submission review-gate evidence. See the | Lifecycle stage | Intended merged status | What is already proven | What remains before v0.1 | | --- | --- | --- | --- | -| Shared setup and project-image model use | **Budgeted runtime and public Operator controls** | One project-lifetime funded account serves exact setup-attempt and approved POST image lineage; immutable bounded profiles; finite multi-call/run envelopes and setup document-tool caps; private ART request/response evidence; stable paid-effect replay/recovery; once-only settlement; exact redacted Operator effect inspection; hosted composition gives only configured provider credentials to the trusted proxy; locked POST profiles select OpenAI Responses or OpenRouter Chat Completions, while ordinary POST needs neither; native KVM composition retains two verified trial receipts, four controlled model effects and no-call replay; active nested cancellation/child-death/controller-restart proofs retain no false completion or trial ART effects and prove full cleanup/capacity reuse; the owned hosted baseline tracked separately in [PR #692](https://github.com/Flow-Research/workstream/pull/692) qualifies its disclosed fixture scope | Complete only the authorized aggregate-capped live-provider proof with the exact account/profile and retain its actual verdict and paid custody. Native composition uses a controlled provider; the initial hosted model attempt stopped before any provider effect. Substantive rubric quality, production Flow identity and the full Terminal Systems journey remain separate | +| Shared setup and project-image model use | **Budgeted runtime and public Operator controls** | One project-lifetime funded account serves exact setup-attempt and approved POST image lineage; immutable bounded profiles; finite multi-call/run envelopes and setup document-tool caps; private ART request/response evidence; stable paid-effect replay/recovery; once-only settlement; exact redacted Operator effect inspection; hosted composition gives only configured provider credentials to the trusted proxy; locked POST profiles select OpenAI Responses or OpenRouter Chat Completions, while ordinary POST needs neither; native KVM composition retains two verified trial receipts, four controlled model effects and no-call replay; active nested cancellation/child-death/controller-restart proofs retain no false completion or trial ART effects and prove full cleanup/capacity reuse; the owned hosted baseline tracked separately in [PR #692](https://github.com/Flow-Research/workstream/pull/692) qualifies its disclosed fixture scope | One scoped live OpenRouter POST at `5ab96251f` reused the retained prepared effect, reached governed acceptance and settled US$0.016527 in the pinned-rate ledger with no outstanding reservation or duplicate effect. This proves that transport/accounting path; substantive rubric quality, production Flow identity, live setup rehearsal and the full Terminal Systems journey remain separate | | Identity and actor resolution | **Live foundation** | Flow-token verification; canonical ActorProfile and ActorIdentityLink; human/service separation; lifecycle controls; canonical `/actors/me` self-read with duplicate `/auth/me` removed; suspension denies both self-read and self-update; CLI privileged exact-actor/profile and identity-link reads and explicit separate profile/link lifecycle writes remain distinct from self-service and backend guards | Final end-to-end operational and conformance proof | | Authorization kernel | **Live foundation** | Closed action/permission catalogues; deny-by-default evaluation; grants; exact-project, privacy-safe candidate read now consumed by the CLI; fixed services; rate controls; opaque transaction-bound PREP; atomic decision evidence; CLI scope-filtered administrative grant/catalogue reads and explicit keyed grant issue/revoke consume existing public APIs without local authority decisions | Activate only the remaining owner-proven TASK, checker, REV, and CON boundaries; remove obsolete authority after replacement paths are live | | System project catalogue | **Bounded public API and CLI read** | WS-CLI-001-46 adds `GET /api/v1/projects` and `workstream project list` for current system Project Manager, Operator, Finance or Audit `project.read` grants; each signed page exposes only ID, name and status. `me projects` retains exact-grant membership semantics. | System-catalogue TUI picker integration, deployed CLI and hosted conformance remain separate | @@ -585,7 +592,7 @@ cannot be reused as post-submission review-gate evidence. See the | Operator artifact diagnosis | **Public API and CLI with exact Operator authority** | Eight bounded artifact GETs use current persisted system-Operator grants and exact AUTH decisions, with response-validated read commits and concealed denials. An independently guarded reasoned verification retry atomically records its decisions, recovery chain and audit; exact replay rechecks authority. The CLI consumes all nine routes, with malformed-lineage rejection and receipt-specific pagination. Signed PostgreSQL proof covers eligible retry, exact replay, wrong-resource refusal and revoked-Operator concealment. | Hosted S3 proof remains separate; diagnosis does not grant provider access or activate AWS | | Bulk task import | **Public backend and CLI batch journey** | PILOT-02A supplies exact source custody; PILOT-02B imports 1..500 verified JSON rows under one immutable native batch, retains per-Task AUTH/TaskCreated facts, recovers exact original IDs with fresh authority and exposes batch status; actual 200-row, rollback, concurrent-wait and SQL-custody controls pass on PostgreSQL/MinIO. PILOT-02C adds atomic whole-batch screen/release with fresh per-Task authority, locked guide/policy custody and explicit CLI/CSV commands. WS-CLI-001-31 adds bounded, hash-checked CLI source download through public custody with no-overwrite local publication. Optional task-local criteria remain optional through single/bulk screening and release; required instructions and guide locks remain enforced. | No remaining PILOT-02 implementation gate; deployment and full contributor-lifecycle proof are separate | | Contributor artifact preparation | **Public initial and checker-remediation entry** | One outer ZIP through the published preparation route; unreadable or structurally rejected ZIPs return sanitized non-retryable 422 feedback before PRE reservation or durable effects; bounded scratch inspection; canonical manifest; shared evaluation-content capacity and exact locked-policy checks before attempts or durable intent; platform and project prechecks; unchanged-work rejection; durable put intent; verification; capacity-charged ready admission; final handoff validates the exact activated historical guide through owner ports; same-key ready replay does not repeat checker or PUT. The preparing contributor reads back their own preparations with outcome, storage state, finding counts and intake findings (PILOT-READ-2); a Project Manager lists a project's preparations with finding counts only (PILOT-READ-3). CLI preparation, read-back, creation/replay and history consume these public owners | Bounded local API and native CLI journeys passed, including CLI rejected-intake recovery; hosted operation remains separate | -| Pre-submission intake checking | **Public selected entry with approved-guide lineage and bounded feedback** | One versioned Workstream default checker enforces safe ZIP/resource bounds, independently verified digest/manifest, high-confidence secret rejection and unchanged-work rejection. Missing summary/attestation produces warnings only. Successful preparation and its exact replay return the same bounded intake results as a blocked one (PILOT-READ-1), so nonblocking findings such as coverage gaps stay visible; CLI preparation prints them. Approved project PRE checks are immutable digest-pinned images run by the Rust service on verified read-only ART material. A supported system Operator API and explicit CLI commands register and read exact immutable external-checker entries with server-selected UUIDv7 identity, fresh authority and replay recovery. Durable selected custody takes TASK before CHECKERS; only a complete pass grants admission to the existing atomic Submission transaction. Both executable catalogues are removed. CLI preparation prints bounded contributor failure details. The same typed adapter also accepts the Smol node's fresh microVM execution receipts, with bounded concurrency and exact-lease recovery; ART uses a separate typed trusted intake capability, prepares during authorized upload/download, verifies raw guest exports and closes intake before project execution. API and Celery workers require a configured pinned Smol intake image/socket, including when project execution uses Docker/runsc. Hosted API/Celery composition requires and validates the pinned intake socket/image. The isolated public Smol drill proves upload overlap, zero/two PRE bindings, cancellation, bounded runtime retry/recovery and public Operator POST retry through both governed policy branches | Bounded self-hosted public Smol/Celery drill passed; production Flow, native AWS S3 and intended hosted deployment proof remain separate. Registry publication alone does not activate an image or repair an approved compilation. Work findings create no Submission and preserve in_progress; infrastructure failures retain recoverable outcomes. The release configuration is not deployed gVisor execution. Real-project setup must identify supported checks and missing capabilities before engineering, registration, approved-policy activation and production-limit selection; deployed hosted operation and public REV lifecycle controls remain separate | +| Pre-submission intake checking | **Public selected entry with approved-guide lineage and bounded feedback** | One versioned Workstream default checker enforces safe ZIP/resource bounds, independently verified digest/manifest, high-confidence secret rejection and unchanged-work rejection. Missing summary/attestation produces warnings only. Successful preparation and its exact replay return the same bounded intake results as a blocked one (PILOT-READ-1), so nonblocking findings such as coverage gaps stay visible; CLI preparation prints them. Approved project PRE checks are immutable digest-pinned images run by the Rust service on verified read-only ART material. A supported system Operator API and explicit CLI commands register and read exact immutable external-checker entries with server-selected UUIDv7 identity, fresh authority and replay recovery. Durable selected custody takes TASK before CHECKERS; only a complete pass grants admission to the existing atomic Submission transaction. Both executable catalogues are removed. CLI preparation prints bounded contributor failure details. The same typed adapter also accepts the Smol node's fresh microVM execution receipts, with bounded concurrency and exact-lease recovery; ART uses a separate typed trusted intake capability, prepares during authorized upload/download, verifies raw guest exports and closes intake before project execution. API and Celery workers require a configured pinned Smol intake image/socket. Hosted API/Celery composition uses one native Smol endpoint for typed intake and project execution, with read-only client socket mounts and no Compose checker controller. The isolated public Smol drill proves upload overlap, zero/two PRE bindings, cancellation, bounded runtime retry/recovery and public Operator POST retry through both governed policy branches | Bounded self-hosted public Smol/Celery drill passed; production Flow, native AWS S3 and intended hosted deployment proof remain separate. Registry publication alone does not activate an image or repair an approved compilation. Work findings create no Submission and preserve in_progress; infrastructure failures retain recoverable outcomes. Prepared native Smol configuration is not deployed execution proof. Real-project setup must identify supported checks and missing capabilities before engineering, registration, approved-policy activation and production-limit selection; deployed hosted operation and public REV lifecycle controls remain separate | | Immutable Submission creation | **Public admission-backed entry with explicit CLI consumption** | Canonical POST derives contributor identity and consumes a ready admission for initial work or exact checker remediation; durable pre-submit reservation and same-key ready recovery without rerunning checks or PUT; TASK-owned atomic creation with exact assignment ContributionPolicyVersion, locked policy lineage and service/database checked-packet custody; fixed-service artifact binding; exact replay, authority, concurrency and rollback proof. CLI creation/replay and bounded history use these public owners | Bounded local API and native CLI unpaid automated-acceptance journeys passed; live human-review revision and hosted operation remain separate | | Post-submission evaluation and governed outcome | **Selected external execution and production completion routing** | Immutable approved image bindings, Rust service/SDK and typed ART callback grants; exact verified input custody, bounded normalized JSON results and phase authority. Project-owned digest-pinned images implement project requirements inside the generic service contract; the former named terminal-benchmark continuation is retired. The shared service does not interpret project-private build plans or provide a host-side continuation. Production outbox request/completion handlers run selected images and preserve exact restart/replay. True-policy success reaches review_pending; false-policy success invokes shared authorized FinalAcceptance and atomically creates the submitter ContributionRecord and complete frozen award set. Contributor-correctable failures retain exact remediation causes; a supported public system-Operator read and explicit CLI page list expose known finalized infrastructure failures; exact public/CLI retry invokes the existing owner with fresh authority and no spent contributor round. Supported uncertain request/completion reconciliation uses at most one immutable child to verify committed facts or invoke the existing owner after proving no effect; unresolved sources and an UNKNOWN child stay held. PILOT-READ-1 extends the single external result contract with findings carrying `check_id`, `path`, `line` and the `coverage_incomplete` level, and stores each finding immutably. A coverage gap without errors never auto-accepts: under a true ReviewPolicy it reaches review_pending, otherwise it returns to the contributor as `needs_revision` (`checker_coverage_incomplete`) without spending a revision round. Each Submission records its evidence-guarded attempt outcome and decision time. The bounded local public API/Celery drill passed both branches, prefork Celery process restart and late authority revocation | Live human review and human-triggered contributor revision remain separate #554 work. Unsupported or human-review-only approved requirements block false activation. New false/pass effects require LIVE REV bound to the current reviewed manifest; a known pre-effect refusal retries within the existing OUTBOX budget, then dead-letters if exhausted. The full hosted interruption drill for uncertain request/completion reconciliation remains outstanding. True handoff has no REV/CON prerequisite. Project-specific image correctness, registration, approved-policy activation and qualification on the intended runtime remain separate requirements. Former named-profile experiments are historical evidence. Hosted operation and deployment remain unproven | | Operator uncertain-delivery CLI | **Existing public delivery routes consumed by explicit CLI commands** | WS-CLI-001-40 adds one project-bound page of finalized UNKNOWN deliveries and a deliberate reasoned, UUID-keyed reconciliation of one exact event. Fresh backend authority and classification remain decisive; a page never authorizes a later write. | Native signed PostgreSQL proof is scoped to an owner-produced source; a hosted interruption and deployed CLI remain unproven | @@ -596,7 +603,7 @@ cannot be reused as post-submission review-gate evidence. See the | CLI and terminal interface | **Public CLI clients and opt-in contributor read workbench** | Existing explicit commands retain text/JSON/noninteractive behavior. [WS-CLI-001-48](../.commitrail/initiatives/WS-CLI-001/WS-CLI-001-48.md) adds exact-grant project, Ready/active Mine/own Submission browsing and fresh details/context/intake with explicit pagination and fresh-denial handling. [WS-CLI-001-49](../.commitrail/initiatives/WS-CLI-001/WS-CLI-001-49.md) adds locked-guide metadata, confirmed verified local downloads and paged checker findings through the shared client | Contribution Evidence remains a manual CLI handoff; a shared typed single-page contribution port precedes in-TUI Contribution Evidence. TUI product writes/recovery and management remain separate. Protected first binary publication and hosted journey proof remain outstanding | | Fulfillment, reconciliation, and audit | **Planned** | Shared audit foundations, provider-neutral adapter convention, AUTH-OUTBOX-02 live dispatcher authority, retained phase audit decisions, Celery delivery/recovery scans and CON-02B custody | Feature-specific handlers and authority, conditional award fulfillment, callbacks, idempotent recovery, reconciliation, bounded operational reads, and release controls | | Runtime diagnostics | **Implemented foundation; release configuration prepared** | Closed structured logging, explicit API/Celery tracing, bounded metrics, safe correlation and optional typed OTLP export; hosted composition accepts only an optional HTTPS collector origin and keeps exporter outage outside product correctness | Restrict broker publishers; configure and secure collector/log access, egress, encryption and finite deletion; prove privacy, correlation and collector outage during the deployed release drill; no deployed monitoring claim | -| Frontend and pilot | **Local runtime foundation; release configuration prepared; project inputs prepared** | An original [Terminal Systems guide, example and five task briefs](../pilots/terminal_systems/README.md) provide inputs for fresh project onboarding. The fixed five-task sample, its checker, private runtime continuation and sample-only proof workflow are retired. Checkout-isolated Docker Compose API, prefork Celery, Beat, PostgreSQL, Redis and MinIO remain available, alongside digest-only hosted release composition and the generic external image service. The generic image command tests project samples and publishes a verified digest to ECR without registering or activating it. The opt-in [local agent-loop overlay](engineering/local-agent-loop.md) adds that service, a separately built networkless proof checker and operator scripts; Docker Desktop (Apple Silicon) proof covers registration, checker execution and lifecycle activation, not this project’s requirements. Setup supports inference and explicitly manager-authored rules through the same governed proposal path. Inference is limited to PRE/POST specifications, with strict JSON and bounded correction; PM policy choices remain separate. The retained real Terminal Systems proposal preserves null project limits on the unchanged guide; its independent review predates the PRE/POST-only scope, with ten explicit implementation/runtime gaps and no bindings in that retained proposal. The [project-owned PRE image](../pilots/terminal_systems/checkers/README.md) now implements layout and metadata intake with local rule and networkless container samples; it has no task-name allowlist or shared-runner project code. The project-owned POST implementation consumes generic model/trial capabilities and exact approved predecessor inputs; its qualification boundaries are recorded with the checker implementation. The independent offline gVisor feasibility spike remains historical mechanism evidence | Obtain a fresh PRE/POST-only correction of the retained proposal, reconcile the implemented PRE/POST images and qualify the exact runtime/material path; register the required images, obtain corrected policy bindings and separate approvals, then activate, import, screen and release tasks. Then prove the real contribution journey, runtime limits and interruption recovery on the intended local/hosted stack. Guide assets, image builds and former sample results are not live-project evidence. Hosted identity/AWS configuration, isolation, diagnostics, backup/restore and broader macOS runtime proof remain; the agent-loop overlay still needs approved guide bindings and a persistent-stack FinalAcceptance drill | +| Frontend and pilot | **Local runtime foundation; release configuration prepared; project inputs prepared** | An original [Terminal Systems guide, example and five task briefs](../pilots/terminal_systems/README.md) provide inputs for fresh project onboarding. The fixed five-task sample, its checker, private runtime continuation and sample-only proof workflow are retired. Checkout-isolated Docker Compose API, prefork Celery, Beat, PostgreSQL, Redis and MinIO remain available, alongside digest-only hosted backend composition and the existing native Smol image service. The generic image command tests project samples and publishes a verified digest to ECR without registering or activating it. The opt-in [local agent-loop overlay](engineering/local-agent-loop.md) adds that service, a separately built networkless proof checker and operator scripts; Docker Desktop (Apple Silicon) proof covers registration, checker execution and lifecycle activation, not this project’s requirements. Setup supports inference and explicitly manager-authored rules through the same governed proposal path. Inference is limited to PRE/POST specifications, with strict JSON and bounded correction; PM policy choices remain separate. The retained real Terminal Systems proposal preserves null project limits on the unchanged guide; its independent review predates the PRE/POST-only scope, with ten explicit implementation/runtime gaps and no bindings in that retained proposal. The [project-owned PRE image](../pilots/terminal_systems/checkers/README.md) now implements layout and metadata intake with local rule and networkless container samples; it has no task-name allowlist or shared-runner project code. The project-owned POST implementation consumes generic model/trial capabilities and exact approved predecessor inputs; its qualification boundaries are recorded with the checker implementation. The independent offline gVisor feasibility spike remains historical mechanism evidence | Obtain a fresh PRE/POST-only correction of the retained proposal, reconcile the implemented PRE/POST images and qualify the exact runtime/material path; register the required images, obtain corrected policy bindings and separate approvals, then activate, import, screen and release tasks. Then prove the real contribution journey, runtime limits and interruption recovery on the intended local/hosted stack. Guide assets, image builds and former sample results are not live-project evidence. Hosted identity/AWS configuration, isolation, diagnostics, backup/restore and broader macOS runtime proof remain; the agent-loop overlay still needs approved guide bindings and a persistent-stack FinalAcceptance drill | ## What Has Been Completed @@ -1521,9 +1528,10 @@ v0.1 is not ready until all of the following are true: The provider-free `manager_authored` runtime removes the inference dependency from local and hosted setup and selects no provider credential. It proves nothing about model quality. Both inference runtimes now pass the bounded - original-guide qualification. Exact OpenRouter account/plugin evidence, the - selected-image hosted/AWS proof and the broader Flow contribution journey - remain. + original-guide qualification. The scoped selected-image hosted/AWS baseline + and one live OpenRouter POST are now qualified below. Substantive setup/model + quality, historical spend reconciliation and the broader production Flow + contribution journey remain. - Publish the reviewed backend and checker-service source builds, substitute their registry digests and the approved exact checker cache into the prepared release configuration, and rerun its full host preflight. Then prove actual @@ -1536,12 +1544,15 @@ v0.1 is not ready until all of the following are true: and invocation sockets under the intended isolation, and recover interrupted paid effects without another create. The Terminal Systems live rehearsal has one US$10 aggregate cap across SDKs, images and retries; offline transport, - ART and ledger tests do not establish its full project journey. -- Provision the pinned Smol intake image/socket in the intended hosted deployment - and repeat the public PRE -> immutable Submission -> durable POST journey with - actual Flow/AWS identities and its measured resource limits. The isolated - self-hosted Smol drill already covers both governed policy outcomes and recovery; - its local Flow fixture, recorded guide inference and MinIO are not hosted certification. + ART and ledger tests do not establish its full project journey. The separate + authorized US$10 model-only qualification passed one live POST with retained + paid-effect recovery and replay; it does not reconcile the earlier setup spend. +- Apply the qualified Smol intake/execution wiring to the intended production + deployment and repeat the public PRE -> immutable Submission -> durable POST + journey with production Flow identity. The isolated hosted AWS baseline covers + both governed policy outcomes, recovery and measured limits, and the scoped + live POST proves model transport/accounting. Its fixture identity and synthetic + engineering guide remain distinct from production project qualification. - Prove artifact diagnosis/retry in the intended hosted S3 environment; local public API/CLI proof does not establish hosted provider operation. - A task cannot enter `READY` without that complete locked context. @@ -1639,9 +1650,12 @@ proof-checker trust limit, operator scripts and the unproven persistent-stack FinalAcceptance path. The [hosted release configuration record](../.commitrail/changes/mvp-hosted-release-configuration.md) -and [operator runbook](../deploy/hosted/README.md) define the digest-only process -topology, dedicated Docker/runsc trust boundary, private configuration and -runsc/cache checks repeated before every launch, and the live environment evidence that remains before deployment can be claimed. +and [operator runbook](../deploy/hosted/README.md), extended by the +[hosted Smol integration](../.commitrail/changes/hosted-smol-project-execution.md), +define the digest-only backend topology, existing native Smol node, one private +endpoint for two typed capabilities, root-derived absolute paths and live health +checks before API/Celery worker release. Deployed operation and full public journey +evidence remain separate. The [Terminal Systems project inputs](../pilots/terminal_systems/README.md) provide an original contributor guide, one illustrative task and five authoring @@ -1716,13 +1730,18 @@ or trial ART effects, full owned VM/relay cleanup and both admission slots reuse That source changes test observation only; the native build remains `b2ce3fa45`. Hosted Smol composition and its scoped AWS proof are tracked separately in [PR #692](https://github.com/Flow-Research/workstream/pull/692). That owned baseline -passed with fixture identity; its initial model-only attempt stopped before -provider effects. No live-provider outcome -or full Terminal Systems guide-to-contribution journey is qualified here. +passed with fixture identity. The scoped model-only continuation at `5ab96251f` +then reused its original prepared effect through the public Operator retry, +received a real OpenRouter passing verdict, and produced one FinalAcceptance and +one ContributionRecord. The pinned-rate ledger settled US$0.016527 with no +outstanding reservation; exact Submission replay and duplicate deliveries added +no effects. The [hosted qualification record](../.commitrail/changes/hosted-smol-project-execution.md#live-hosted-model-continuation) +distinguishes this synthetic, single-rubric proof from substantive grading quality +and the full Terminal Systems guide-to-contribution journey. Model-enabled checks require exact approved project profiles before binding; project publication and complete live-journey qualification remain separate. Registration, corrected policy binding and approval must precede activation; -no approved binding, activated guide or contribution outcome is established. +the real Terminal Systems project remains unactivated by these engineering proofs. The [Smol node lifecycle](../.commitrail/initiatives/WS-EXEC-001/WS-EXEC-001-01.md) adds production engine support through the existing typed adapter, independently diff --git a/experiments/pilot04_external_checker_runtime/public_fault_files.py b/experiments/pilot04_external_checker_runtime/public_fault_files.py index cabe92cf1..8dd296b32 100644 --- a/experiments/pilot04_external_checker_runtime/public_fault_files.py +++ b/experiments/pilot04_external_checker_runtime/public_fault_files.py @@ -7,29 +7,48 @@ from uuid import uuid4 -def private_path(path, mode, *, directory=False): +def private_path(path, mode, *, directory=False, uid=None, gid=None, trusted_uids=()): """Reject aliasing, foreign ownership and any widening of the private mode.""" path = Path(path) if not path.is_absolute() or path.resolve() != path: raise ValueError('fault proof path is not absolute and canonical') + uid = os.getuid() if uid is None else uid + gid = os.getgid() if gid is None else gid info = path.lstat() expected = stat.S_ISDIR if directory else stat.S_ISREG - if not expected(info.st_mode) or info.st_uid != os.getuid() or stat.S_IMODE(info.st_mode) != mode: + if (not expected(info.st_mode) or info.st_uid != uid or info.st_gid != gid + or stat.S_IMODE(info.st_mode) != mode): raise ValueError('fault proof path ownership/mode differs') if not directory and info.st_nlink != 1: raise ValueError('fault proof file has another link') + anchor = path if directory else path.parent + metadata = anchor.lstat() + if (not stat.S_ISDIR(metadata.st_mode) or metadata.st_uid != uid + or metadata.st_gid != gid or stat.S_IMODE(metadata.st_mode) != 0o700): + raise ValueError('fault proof private anchor ownership/mode differs') for parent in path.parents: metadata = parent.lstat() - if not stat.S_ISDIR(metadata.st_mode) or metadata.st_mode & 0o022: + sticky_root = (parent in anchor.parents and metadata.st_uid == 0 + and metadata.st_mode & stat.S_ISVTX) + if (not stat.S_ISDIR(metadata.st_mode) + or metadata.st_uid not in {0, uid, os.getuid(), *trusted_uids} + or (metadata.st_mode & 0o022 and not sticky_root)): raise ValueError('fault proof ancestor is writable by another identity') class PrivateControlDirectory: """Keep fixture control reads/writes anchored to the validated directory inode.""" - def __init__(self, root): - private_path(root, 0o700, directory=True) + def __init__(self, root, *, uid=None, gid=None, trusted_uids=()): + self.uid = os.getuid() if uid is None else uid + self.gid = os.getgid() if gid is None else gid + private_path(root, 0o700, directory=True, uid=self.uid, gid=self.gid, + trusted_uids=trusted_uids) + before = Path(root).lstat() self.fd = os.open(root, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW) + if not os.path.samestat(before, os.fstat(self.fd)): + os.close(self.fd) + raise ValueError('control directory identity changed') def close(self): os.close(self.fd) @@ -37,22 +56,32 @@ def close(self): def read(self, name): if Path(name).name != name: raise ValueError('control file is not a basename') - descriptor = os.open(name, os.O_RDONLY | os.O_NOFOLLOW, dir_fd=self.fd) + descriptor = os.open(name, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC, + dir_fd=self.fd) with os.fdopen(descriptor, 'rb') as source: info = os.fstat(source.fileno()) - if (not stat.S_ISREG(info.st_mode) or info.st_uid != os.getuid() + if (not stat.S_ISREG(info.st_mode) or info.st_uid != self.uid or info.st_gid != self.gid or stat.S_IMODE(info.st_mode) != 0o600 or info.st_nlink != 1 or info.st_size > 65536): raise ValueError('control file ownership/mode/size differs') - return json.load(source) + return json.loads(source.read(65537)) def write(self, name, value): if Path(name).name != name: raise ValueError('control file is not a basename') + try: + existing = os.stat(name, dir_fd=self.fd, follow_symlinks=False) + except FileNotFoundError: + existing = None + if existing is not None and (not stat.S_ISREG(existing.st_mode) + or existing.st_uid != self.uid or existing.st_gid != self.gid + or stat.S_IMODE(existing.st_mode) != 0o600 or existing.st_nlink != 1): + raise ValueError('existing control file ownership/mode differs') temporary = uuid4().hex + '.tmp' descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600, dir_fd=self.fd) try: with os.fdopen(descriptor, 'w') as target: + os.fchown(target.fileno(), self.uid, self.gid) json.dump(value, target) target.flush() os.fsync(target.fileno()) diff --git a/experiments/pilot04_external_checker_runtime/public_guide_setup.py b/experiments/pilot04_external_checker_runtime/public_guide_setup.py index dbce8c974..fa23723cb 100644 --- a/experiments/pilot04_external_checker_runtime/public_guide_setup.py +++ b/experiments/pilot04_external_checker_runtime/public_guide_setup.py @@ -46,8 +46,14 @@ async def _public_policy_settings(client, root, token, human_review_required, ca headers={**headers, "Idempotency-Key": str(uuid4())}, json=body) -async def _finance_policy(client, factory, project_id, token_for, compensated, call): - finance, _ = await seed_review_actor(factory, project_id, role="finance_authority") +async def _finance_policy(client, factory, project_id, token_for, compensated, call, attachment=None): + if attachment: + await attachment.grant(client, 'finance', 'finance_authority', project_id) + finance = attachment.actors['finance'] + if compensated: + raise AssertionError('hosted runtime fixture requires unpaid policy') + else: + finance, _ = await seed_review_actor(factory, project_id, role="finance_authority") token = token_for(finance) definitions = [] if compensated: @@ -167,24 +173,38 @@ async def _source_evidence(factory, guide, setup, package, original): "uploaded_sha256": "sha256:" + hashlib.sha256(original).hexdigest()} -async def public_guide_setup(client, factory, human_review_required, call, token_for, *, pre_requirements=1): - original = guide_bytes(pre_requirements) - manager, _ = await seed_review_actor(factory, None, scope="system") +async def public_guide_setup(client, factory, human_review_required, call, token_for, *, + pre_requirements=1, attachment=None): + model_post = attachment.model_post if attachment else None + if model_post and 'project_id' in model_post.state: + raise RuntimeError('this model qualification already owns one project') + original = model_post.guide_bytes() if model_post else guide_bytes(pre_requirements) + if attachment: + manager = attachment.actors['manager'] + else: + manager, _ = await seed_review_actor(factory, None, scope="system") token = token_for(manager) project = await call(client, "POST", "/api/v1/projects", token, 201, headers={"Idempotency-Key": str(uuid4())}, json={"name": "Connected guide journey", "slug": "journey-" + uuid4().hex}) - await seed_review_actor(factory, UUID(project["id"]), actor=manager) - from guide_compilation_e2e import publish_scripted_model_profile - operator, _ = await seed_review_actor(factory, None, role="operator", scope="system") - profile = await publish_scripted_model_profile(client, token_for(operator), project["id"]) - if (profile["spec"]["project_id"] != project["id"] - or profile["spec"]["profile_version"] != 1): - raise AssertionError("scripted setup model profile differs from the new project") + if model_post: + await model_post.bind_project(client, project['id'], attachment) + if attachment: + await attachment.grant(client, 'manager', 'project_manager', project['id']) + else: + await seed_review_actor(factory, UUID(project["id"]), actor=manager) + from guide_compilation_e2e import publish_scripted_model_profile + operator, _ = await seed_review_actor(factory, None, role="operator", scope="system") + profile = await publish_scripted_model_profile(client, token_for(operator), project["id"]) + if (profile["spec"]["project_id"] != project["id"] + or profile["spec"]["profile_version"] != 1): + raise AssertionError("scripted setup model profile differs from the new project") root = f"/api/v1/projects/{project['id']}/guides" body = {"version": "initial", "change_summary": "Original sample worker guide", "task_examples": TASK_EXAMPLES, "documents": [{"label": "guide.md", "media_type": "text/markdown"}]} + if attachment: + body['documents'].append({'label': 'workstream-rules.json', 'media_type': 'text/markdown'}) headers = {"Idempotency-Key": str(uuid4())} guide = await call(client, "POST", root, token, 201, headers=headers, json=body) replay = await call(client, "POST", root, token, 201, headers=headers, json=body) @@ -196,12 +216,18 @@ async def public_guide_setup(client, factory, human_review_required, call, token await call(client, "POST", upload, token, 202, headers={"Idempotency-Key": str(uuid4()), "Content-Type": "text/markdown"}, content=original) + if attachment: + from experiments.pilot04_external_checker_runtime.public_hosted_attachment import manager_rules + await call(client, 'POST', root + f"/documents/{guide['documents'][1]['document_id']}/content", + token, 202, headers={'Idempotency-Key': str(uuid4()), 'Content-Type': 'text/markdown'}, + content=(model_post.rules(original, attachment.capabilities, pre_requirements) if model_post else + manager_rules(original, attachment.capabilities, pre_requirements))) setup = await wait_for_guide_read(client, root + "/setup-runs/latest", token, lambda v: v["status"] == "policy_draft_ready", call) compilation_root = root + f"/compilations/{setup['finalized_compilation_id']}" initial = await call(client, "GET", compilation_root + "/proposal", token, 200) initial_setup = setup - if human_review_required: + if human_review_required and attachment is None: correction = await _correct_public_proposal(client, compilation_root, initial, token, call) setup = await wait_for_guide_read(client, root + "/setup-runs/latest", token, lambda v: v["status"] == "policy_draft_ready" @@ -219,7 +245,7 @@ async def public_guide_setup(client, factory, human_review_required, call, token "acknowledged_warning_hashes": initial["warning_hashes"]}) compilation_root = root + f"/compilations/{setup['finalized_compilation_id']}" await _finance_policy(client, factory, UUID(project["id"]), token_for, - not human_review_required, call) + not human_review_required and attachment is None, call, attachment) post = await _approve_policies(client, compilation_root, token, call) context = post["activation_context"] activation = {"target": post["target"], **{field: context[field] for field in ( diff --git a/experiments/pilot04_external_checker_runtime/public_hosted_attachment.py b/experiments/pilot04_external_checker_runtime/public_hosted_attachment.py new file mode 100644 index 000000000..f02f472fe --- /dev/null +++ b/experiments/pilot04_external_checker_runtime/public_hosted_attachment.py @@ -0,0 +1,580 @@ +"""Attach the public runtime fixture to an already running hosted deployment. + +This module never launches an API, worker, Beat, issuer or model provider. The +RS256 issuer is an explicitly simulated identity fixture. Manager-authored guide +rules prove runtime wiring, not a setup-agent rehearsal or a real pilot project. + +Invoke the existing run_public_journey_probe module with --hosted-attachment, +--database-url and --broker-url. The private JSON attachment names origin, +release_env, identity_directory, node_root, fixture_ca_sha256 and capabilities +({pre_submit,post_submit}, each with capability_id/capability_version). Baseline +setup registers those exact immutable capabilities through the public Operator +registry after its initial Operator grant. The fixture-only derived release images must already contain the public CA +appended to their ordinary certifi bundle and pass the canonical image verifier. + +The driver environment supplies the same isolated database/broker settings for +the established bootstrap/lifecycle setup tools and duplicate broker delivery. +It does not replace any running service's environment. Original-lease expiry +uses WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS (default 3400s), +including the real wait. Existing Settings require ZIP inspection < preparation +and preparation + cleanup margin < scratch TTL; AWS operation/complete-read and +terminal persistence bounds remain independently enforced by hosted preflight. +""" + +from __future__ import annotations + +import asyncio +from dataclasses import dataclass, field +from functools import partial +import hashlib +import importlib.util +import json +import os +from pathlib import Path +import socket +import subprocess +import sys +import time +from urllib.parse import urlparse +from uuid import UUID, uuid4 + +ROOT = Path(__file__).resolve().parents[2] + + +def require(condition, message): + if not condition: + raise RuntimeError(message) + + +def _command(arguments, *, timeout=30, environment=None, combined=False): + """Keep secret-bearing child output out of exception diagnostics.""" + result = subprocess.run(arguments, capture_output=True, timeout=timeout, + env=environment, check=False, cwd=ROOT) + require(result.returncode == 0, "hosted proof management command failed") + output = result.stdout + result.stderr if combined else result.stdout + require(len(output) <= 4 * 1024 * 1024, "hosted proof command output exceeded bound") + return output + + +def deliver_duplicate(task, hostname, event_id, project_id): + """Observe one real broker delivery without changing production logging.""" + from kombu import Consumer + + identifier = str(uuid4()) + ready = False + received = False + completed = False + + def observe(event): + nonlocal ready, received, completed + if event['type'] == 'worker-heartbeat' and event.get('hostname') == hostname: + ready = True + return + if event.get('uuid') != identifier or event['type'] not in { + 'task-received', 'task-succeeded', 'task-failed', 'task-rejected', + 'task-revoked', 'task-retried', + }: + return + require(event.get('hostname') == hostname, 'duplicate ran on another worker') + if event['type'] == 'task-received': + require(event.get('name') == task.name, 'duplicate task name differs') + received = True + elif event['type'] == 'task-succeeded': + require(received and event.get('result') == "{'status': 'delivery_unavailable'}", + 'duplicate terminal result differs') + completed = True + elif event['type'] in {'task-failed', 'task-rejected', 'task-revoked', 'task-retried'}: + raise RuntimeError('duplicate task did not succeed') + + def control(command): + replies = task.app.control.broadcast(command, destination=[hostname], reply=True, + timeout=5, connection=control_connection) + require(len(replies) == 1 and set(replies[0]) == {hostname}, + 'owned worker event-control acknowledgement differs') + return replies[0][hostname] + + restore = False + options = {'connect_timeout': 5, 'transport_options': { + 'max_retries': 0, 'socket_connect_timeout': 5, 'socket_timeout': 5, + 'retry_on_timeout': False, + }} + # Redis's event subscription and control replies use distinct consumers. + # Sharing their connection can lose the restoration acknowledgement. + with task.app.connection_for_read(**options) as connection, \ + task.app.connection_for_write(**options) as control_connection: + connection.ensure_connection(max_retries=0, timeout=5) + control_connection.ensure_connection(max_retries=0, timeout=5) + receiver = task.app.events.Receiver( + connection, handlers={'*': observe}, node_id='ws-proof-' + identifier, + routing_key='#', queue_ttl=30, queue_expires=180, + queue_exclusive=True, queue_durable=False, accept={'json'}, + ) + try: + # Redis installs its fanout subscription on the first drain, not + # basic_consume. A real heartbeat proves this queue is listening. + with receiver.get_consumers(partial(Consumer, receiver.channel), receiver.channel)[0]: + deadline = time.monotonic() + 15 + while not ready: + require(time.monotonic() < deadline, 'owned worker event subscription is not ready') + try: + connection.drain_events(timeout=max(.01, min(1, deadline - time.monotonic()))) + except socket.timeout: + pass + reply = control('enable_events') + require(reply in ({'ok': 'task events enabled'}, {'ok': 'task events already enabled'}), + 'owned worker did not enable task events') + restore = reply == {'ok': 'task events enabled'} + task.apply_async(args=(str(event_id), str(project_id)), task_id=identifier, + ignore_result=True, retry=False, + producer=task.app.amqp.Producer(control_connection)) + deadline = time.monotonic() + 120 + while not completed: + require(time.monotonic() < deadline, 'exact duplicate completion was not observed') + try: + connection.drain_events(timeout=max(.01, min(1, deadline - time.monotonic()))) + except socket.timeout: + pass + finally: + try: + if restore: + require(control('disable_events') == {'ok': 'task events disabled'}, + 'owned worker task events were not restored') + finally: + receiver.queue(connection.default_channel).delete(if_unused=False, if_empty=False) + return {'task_id': identifier, 'result': {'status': 'delivery_unavailable'}} + + +def validate_container(service, rendered, container, image, secrets, networks): + """Compare actual container custody with the validated production render.""" + actual = container['Config'] + host = container['HostConfig'] + require(container['State']['Running'] and container['State'].get('Health', {}).get('Status') == 'healthy', + "hosted service is not healthy: " + service) + require(actual['Image'] == rendered['image'] and container['Image'] == image['Id'], + "hosted service image differs: " + service) + require(actual['Cmd'] == rendered['command'] + and actual['Entrypoint'] == image['Config']['Entrypoint'] + and actual['User'] == rendered['user'], "hosted service executable differs: " + service) + require(host['ReadonlyRootfs'] is True + and host['Privileged'] is False + and set(host.get('CapDrop') or []) == {'ALL'} + and {item.removeprefix('CAP_') for item in host.get('CapAdd') or []} + == set(rendered.get('cap_add') or []) + and 'no-new-privileges:true' in (host.get('SecurityOpt') or []), + "hosted service isolation differs: " + service) + environment = dict(item.split('=', 1) for item in actual['Env']) + expected = {key: str(value) for key, value in rendered['environment'].items()} + require(all(environment.get(key) == value for key, value in expected.items()), + "hosted service environment differs: " + service) + image_environment = dict(item.split('=', 1) for item in image['Config'].get('Env', [])) + require(environment == image_environment | expected, "hosted service has extra environment: " + service) + require(not environment.get('WORKSTREAM_FLOW_AUTH_LOCAL_HMAC_SECRET') + and environment.get('WORKSTREAM_CELERY_TASK_ALWAYS_EAGER', 'false') == 'false' + and not any(key.startswith('WORKSTREAM_PROBE_') for key in environment), + "hosted service contains local proof substitution") + expected_mounts = {(item['source'], item['target'], not item.get('read_only', False)) + for item in rendered.get('volumes', []) if item['type'] == 'bind'} + for item in rendered.get('secrets', []): + expected_mounts.add((secrets[item['source']]['file'], '/run/secrets/' + item['target'], False)) + mounts = {(item['Source'], item['Destination'], item['RW']) + for item in container['Mounts'] if item['Type'] == 'bind'} + require(mounts == expected_mounts + and all(item['Type'] in {'bind', 'tmpfs'} for item in container['Mounts']), + "hosted service mounts differ: " + service) + expected_tmpfs = {} + for specification in rendered.get('tmpfs', []): + target, separator, options = specification.partition(':') + require(separator and target not in expected_tmpfs, + 'rendered tmpfs is ambiguous: ' + service) + expected_tmpfs[target] = set(options.split(',')) + actual_tmpfs = {target: set(options.split(',')) + for target, options in (host.get('Tmpfs') or {}).items()} + # Docker reports configured tmpfs mounts in HostConfig.Tmpfs, usually omitting + # them from Mounts. If it also lists one there, it must name the same target. + require(actual_tmpfs == expected_tmpfs + and all(item['Destination'] in expected_tmpfs + and item['RW'] == ('ro' not in expected_tmpfs[item['Destination']]) + for item in container['Mounts'] if item['Type'] == 'tmpfs'), + 'hosted service tmpfs differs: ' + service) + if rendered.get('network_mode') == 'none': + require(host.get('NetworkMode') == 'none', 'hosted service network mode differs: ' + service) + expected_networks = {'none'} + else: + expected_networks = {networks[name]['name'] for name in rendered['networks']} + require(set(container['NetworkSettings']['Networks']) == expected_networks, + 'hosted service networks differ: ' + service) + for setting, field_name, multiplier in ( + ('mem_limit', 'Memory', 1), ('memswap_limit', 'MemorySwap', 1), + ('pids_limit', 'PidsLimit', 1), ('cpus', 'NanoCpus', 1_000_000_000), + ): + if setting in rendered: + require(host.get(field_name) == int(float(rendered[setting]) * multiplier), + 'hosted service resource limit differs: ' + service) + expected_ports = {} + for port in rendered.get('ports', []): + key = str(port['target']) + '/' + port.get('protocol', 'tcp') + expected_ports.setdefault(key, []).append({'HostIp': port.get('host_ip', ''), + 'HostPort': str(port['published'])}) + require((host.get('PortBindings') or {}) == expected_ports, + 'hosted service published ports differ: ' + service) + + +@dataclass +class HostedAttachment: + origin: str + release_env: Path + identity_directory: Path + node_root: Path + capabilities: dict + fixture_ca_sha256: str + actors: dict = field(default_factory=dict) + values: dict = field(default_factory=dict) + custody: dict = field(default_factory=dict) + docker_environment: dict = field(default_factory=dict) + native_binding: dict = field(default_factory=dict) + model_post: object | None = None + + @classmethod + def read(cls, path): + from experiments.hosted_drill_identity.fixture import read_private + + value = json.loads(read_private(path)) + require(set(value) == {'origin', 'release_env', 'identity_directory', 'node_root', + 'capabilities', 'fixture_ca_sha256'}, "hosted attachment fields differ") + origin = urlparse(value['origin']) + require(origin.scheme == 'http' and origin.hostname in {'127.0.0.1', 'localhost'} + and origin.port and not origin.username and not origin.password + and origin.path == '' and not origin.query and not origin.fragment, + "hosted proof must attach to its loopback published API") + for key in ('release_env', 'identity_directory', 'node_root'): + path_value = Path(value[key]) + require(path_value.is_absolute() and path_value.resolve() == path_value, + "hosted attachment path must be canonical") + value[key] = path_value + require(set(value['capabilities']) == {'pre_submit', 'post_submit'}, "hosted capability stages differ") + for stage, capability in value['capabilities'].items(): + require(set(capability) == {'capability_id', 'capability_version'} + and all(isinstance(item, str) and item for item in capability.values()), + "hosted capability identity is incomplete: " + stage) + return cls(**value) + + def inspect(self): + """Reuse hosted validation; inspect exact running services, never replace them.""" + from experiments.hosted_drill_identity.fixture import ISSUER, AUDIENCE, read_private + from experiments.pilot04_external_checker_runtime.public_node_faults import runtime_control_root + + name = '_public_hosted_preflight' + spec = importlib.util.spec_from_file_location(name, ROOT / 'deploy/hosted/preflight.py') + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + values = module.parse_environment(self.release_env) + module.validate_environment(values) + rendered = module.render_compose(self.release_env, values) + module.validate_compose(rendered, values) + config = module.load_checker_config(Path(values['WORKSTREAM_CHECKER_CONFIG_FILE'])) + module.validate_checker_config(config, values) + module.validate_private_host_inputs(values) + module.validate_smol_runtime(config, values) + module.validate_trusted_intake(values) + require(values['WORKSTREAM_PROJECT_AGENT_RUNTIME'] == 'manager_authored', + "hosted runtime fixture requires explicit manager-authored rules") + require(values['WORKSTREAM_TOKEN_ISSUER'] == ISSUER + and values['WORKSTREAM_TOKEN_AUDIENCE'] == AUDIENCE + and values['WORKSTREAM_TOKEN_JWKS_URL'] == ISSUER + '/jwks.json' + and values['WORKSTREAM_TOKEN_ALGORITHMS'] == 'RS256', + "hosted verifier does not match the simulated RS256 fixture") + require(self.fixture_ca_sha256 == 'sha256:' + hashlib.sha256( + read_private(self.identity_directory / 'ca.pem')).hexdigest(), "fixture public CA differs") + port = urlparse(self.origin).port + require(port == int(values.get('WORKSTREAM_API_HOST_PORT', '8000')), "published API port differs") + environment = module.compose_environment(values) + services = ['api', 'worker', 'beat', 'guide-document-parser'] + if values['WORKSTREAM_MODEL_PROXY_ENABLED'] == 'true': + services.append('model-proxy') + observed = {} + for service in services: + ids = _command(['docker', 'ps', '--filter', 'label=com.docker.compose.project=' + + values['COMPOSE_PROJECT_NAME'], '--filter', + 'label=com.docker.compose.service=' + service, '--format', '{{.ID}}'], + environment=environment).decode().splitlines() + require(len(ids) == 1, "hosted service is absent or duplicated: " + service) + container = json.loads(_command(['docker', 'inspect', ids[0]], environment=environment))[0] + image = json.loads(_command(['docker', 'image', 'inspect', + rendered['services'][service]['image']], environment=environment))[0] + validate_container(service, rendered['services'][service], container, image, + rendered['secrets'], rendered['networks']) + observed[service] = {'container_id': container['Id'], 'image_id': container['Image'], + 'image_reference': container['Config']['Image']} + require((self.node_root / 'supervisor-ready').exists(), "owned node supervisor is absent") + binding = json.loads((self.node_root / 'node-binding.json').read_text()) + require(binding['source_tree'] == _command(['git', 'rev-parse', 'HEAD']).decode().strip(), + "native proof source differs") + wrapper = runtime_control_root(self.node_root) / 'runtime-wrapper' + original = json.loads(json.dumps(config)) + require(config['smol']['binary'] == str(wrapper) + and config['smol']['binary_sha256'] == binding['wrapper_sha256'] + == 'sha256:' + hashlib.sha256(wrapper.read_bytes()).hexdigest(), "native trace wrapper differs") + original['smol']['binary'] = binding['node_config']['smol']['binary'] + original['smol']['binary_sha256'] = binding['node_config']['smol']['binary_sha256'] + require(original == binding['node_config'], "native proof configuration differs") + self.values, self.custody, self.docker_environment = values, observed, environment + self.native_binding = binding + return {'containers': observed, 'native': binding, + 'fixture_ca_sha256': self.fixture_ca_sha256, + 'identity_scope': 'simulated RS256 issuer; fixture-only derived image trust'} + + def token_for_label(self, label): + from experiments.hosted_drill_identity.fixture import mint, read_private + + path = self.identity_directory / ('token-' + uuid4().hex) + try: + mint(self.identity_directory, label, path) + return read_private(path).decode() + finally: + path.unlink(missing_ok=True) + + def token(self, actor): + labels = [label for label, facts in self.actors.items() + if facts.actor_profile_id == actor.actor_profile_id] + require(len(labels) == 1, "actor is outside the hosted fixture") + return self.token_for_label(labels[0]) + + async def grant(self, client, label, role, project_id=None): + from experiments.pilot04_external_checker_runtime.public_probe_support import _json + + await _json(client, 'POST', '/api/v1/admin-role-grants', self.token_for_label('access-admin'), 201, + headers={'Idempotency-Key': str(uuid4())}, json={ + 'target_actor_profile_id': str(self.actors[label].actor_profile_id), + 'role': role, 'scope_type': 'project' if project_id else 'system', + 'scope_project_id': str(project_id) if project_id else None, + 'reason': 'Assign authority for this isolated hosted runtime fixture.'}) + + async def admit_actors(self, client, factory): + from sqlalchemy import select + from app.modules.actors.models import ActorIdentityLink + from app.modules.authorization.api import ActorIdentityFacts, ActorKind + from experiments.hosted_drill_identity.fixture import ISSUER, IDENTITIES + from experiments.pilot04_external_checker_runtime.public_probe_support import _json + + for label in IDENTITIES: + admitted = await _json(client, 'GET', '/api/v1/actors/me', self.token_for_label(label), 200) + async with factory() as session: + link = (await session.scalars(select(ActorIdentityLink).where( + ActorIdentityLink.actor_profile_id == admitted['actor_profile_id'], + ActorIdentityLink.issuer == ISSUER, ActorIdentityLink.status == 'active'))).one() + self.actors[label] = ActorIdentityFacts(UUID(admitted['actor_profile_id']), UUID(link.id), ActorKind.HUMAN) + + async def prepare_actors(self, client, factory): + await self.admit_actors(client, factory) + # Deployment setup uses the established local trust-root operation. Every + # subsequent grant, guide, task and submission uses the running public API. + await asyncio.to_thread(self.setup_command, 'bootstrap_access_administrator.py', + '--actor-profile-id', str(self.actors['access-admin'].actor_profile_id), '--execute') + await self.provision_services(client) + await self.grant(client, 'operator', 'operator') + await self.register_baseline_capabilities(client) + await self.grant(client, 'manager', 'project_manager') + await self.transition('shadow') + + async def register_baseline_capabilities(self, client): + """Register only the two requested fixture identities under public Operator authority.""" + from app.modules.checkers.api.external import ExternalCheckerRegistryEntry + from experiments.pilot04_external_checker_runtime.public_probe_support import _json + from experiments.pilot04_external_checker_runtime.run_selected_end_to_end_probe import _locked_spec + + images = [item for item in self.native_binding.get('node_config', {}).get('cache', []) + if item['repository'] == 'workstream-test/public-journey'] + require(len(images) == 1, 'validated generic public image is absent or ambiguous') + specs = [] + for phase in ('pre_submit', 'post_submit'): + base = _locked_spec(phase, images[0]) + values = base.model_dump(mode='json') | self.capabilities[phase] + values['image_digest'] = images[0]['platform_manifest_digest'] + spec = type(base).model_validate(values) + require(spec.phase == phase, 'baseline fixture registry phase differs') + specs.append(spec) + require(len({(spec.capability_id, spec.capability_version) for spec in specs}) == 2, + 'baseline PRE and POST capability identities must be distinct') + for spec in specs: + operation_id = str(uuid4()) + value = await _json(client, 'POST', '/api/v1/operations/external-checker-registry', + self.token_for_label('operator'), 201, headers={'Idempotency-Key': operation_id}, + json=spec.model_dump(mode='json')) + registered = ExternalCheckerRegistryEntry.model_validate_json(json.dumps(value)) + require(registered.entry_digest == spec.spec_digest + and str(registered.registration_operation_id) == operation_id + and registered.registered_by_actor_profile_id == self.actors['operator'].actor_profile_id, + 'public baseline registration substituted specification or authority') + + async def provision_services(self, client): + from provision_local_service_actors import JOURNEY_SERVICE_IDENTITIES, idempotency_key, subject_for + from experiments.pilot04_external_checker_runtime.public_probe_support import _json + + prefix = 'hosted-' + str(self.actors['access-admin'].actor_profile_id) + for identity in JOURNEY_SERVICE_IDENTITIES: + subject = subject_for(identity, prefix) + await _json(client, 'POST', '/api/v1/service-actors', self.token_for_label('access-admin'), 201, + headers={'Idempotency-Key': idempotency_key(identity, subject)}, + json={'service_identity': identity, 'subject': subject, + 'reason': 'Provision this isolated hosted runtime fixture.'}) + + def setup_command(self, script, *arguments): + require(script in {'bootstrap_access_administrator.py', 'transition_review_lifecycle.py'}, + "unsupported hosted setup operation") + _command([sys.executable, str(ROOT / 'backend/scripts' / script), *arguments], timeout=120) + + async def transition(self, phase): + require(phase in {'shadow', 'live'}, "unsupported hosted activation phase") + await asyncio.to_thread(self.setup_command, 'transition_review_lifecycle.py', '--actor-profile-id', + str(self.actors['operator'].actor_profile_id), '--phase', phase, + '--reason', 'Activate this isolated hosted runtime fixture.') + + def stop_worker(self): + _command([sys.executable, str(ROOT / 'deploy/hosted/compose.py'), '--env-file', + str(self.release_env), 'stop', 'worker'], timeout=7380) + + async def start_worker(self): + old = self.custody['worker']['container_id'] + await asyncio.to_thread(_command, [sys.executable, str(ROOT / 'deploy/hosted/compose.py'), + '--env-file', str(self.release_env), 'start', 'worker'], timeout=150) + # Compose may restart the same container. Its worker process is a new + # generation; OUTBOX claim custody proves consumption after this boundary. + self.inspect() + return {'previous_container': old, 'container': self.custody['worker']['container_id']} + + async def deliver_duplicate(self, event_id, project_id): + from app.workers.outbox import deliver_event + + hostname = (await asyncio.to_thread( + _command, ['docker', 'inspect', '--format', '{{.Config.Hostname}}', + self.custody['worker']['container_id']], environment=self.docker_environment, + )).decode().strip() + require(hostname and len(hostname) <= 253 + and all(c.isascii() and (c.isalnum() or c in '.-_') for c in hostname), + 'owned worker hostname is invalid') + return await asyncio.to_thread(deliver_duplicate, deliver_event, 'worker@' + hostname, + event_id, project_id) + + +def manager_rules(original, capabilities, count): + """Author explicit synthetic marker rules; do not install a recorded runtime.""" + from experiments.pilot04_external_checker_runtime.guide_runtime import _recorded_specification + + refs = [{'document': 'sha256:' + hashlib.sha256(original).hexdigest(), 'start_page': 1, 'end_page': 1}] + requirements, bindings = [], {'pre_submit': [], 'post_submit': []} + for stage in bindings: + total = count if stage == 'pre_submit' else 1 + for index in range(total): + key = 'project.' + stage + (f'.{index + 1:02d}' if total == 2 else '') + requirements.append({'requirement_id': key, 'statement': 'Apply the registered fixture marker rule.', + 'disposition': 'supported_' + stage, 'evidence_refs': refs, + 'classification_rationale': 'Explicit manager-authored runtime fixture.', + 'check_specification': _recorded_specification(stage, index=index).model_dump(mode='json')}) + bindings[stage].append({'requirement_id': key, **capabilities[stage], 'stage': stage}) + return json.dumps({'format': 'workstream.project_guide_rules.v1', 'status': 'draft_ready', + 'findings': [{'severity': 'info', 'code': 'probe.runtime_fixture', + 'message': 'Synthetic hosted runtime proof, not setup-agent inference.', + 'evidence_refs': refs}], + 'submission_artifact_policy': {'maximum_file_size_bytes': 1_000_000, + 'maximum_package_size_bytes': 5_000_000, 'required_artifacts': ['task.toml'], + 'required_evidence': ['results'], 'attestation_terms': ['rights_confirmed']}, + 'requirements': requirements, 'pre_submit_bindings': bindings['pre_submit'], + 'post_submit_bindings': bindings['post_submit'], 'capability_suggestions': [], + 'setup_notes': []}, sort_keys=True, separators=(',', ':')).encode() + + +async def run(attachment, database_url, broker_url, model_post_path=None): + from experiments.pilot04_external_checker_runtime.public_hosted_model_post import ( + ModelPostQualification, validate_runtime_window, + ) + + if model_post_path: + from app.core.config import get_settings + + attachment.inspect() + validate_runtime_window(attachment.values, get_settings()) + qualification = ModelPostQualification.read_and_claim(model_post_path, attachment.node_root) if model_post_path else None + try: + return await _run(attachment, database_url, broker_url, qualification) + finally: + if qualification: + qualification.close() + + +async def _run(attachment, database_url, broker_url, qualification): + import httpx + from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine + from api_contract_e2e import assert_isolated_database_url + from experiments.pilot04_external_checker_runtime.public_intake_trials import public_intake_trials + from experiments.pilot04_external_checker_runtime.public_runtime_trials import public_runtime_trials + from experiments.pilot04_external_checker_runtime.run_public_journey_probe import ( + _one_mode, _scan_audit_count, _celery_config_mismatches, + ) + from app.core.config import get_settings + + assert_isolated_database_url(database_url) + require(os.environ.get('WORKSTREAM_DATABASE_URL') == database_url + and os.environ.get('WORKSTREAM_CELERY_BROKER_URL') == broker_url, + 'hosted proof setup must use the explicit isolated DB and broker') + require(not os.environ.get('WORKSTREAM_FLOW_AUTH_LOCAL_HMAC_SECRET'), 'hosted proof refuses HMAC') + require(get_settings().celery_task_always_eager is False + and not _celery_config_mismatches(get_settings()), 'hosted proof broker publisher configuration differs') + custody = attachment.inspect() + engine = create_async_engine(database_url) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with httpx.AsyncClient(base_url=attachment.origin, timeout=180, trust_env=False) as client: + if qualification: + from experiments.pilot04_external_checker_runtime.public_hosted_model_post import run_model_post + attachment.model_post = qualification + await attachment.admit_actors(client, factory) + model_result = await run_model_post(attachment, client, factory, qualification) + from experiments.pilot04_external_checker_runtime.public_node_faults import node_command + cleanup = await node_command(attachment.node_root, 'drained') + require(cleanup['foreign_preserved'] and not any(cleanup['resources'].values()), + 'hosted model proof retained runtime resources or changed foreign custody') + attachment.inspect() + return {'scope': 'public hosted model POST engineering fixture', 'custody': custody, + 'provider_calls_by_driver': 0, 'model_post': model_result, 'cleanup': cleanup} + await attachment.prepare_actors(client, factory) + scans_before = await _scan_audit_count(factory) + deadline = time.monotonic() + 300 + while await _scan_audit_count(factory) <= scans_before: + require(time.monotonic() < deadline, 'hosted Beat did not dispatch an authorized artifact scan') + await asyncio.sleep(2) + intake = await public_intake_trials(client, factory, attachment, None, attachment.node_root) + runtime = await public_runtime_trials(client, factory, attachment, attachment.node_root) + human = await _one_mode(client, factory, attachment, human_review_required=True, + operator_id=attachment.actors['operator'].actor_profile_id) + automatic = await _one_mode(client, factory, attachment, human_review_required=False, + operator_id=attachment.actors['operator'].actor_profile_id, + stop_worker=attachment.stop_worker, start_worker=attachment.start_worker, + fault_root=attachment.node_root, + operator_token=attachment.token_for_label('operator')) + from experiments.pilot04_external_checker_runtime.public_node_faults import node_command + cleanup = await node_command(attachment.node_root, 'drained') + require(cleanup['foreign_preserved'] and not any(cleanup['resources'].values()), + 'hosted journey retained owned runtime resources or changed the foreign control') + attachment.inspect() + return {'scope': 'hosted API/Celery/Beat runtime fixture', 'custody': custody, + 'guide_runtime': 'manager_authored', 'setup_agent_rehearsal': False, + 'model_enabled_post_qualification': False, + 'scheduled_artifact_scans': True, + 'provider_calls_by_driver': 0, 'compensation_policy': 'unpaid', + 'intake_trials': intake, 'runtime_trials': runtime, 'human': human, + 'automatic': automatic, 'cleanup': cleanup} + except BaseException as error: + if qualification: + fields = {'phase': 'stopped', 'stopped_after': qualification.state['phase'], + 'error_type': type(error).__name__, 'automatic_paid_retry': False} + if 'project_id' in qualification.state: + from experiments.pilot04_external_checker_runtime.public_hosted_model_post import account_snapshot + try: + fields['ledger'] = await account_snapshot(factory, qualification.state['project_id']) + except Exception: + fields['ledger_observation_unavailable'] = True + qualification.checkpoint(**fields) + raise + finally: + await engine.dispose() diff --git a/experiments/pilot04_external_checker_runtime/public_hosted_model_post.py b/experiments/pilot04_external_checker_runtime/public_hosted_model_post.py new file mode 100644 index 000000000..32cd56c4f --- /dev/null +++ b/experiments/pilot04_external_checker_runtime/public_hosted_model_post.py @@ -0,0 +1,594 @@ +"""One public model POST engineering proof, through the actual hosted workers. + +The operator supplies a private JSON file with image_reference, +image_config_digest, image_archive_sha256, proof_state and +historical_uncertain_reference. The existing hosted attachment supplies service +custody and simulated RS256 actors. Run the ordinary hosted journey first to +bootstrap actors/services. An existing proof_state always stops a new run: inspect +the retained project/account instead of creating another paid project. This driver +never calls the proxy or provider directly and never retries an ambiguous effect. +The fixed claim path is node_root/model-post-qualification.json; preserve it and +the same lifetime account for any later full-trial stage. +""" + +from __future__ import annotations + +from dataclasses import dataclass +import hashlib +from io import BytesIO +import json +import os +from pathlib import Path +import stat +import time +from uuid import UUID, uuid4 +from zipfile import ZipFile, ZipInfo + +from experiments.pilot04_external_checker_runtime.public_hosted_attachment import ROOT, require + +CHECK = 'solver_brief_and_extension_disclosure' +CEILING = 10_000_000_000 +MODEL = 'openai/gpt-5.6-terra' +MODEL_CONTEXT = ROOT / 'pilots/terminal_systems/checkers/post/model' +POST_IMAGE = 'workstream/terminal-systems-post@sha256:94a0a10e81c67deb6d433312d2c40fbca52ec7d27ccc0f714979cf51946794be' +POST_CONFIG = 'sha256:d46fea222c9a91c3cb1425a6c00e73b2bacf9756c6226017db8773cefe6003f7' +POST_ARCHIVE = 'sha256:ff87ecf15599f504f461dbf28868ad98206796c57a733de353747f049b7b80f8' +POST_CONTEXT_TREE = '82d968cf2d919e8ccd938d3780745b2b32bce314' + + +def canonical(value): + return json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False).encode() + + +def digest(value): + return 'sha256:' + hashlib.sha256(value).hexdigest() + + +def specification(): + """Use the actual project handoff and bundled rubric, without copying grading.""" + from app.interfaces.project_agents import CheckSpecification + + handoff = json.loads((MODEL_CONTEXT.parent.parent / 'setup-handoff.json').read_text()) + bundled = json.loads((MODEL_CONTEXT / 'rubrics.json').read_text()) + original = next(item for item in handoff['requirements'] if item['requirement_id'] == CHECK) + rubric = next(item for item in bundled['requirements'] if item['requirement_id'] == CHECK) + value = original['check_specification'] + require(all(value[key] == expected for key, expected in rubric['check_specification'].items()), + 'model image rubric differs from retained project handoff') + value['runtime_dependencies'] = [{**dependency, 'provisioning': 'model_proxy'} + for dependency in value['runtime_dependencies']] + return CheckSpecification.model_validate(value), original['statement'] + + +def profile_spec(project_id, pricing_id): + from app.modules.model_usage.api import ProjectModelProfileSpec + + pricing = {'provider': 'openrouter', 'api': 'chat_completions', 'model': MODEL, + 'uncached_input_nanodollars_per_token': 5000, + 'cached_input_nanodollars_per_token': 400, + 'output_nanodollars_per_token': 18000} + value = ProjectModelProfileSpec.model_validate({ + 'project_id': project_id, 'profile_version': 1, 'provider': 'openrouter', + 'api': 'chat_completions', 'model_snapshot': MODEL, + 'settings': {'reasoning_effort': 'low', 'text_verbosity': 'low'}, + 'maximum_request_bytes': 262144, 'maximum_output_bytes': 32768, + 'maximum_response_bytes': 200704, 'maximum_input_tokens': 1050000, + 'maximum_output_tokens': 4096, 'maximum_calls': 1, + 'maximum_total_request_bytes': 262144, 'maximum_total_response_bytes': 200704, + 'maximum_total_input_tokens': 1050000, 'maximum_total_output_tokens': 4096, + 'allowed_tools': (), 'maximum_tool_calls': 0, 'maximum_tool_output_bytes': 0, + 'deadline_ms': 180000, + 'pricing': {'pricing_policy_id': pricing_id, 'pricing_policy_version': 1, + 'pricing_policy_digest': digest(canonical(pricing)), + **{key: pricing[key] for key in pricing if key.endswith('_per_token')}}, + }) + require(value.maximum_reservation_nanodollars == 5_323_728_000 + and value.maximum_reservation_nanodollars <= CEILING, + 'model qualification reservation exceeds its authorization') + return value + + +def registry_spec(image_digest): + from app.modules.checkers.api.external import ExternalCheckerRegistrySpec + + def schema(identifier, document): + return {'schema_id': identifier, 'schema_version': 'v1', 'document': document, + 'schema_sha256': digest(canonical(document))} + + properties = {'check': {'const': CHECK}, 'model_profile_id': {'type': 'string'}, + 'model_profile_version': {'type': 'integer'}, 'model_profile_digest': {'type': 'string'}} + return ExternalCheckerRegistrySpec.model_validate({ + 'capability_id': 'terminal_systems.post.model', 'capability_version': 'hosted-v1', + 'phase': 'post_submit', 'image_digest': image_digest, + 'configuration_schema': schema('terminal_systems.post.hosted_configuration', { + 'type': 'object', 'additionalProperties': False, + 'properties': properties, 'required': sorted(properties)}), + 'input_schema': schema('external_checker_post_submit_input', {'type': 'object'}), + 'output_schema': schema('external_checker_result', {'type': 'object'}), + 'resources': {'cpu_millis': 1000, 'memory_bytes': 256 * 1024 * 1024, + 'deadline_ms': 600000, 'maximum_output_bytes': 32768}, + }) + + +def fixture_archive(): + """A disclosed tiny authored task; the production rubric judges it for real.""" + files = { + 'task.toml': b"[task]\nname='hosted-model-engineering-proof'\n", + 'instruction.md': ( + '# One-line copy engineering fixture\n\n' + 'This is a synthetic hosted model transport qualification, not a real pilot task. ' + 'The starting workspace/seed.txt contains the example line qualified-result. ' + 'Implement a repair that copies its exact bytes to workspace/result.txt without changing seed.txt. ' + 'Any implementation with this behavior is allowed. The only supported commands are ' + '`sh solution/apply.sh` (the supplied reference) and `sh tests/run.sh` (visible verification), ' + 'both run from the package root with POSIX sh, cat, test and cmp. ' + 'The reference overwrites result.txt; missing/unreadable input must produce a nonzero exit. ' + 'Verification succeeds only if result.txt exists and is byte-identical to seed.txt; otherwise it exits nonzero. ' + 'No network, other files, hidden tests, optional metadata extensions or additional grading constraints exist. ' + 'AUTHOR_NOTES.md and evidence/results are author disclosures, not solver examples.\n' + ).encode(), + 'AUTHOR_NOTES.md': b'No metadata extensions. The visible seed is explicitly an example. No runtime execution is claimed by this model-only qualification.\n', + 'workspace/seed.txt': b'qualified-result\n', + 'solution/apply.sh': b'#!/bin/sh\nset -eu\ncat workspace/seed.txt > workspace/result.txt\n', + 'tests/run.sh': b'#!/bin/sh\nset -eu\ntest -f workspace/result.txt\ncmp workspace/seed.txt workspace/result.txt\n', + 'evidence/results': b'Model-only engineering fixture: no execution or trial measurement is claimed.\n', + } + output = BytesIO() + with ZipFile(output, 'w') as archive: + for name, body in sorted(files.items()): + item = ZipInfo(name, date_time=(1980, 1, 1, 0, 0, 0)) + item.create_system, item.external_attr = 3, 0o100644 << 16 + archive.writestr(item, body) + return output.getvalue() + + +def validate_runtime_window(values, settings): + """Refuse an unrunnable paid rehearsal before creating its durable claim.""" + from app.core.config import Settings + from app.modules.checkers.api.external_policy import ( + SELECTED_POST_MATERIALIZATION_BUDGET_MS, + EXTERNAL_CHECKER_EXECUTION_CLEANUP_MARGIN_MS, + ) + + member = registry_spec(POST_IMAGE.rsplit('@', 1)[1]).resources.deadline_ms + execution = member + EXTERNAL_CHECKER_EXECUTION_CLEANUP_MARGIN_MS + required = SELECTED_POST_MATERIALIZATION_BUDGET_MS + execution + artifact_seconds = float(values['WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS']) + socket_seconds = float(values.get( + 'WORKSTREAM_EXTERNAL_CHECKER_SERVICE_TIMEOUT_SECONDS', + Settings.model_fields['external_checker_service_timeout_seconds'].default, + )) + require(settings.artifact_preparation_total_deadline_seconds == artifact_seconds + and settings.external_checker_service_timeout_seconds == socket_seconds, + 'model qualification runtime windows differ from inspected hosted roles') + require(artifact_seconds * 1000 >= required, + 'model qualification exceeds the inspected ART custody window') + require(socket_seconds * 1000 >= execution, + 'model qualification exceeds the inspected checker socket window') + return required + + +def require_completed_effect(ledger): + require(len(ledger['effects']) == 1 and ledger['effects'][0]['state'] == 'completed' + and ledger['effects'][0]['provider_response_known'] + and ledger['effects'][0]['request_binding_id'] and ledger['effects'][0]['response_binding_id'] + and ledger['reserved_nanodollars'] == 0, + 'model POST has an incomplete or ambiguous paid effect; stop') + + +@dataclass +class ModelPostQualification: + image_reference: str + image_config_digest: str + image_archive_sha256: str + proof_state: Path + historical_uncertain_reference: str + descriptor: int = -1 + + @classmethod + def read_and_claim(cls, path, node_root): + from experiments.hosted_drill_identity.fixture import read_private + + value = json.loads(read_private(path)) + require(set(value) == {'image_reference', 'image_config_digest', 'image_archive_sha256', + 'proof_state', 'historical_uncertain_reference'}, 'model proof fields differ') + require(isinstance(value['historical_uncertain_reference'], str) + and 0 < len(value['historical_uncertain_reference']) <= 256, + 'retain the earlier unresolved charge reference separately') + value['proof_state'] = Path(value['proof_state']) + require(value['proof_state'] == node_root / 'model-post-qualification.json', + 'model proof claim must use this deployment\'s single fixed state path') + result = cls(**value) + result.claim() + return result + + def claim(self): + require(self.proof_state.is_absolute() and self.proof_state.resolve() == self.proof_state, + 'model proof state must be canonical') + parent = self.proof_state.parent.stat() + require(stat.S_ISDIR(parent.st_mode) and parent.st_uid == os.getuid() + and parent.st_mode & 0o077 == 0, 'model proof directory must be private and owned') + # An existing claim forbids another paid project, including after a crash. + self.descriptor = os.open(self.proof_state, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + self.state = {'scope': 'manager-authored engineering model POST; no setup-agent/pilot/full-trial claim', + 'phase': 'claimed', 'total_authorization_nanodollars': CEILING, + 'historical_charge': {'reference': self.historical_uncertain_reference, 'status': 'unresolved_separate'}, + 'operation_keys': {key: str(uuid4()) for key in ('profile', 'budget', 'registry')}} + self.checkpoint() + directory = os.open(self.proof_state.parent, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW) + try: + os.fsync(directory) + finally: + os.close(directory) + + def checkpoint(self, **fields): + self.state.update(fields) + os.lseek(self.descriptor, 0, os.SEEK_SET) + data = canonical(self.state) + b'\n' + offset = 0 + while offset < len(data): + count = os.write(self.descriptor, data[offset:]) + require(count > 0, 'model proof state write failed') + offset += count + os.ftruncate(self.descriptor, len(data)) + os.fsync(self.descriptor) + + def close(self): + if self.descriptor >= 0: + os.close(self.descriptor) + self.descriptor = -1 + + def verify_cache(self, attachment): + from experiments.pilot04_external_checker_runtime.public_hosted_attachment import _command + + require((self.image_reference, self.image_config_digest, self.image_archive_sha256) + == (POST_IMAGE, POST_CONFIG, POST_ARCHIVE), 'model POST qualification image identity differs') + context_tree = _command(['git', 'rev-parse', 'HEAD:pilots/terminal_systems/checkers/post/model']).decode().strip() + require(context_tree == POST_CONTEXT_TREE, 'production model image context changed after qualification') + repository, image = self.image_reference.rsplit('@', 1) + require(repository == 'workstream/terminal-systems-post', 'qualification requires the production project image') + binding = json.loads((attachment.node_root / 'node-binding.json').read_text()) + matches = [item for item in binding['node_config']['cache'] if item['repository'] == repository + and item['platform_manifest_digest'] == image and item['config_image_id'] == self.image_config_digest + and item['smol']['archive_sha256'] == self.image_archive_sha256] + require(len(matches) == 1, 'model POST image is absent or substituted in native cache') + require(attachment.values['WORKSTREAM_MODEL_PROXY_ENABLED'] == 'true' + and 'model-proxy' in attachment.custody, 'actual hosted model proxy is required') + specification() + self.checkpoint(image_reference=self.image_reference, image_config_digest=self.image_config_digest, + image_archive_sha256=self.image_archive_sha256, + image_context_tree=context_tree, + rubric_sha256=digest((MODEL_CONTEXT / 'rubrics.json').read_bytes())) + + async def bind_project(self, client, project_id, attachment): + from app.core.identifiers import new_record_id + from app.modules.model_usage.api import ProjectModelProfile, ProjectModelBudgetConfigurationSpec + from experiments.pilot04_external_checker_runtime.public_probe_support import _json + + require('project_id' not in self.state, 'this qualification already owns one project') + self.checkpoint(phase='project_created', project_id=project_id) + token = attachment.token_for_label('operator') + root = f'/api/v1/operations/projects/{project_id}' + profile = profile_spec(UUID(project_id), new_record_id()) + published = await _json(client, 'POST', root + '/model-profiles', token, 201, + headers={'Idempotency-Key': self.state['operation_keys']['profile']}, + json=profile.model_dump(mode='json')) + self.profile = ProjectModelProfile.model_validate_json(canonical(published)) + require(self.profile.spec == profile, 'published model profile differs') + budget = ProjectModelBudgetConfigurationSpec(project_id=UUID(project_id), configuration_version=1, + authorized_total_nanodollars=CEILING, externally_reconciled_prior_spend_nanodollars=0, + lifetime_limit_nanodollars=CEILING) + receipt = await _json(client, 'POST', root + '/model-budget-configurations', token, 201, + headers={'Idempotency-Key': self.state['operation_keys']['budget']}, + json=budget.model_dump(mode='json')) + require(receipt['spec'] == budget.model_dump(mode='json'), 'published model budget differs') + require(receipt['lifetime_reserved_nanodollars'] == receipt['lifetime_settled_nanodollars'] == 0, + 'new qualification project already has model spend or reservations') + spec = registry_spec(self.image_reference.rsplit('@', 1)[1]) + registry = await _json(client, 'POST', '/api/v1/operations/external-checker-registry', token, 201, + headers={'Idempotency-Key': self.state['operation_keys']['registry']}, + json=spec.model_dump(mode='json')) + require(registry['entry_digest'] == spec.spec_digest, 'published model registry differs') + self.registry = registry + self.checkpoint(phase='authority_published', profile=published, budget=receipt, registry=registry, + maximum_run_reservation_nanodollars=profile.maximum_reservation_nanodollars) + + def guide_bytes(self): + _, statement = specification() + return ('# Hosted model POST engineering qualification\n\n' + 'This manager-authored guide selects only the production Terminal Systems ' + f'`{CHECK}` model rubric. It does not claim setup inference, a real pilot or runtime trials.\n\n' + + statement + '\n').encode() + + def rules(self, original, capabilities, count): + from experiments.pilot04_external_checker_runtime.public_hosted_attachment import manager_rules + + require(count == 0, 'model qualification must not grant a PRE model capability') + body = json.loads(manager_rules(original, capabilities, 0)) + spec, statement = specification() + body['requirements'] = [{'requirement_id': CHECK, 'statement': statement, + 'disposition': 'supported_post_submit', 'classification_rationale': 'Production model rubric through authorized hosted POST.', + 'evidence_refs': body['findings'][0]['evidence_refs'], 'check_specification': spec.model_dump(mode='json')}] + configuration = {'check': CHECK, **self.profile.selector.model_dump(mode='json')} + body['post_submit_bindings'] = [{'requirement_id': CHECK, 'stage': 'post_submit', + 'capability_id': self.registry['capability_id'], 'capability_version': self.registry['capability_version'], + 'parameters': [{'name': name, 'value': value} for name, value in sorted(configuration.items())]}] + # Selector DTO uses profile_* names; binding configuration uses reserved model_profile_* names. + for parameter in body['post_submit_bindings'][0]['parameters']: + if parameter['name'].startswith('profile_'): + parameter['name'] = 'model_' + parameter['name'] + return canonical(body) + + +async def account_snapshot(factory, project_id): + from sqlalchemy import or_, select + from app.modules.model_usage.models import ( + ModelUsageBudgetAccount, ModelUsageEffect, CheckerModelUsageSlot, SetupModelUsageSlot, + ) + + async with factory() as session: + account = await session.get(ModelUsageBudgetAccount, project_id) + effects = (await session.scalars(select(ModelUsageEffect) + .outerjoin(CheckerModelUsageSlot, CheckerModelUsageSlot.id == ModelUsageEffect.checker_slot_id) + .outerjoin(SetupModelUsageSlot, SetupModelUsageSlot.id == ModelUsageEffect.setup_slot_id) + .where(or_(CheckerModelUsageSlot.project_id == project_id, SetupModelUsageSlot.project_id == project_id)) + .order_by(ModelUsageEffect.id))).all() + require(account is not None, 'model qualification account is absent') + settled, reserved = account.lifetime_settled_nanodollars, account.lifetime_reserved_nanodollars + require(0 <= settled and 0 <= reserved and settled + reserved <= CEILING, + 'model qualification lifetime ledger exceeds authorization') + return {'settled_nanodollars': settled, 'reserved_nanodollars': reserved, + 'billing_scope': 'conservative ledger charge at pinned rates; not a provider invoice', + 'provider_invoice_qualified': False, + 'configuration_id': account.current_configuration_id, + 'effects': [{'effect_id': item.id, 'state': item.state, + 'settled_nanodollars': item.settled_nanodollars, + 'reserved_nanodollars': item.reserved_nanodollars, + 'provider_response_known': item.provider_response_id is not None, + 'request_binding_id': item.request_binding_id, 'response_binding_id': item.response_binding_id, + 'request_sha256': item.request_sha256, 'checker_slot_id': item.checker_slot_id} + for item in effects]} + + +def model_grant_snapshot(grant, calls): + """Retain the exact immutable grant/call facts before a public recovery.""" + require(len(calls) == 1 and calls[0].call_index == 0, + 'model qualification grant has an unexpected call set') + call = calls[0] + return {'grant_id': grant.id, 'checker_run_id': grant.checker_run_id, + 'registry_entry_id': str(grant.registry_entry_id), + 'capability_hash_sha256': digest(grant.token_hash.encode()), + 'external_request_digest': grant.external_request_digest, + 'execution_facts_sha256': digest(canonical(grant.execution_facts)), + 'paid_slot_sha256': digest(canonical(grant.paid_slot)), + 'authorization_evidence_id': grant.authorization_evidence_id, + 'issued_at': grant.issued_at.isoformat(), 'expires_at': grant.expires_at.isoformat(), + 'effect_id': call.attempt_id, 'request_sha256': call.request_sha256, + 'bound_at': call.bound_at.isoformat(), 'call_index': call.call_index} + + +async def paid_custody(factory, project_id, run_id, submission_id, member, qualification): + from sqlalchemy import select + from app.modules.model_usage.models import ( + CheckerModelInvocationGrant, CheckerModelInvocationGrantCall, CheckerModelUsageSlot, + ) + from app.modules.checkers.models import CheckerRun + from app.modules.checkers.execution_repository import stored_request + + async with factory() as session: + grants = (await session.scalars(select(CheckerModelInvocationGrant).where( + CheckerModelInvocationGrant.project_id == project_id))).all() + slots = (await session.scalars(select(CheckerModelUsageSlot).where( + CheckerModelUsageSlot.project_id == project_id))).all() + retained = qualification.state.get('prior_model_grant') + expected_grants = 1 if retained is None else 2 + require(len(grants) == expected_grants and len(slots) == 1, + 'model qualification has unexpected grant/paid-slot count') + current = [grant for grant in grants if grant.checker_run_id == str(run_id)] + require(len(current) == 1, 'model qualification has no unique current grant') + grant, slot = current[0], slots[0] + calls = (await session.scalars(select(CheckerModelInvocationGrantCall).where( + CheckerModelInvocationGrantCall.grant_id == grant.id))).all() + require(len(calls) == 1 and calls[0].call_index == 0 + and grant.checker_run_id == str(run_id) and grant.external_request_digest == member.request_digest + and str(grant.registry_entry_id) == qualification.registry['registry_entry_id'] + and slot.submission_id == submission_id and slot.requirement_id == CHECK + and slot.profile_id == str(qualification.profile.profile_id) + and slot.profile_digest == qualification.profile.profile_digest + and slot.registry_entry_digest == qualification.registry['entry_digest'], + 'model grant and paid slot do not bind this exact selected POST') + if retained is not None: + prior = next(item for item in grants if item.id != grant.id) + prior_calls = (await session.scalars(select(CheckerModelInvocationGrantCall).where( + CheckerModelInvocationGrantCall.grant_id == prior.id))).all() + require(model_grant_snapshot(prior, prior_calls) == retained, + 'retained predecessor grant or call changed') + previous_run = await session.get(CheckerRun, prior.checker_run_id) + current_run = await session.get(CheckerRun, str(run_id)) + require(previous_run is not None and current_run is not None + and previous_run.status == 'infrastructure_failed' + and previous_run.project_id == current_run.project_id == project_id + and previous_run.submission_id == current_run.submission_id == submission_id + and current_run.supersedes_checker_run_id == previous_run.id + and stored_request(previous_run).policy == stored_request(current_run).policy + and prior.registry_entry_id == grant.registry_entry_id + and prior_calls[0].attempt_id == calls[0].attempt_id + and prior_calls[0].request_sha256 == calls[0].request_sha256 + == slot.first_request_sha256, + 'model recovery changed lineage, policy or its single paid effect') + receipt = {'grant_id': grant.id, 'checker_run_id': grant.checker_run_id, + 'external_request_digest': grant.external_request_digest, + 'slot_id': slot.id, 'call_index': calls[0].call_index, + 'effect_id': calls[0].attempt_id, 'request_sha256': calls[0].request_sha256, + 'submission_id': slot.submission_id, 'profile_id': slot.profile_id} + if retained is not None: + receipt['prior_model_grant'] = retained + return receipt + + +async def run_model_post(attachment, client, factory, qualification): + from app.modules.tasks.models import SubmissionDispatch + from experiments.pilot04_external_checker_runtime.public_probe_support import ( + _public_project, _started_public_task, + ) + from experiments.pilot04_external_checker_runtime.run_public_journey_probe import _submit + + qualification.verify_cache(attachment) + await attachment.transition('live') + project, contributor, manager_token, contributor_token = await _public_project( + client, factory, attachment, human_review_required=False, pre_requirements=0) + project_id = str(project.project_id) + task, assignment = await _started_public_task(client, project_id=project_id, + manager_token=manager_token, contributor_token=contributor_token, source_ref='hosted-model-engineering-proof') + archive = fixture_archive() + qualification.checkpoint(phase='task_started', task_id=task, assignment_id=assignment, + archive_sha256=digest(archive), guide_setup=project.evidence) + summary = 'Synthetic model-only hosted qualification.' + attestation = 'rights_confirmed; no confidential data or credentials.' + saved = await _submit(client, factory, task_id=task, assignment_id=assignment, + contributor_token=contributor_token, archive=archive, summary=summary, + attestation=attestation, predecessor_id=None, + hosted=True) + qualification.checkpoint(phase='submission_created', submission=saved) + async with factory() as session: + dispatch = await session.get(SubmissionDispatch, saved['submission_id']) + require(dispatch is not None, 'model POST submission lacks durable dispatch') + event_id, run_id = dispatch.evaluation_event_id, dispatch.evaluation_attempt_id + return await observe_model_post( + attachment, client, factory, qualification, project=project, contributor=contributor, + task=task, assignment=assignment, archive=archive, summary=summary, + attestation=attestation, saved=saved, event_id=event_id, run_id=run_id, + ) + + +async def observe_model_post(attachment, client, factory, qualification, *, project, + contributor, task, assignment, archive, summary, attestation, + saved, event_id, run_id): + """Qualify one exact public dispatch; this never creates or retries a run.""" + import asyncio + from app.core.config import get_settings + from app.modules.checkers.execution_repository import stored_request, stored_result + from app.modules.checkers.api.external_policy import ( + selected_post_submit_execution_budget_ms, + SELECTED_POST_HANDLER_PRELUDE_MARGIN_MS, + SELECTED_POST_FINALIZATION_MARGIN_MS, + ) + from app.modules.checkers.models import CheckerRun + from app.modules.outbox.models import OutboxEvent + from experiments.pilot04_external_checker_runtime.public_probe_support import ( + _json, _wait_outbox_acknowledged, _fixture_hashes, + ) + from experiments.pilot04_external_checker_runtime.run_public_journey_probe import ( + _outbox_snapshot, _project_effects_snapshot, _assert_outcome, _assert_revision, + ) + + project_id = str(project.project_id) + async with factory() as session: + run = await session.get(CheckerRun, run_id) + require(run is not None and str(run.project_id) == project_id + and str(run.task_id) == str(task) + and str(run.submission_id) == saved['submission_id'], + 'observed model dispatch differs from the retained project and Submission') + budget = selected_post_submit_execution_budget_ms( + stored_request(run).policy, + artifact_deadline_ms=int(get_settings().artifact_preparation_total_deadline_seconds * 1000), + ) + require(budget == validate_runtime_window(attachment.values, get_settings()), + 'locked model qualification execution budget differs') + deadline = time.monotonic() + ( + budget + SELECTED_POST_HANDLER_PRELUDE_MARGIN_MS + SELECTED_POST_FINALIZATION_MARGIN_MS + ) / 1000 + while time.monotonic() < deadline: + async with factory() as session: + run = await session.get(CheckerRun, run_id) + state = run.status + if state in {'completed', 'infrastructure_failed'}: + break + await asyncio.sleep(.5) + else: + raise RuntimeError('model POST observation deadline reached; stop without another dispatch') + await _wait_outbox_acknowledged(factory, event_id, project.project_id) + ledger = await account_snapshot(factory, project_id) + qualification.checkpoint(phase='post_observed', ledger=ledger) + require_completed_effect(ledger) + async with factory() as session: + run = await session.get(CheckerRun, run_id) + require(run.status == 'completed' and run.completion_event_id is not None, + 'model POST did not retain a completed grader outcome') + result = stored_result(run) + members = result.execution.external_members if result.execution else [] + require(len(members) == 1 and members[0].requirement_id == CHECK + and members[0].result.outcome == 'completed', 'model POST has no exact production member result') + member = members[0] + require(str(member.registry_entry_id) == qualification.registry['registry_entry_id'] + and member.registry_entry_digest == qualification.registry['entry_digest'] + and (member.isolation.repository + '@' + member.isolation.platform_manifest_digest) == POST_IMAGE + and member.isolation.config_image_id == POST_CONFIG + and (member.isolation.runtime, member.isolation.isolation_mode) == ('smolvm', 'smol-microvm'), + 'model POST did not execute the exact approved native project image') + completion = run.completion_event_id + verdict, recommendation = members[0].result.verdict, run.routing_recommendation + require(verdict in {'passed', 'failed'}, 'model judgment has no supported work verdict') + await _wait_outbox_acknowledged(factory, completion, project.project_id) + custody = await paid_custody(factory, project_id, run_id, saved['submission_id'], members[0], qualification) + effect_id = ledger['effects'][0]['effect_id'] + require(custody['effect_id'] == effect_id and custody['slot_id'] == ledger['effects'][0]['checker_slot_id'] + and custody['request_sha256'] == ledger['effects'][0]['request_sha256'], + 'retained model call differs from the exact effect') + inspection = await _json(client, 'GET', f'/api/v1/operations/projects/{project_id}/model-usage/effects/{effect_id}', + attachment.token_for_label('operator'), 200) + require(inspection['owner_kind'] == 'checker' and inspection['owner']['submission_id'] == saved['submission_id'] + and inspection['state'] == 'completed' and inspection['request_evidence'] + and inspection['response_evidence'] and inspection['usage'], 'public effect inspection lost paid custody') + from app.modules.model_usage.api import ModelCallUsage, model_usage_cost + usage = ModelCallUsage.model_validate_json(canonical(inspection['usage'])) + charge = model_usage_cost(qualification.profile.spec, usage) + require(inspection['settled_nanodollars'] == ledger['settled_nanodollars'] == charge + and inspection['budget_configuration_id'] == ledger['configuration_id'] + == qualification.state['budget']['configuration_id'] + and inspection['owner']['profile_id'] == str(qualification.profile.profile_id) + and inspection['owner']['requirement_id'] == CHECK, + 'public model accounting differs from the exact profile or lifetime ledger') + expected = 'accepted' if verdict == 'passed' else 'needs_revision' + require(recommendation == ('allow_review' if verdict == 'passed' else 'needs_revision'), + 'selected POST recommendation differs from actual model verdict') + if verdict == 'passed': + governed = await _assert_outcome(factory, guide_source=project, project_id=project.project_id, + task_id=task, submission_id=saved['submission_id'], initial_submission_id=saved['submission_id'], + contributor_id=contributor.actor_profile_id, human_review_required=False, expected_awards=0, + expected_submissions=1, expected_inputs={saved['submission_id']: _fixture_hashes(archive)}) + else: + await _assert_revision(factory, project_id=project.project_id, task_id=task, + submission_id=saved['submission_id'], attempt_id=str(run_id)) + governed = {'task_status': expected, 'acceptance_qualified': False} + async with factory() as session: + governed_before = await _project_effects_snapshot(session, project.project_id) + event = await session.get(OutboxEvent, completion) + require(event.delivery_state == 'acknowledged', 'model completion was not acknowledged') + before = {str(identifier): await _outbox_snapshot(factory, identifier) for identifier in (event_id, completion)} + replay = await _json(client, 'POST', f'/api/v1/tasks/{task}/submissions', + attachment.token_for_label('contributor'), 201, + json={'assignment_id': assignment, 'admission_id': saved['admission_id'], + 'predecessor_submission_id': None, 'summary': summary, 'contributor_attestation': attestation}) + require(replay == saved, 'completed public Submission replay changed its receipt') + duplicate_tasks = [] + for identifier in (event_id, completion): + duplicate = await attachment.deliver_duplicate(identifier, project_id) + require(duplicate['result'] == {'status': 'delivery_unavailable'}, + 'duplicate model delivery result differs') + duplicate_tasks.append({'event_id': str(identifier), 'task_id': duplicate['task_id']}) + require(await account_snapshot(factory, project_id) == ledger + and {str(identifier): await _outbox_snapshot(factory, identifier) for identifier in (event_id, completion)} == before + and await paid_custody(factory, project_id, run_id, saved['submission_id'], members[0], qualification) == custody, + 'duplicate changed model usage, slots, grants or OUTBOX custody') + async with factory() as session: + require(await _project_effects_snapshot(session, project.project_id) == governed_before, + 'duplicate changed governed model outcome') + outcome = {'model_enabled_post_qualification': True, 'full_trial_qualification': False, + 'acceptance_qualified': verdict == 'passed', 'actual_verdict': verdict, + 'task_status': expected, 'project_id': project_id, 'submission': saved, + 'checker_run_id': str(run_id), 'completion_event_id': str(completion), + 'duplicate_deliveries': duplicate_tasks, 'ledger': ledger, 'effect_inspection': inspection, + 'paid_custody': custody, 'governed_outcome': governed} + qualification.checkpoint(phase='completed', outcome=outcome) + return outcome diff --git a/experiments/pilot04_external_checker_runtime/public_intake_trials.py b/experiments/pilot04_external_checker_runtime/public_intake_trials.py index 9e060c038..53e15a8a4 100644 --- a/experiments/pilot04_external_checker_runtime/public_intake_trials.py +++ b/experiments/pilot04_external_checker_runtime/public_intake_trials.py @@ -64,7 +64,7 @@ def _assert_ordered_pre_and_stop(row, *, failed): def _events(path): - return [json.loads(line) for line in path.read_text().splitlines()] if path.exists() else [] + return [json.loads(line) for line in path.read_text().splitlines()] if path and path.exists() else [] def _assert_project_invocations(row, trace, observer_start, node_root, runtime_start): @@ -73,12 +73,12 @@ def _assert_project_invocations(row, trace, observer_start, node_root, runtime_s observed = _events(trace)[observer_start:] starts = [event for event in observed if event['event'] == 'project_started'] ends = [event for event in observed if event['event'] == 'project_completed'] - _require([event['session'] for event in starts] == expected - and [event['session'] for event in ends] == expected, + _require(trace is None or ([event['session'] for event in starts] == expected + and [event['session'] for event in ends] == expected), 'project adapter invocations differ from retained ordered members') order = [(event['event'], event['session']) for event in observed if event['event'] in {'project_started', 'project_completed'}] - _require(order == [(kind, digest) for digest in expected + _require(trace is None or order == [(kind, digest) for digest in expected for kind in ('project_started', 'project_completed')], 'project checker invocations overlap or run outside canonical order') machines = assert_launches(node_root, runtime_start, creates=len(expected), @@ -87,7 +87,7 @@ def _assert_project_invocations(row, trace, observer_start, node_root, runtime_s if event['project'] and event['operation'] == 'exec'] _require([event['name'] for event in executions] == machines, 'project guest execution order differs from fresh allocation order') - for started, ended, executed in zip(starts, ends, executions, strict=True): + for started, ended, executed in zip(starts, ends, executions if trace else [], strict=True): _require(started['monotonic'] <= executed['monotonic'] <= ended['monotonic'], 'project VM execution is outside its exact adapter request') return machines @@ -104,6 +104,35 @@ async def _wait_event(path, start, event, session=None): raise RuntimeError("real intake adapter did not acknowledge " + event) +async def _native_event(root, start, operation, *, name=None, timeout=45): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + matches = [event for event in trace_events(root)[start:] + if event['operation'] == operation and event['project'] is False + and (name is None or event['name'] == name)] + if matches: + _require(len(matches) == 1 and matches[0].get('lease') + and matches[0]['lease']['name'] == matches[0]['name'], + 'native preparation evidence is ambiguous or lacks a lease') + return matches[0] + await asyncio.sleep(.05) + raise RuntimeError('native intake did not reach ' + operation) + + +async def _prepared(trace, start, node_root, runtime_start): + if trace is not None: + return await _wait_event(trace, start, 'prepare_ready') + event = await _native_event(node_root, runtime_start, 'prepare_completed') + return {'evidence': 'native_prepare_completed', 'session': event['name'], **event} + + +async def _closed(trace, start, session, node_root, runtime_start): + if trace is not None: + await _wait_event(trace, start, 'close_confirmed', session) + else: + await _native_event(node_root, runtime_start, 'delete', name=session) + + async def _cancel_upload(client, factory, route, headers, token, archive, trace, task, node_root): start = len(_events(trace)) key = str(uuid4()) @@ -114,7 +143,7 @@ async def _cancel_upload(client, factory, route, headers, token, archive, trace, async def incomplete(): nonlocal session_id yield archive[:1] - session_id = (await _wait_event(trace, start, "prepare_ready"))["session"] + session_id = (await _prepared(trace, start, node_root, runtime_start))["session"] ready.set() await asyncio.Event().wait() @@ -127,7 +156,7 @@ async def incomplete(): finally: pending.cancel() await asyncio.gather(pending, return_exceptions=True) - await _wait_event(trace, start, "close_confirmed", session_id) + await _closed(trace, start, session_id, node_root, runtime_start) cleanup = await node_command(node_root, "drained") _require(cleanup["foreign_preserved"] and not any(cleanup["resources"].values()), "cancelled upload retained owned resources or changed the foreign control") @@ -142,6 +171,44 @@ async def incomplete(): "cleanup": cleanup, "reserved_runs": 0} +async def _expire_upload(client, factory, route, headers, token, archive, task, node_root): + """Hold a real HTTP body open through its original native lease deadline.""" + key, start = str(uuid4()), len(trace_events(node_root)) + observed = {} + + async def stalled(): + yield archive[:1] + prepared = await _native_event(node_root, start, 'prepare_completed') + remaining = (prepared['lease']['expires_ms'] - time.time_ns() // 1_000_000) / 1000 + _require(0 < remaining <= 3600, 'native intake expiry exceeds bounded public proof window') + deleted = await _native_event(node_root, start, 'delete', name=prepared['name'], timeout=remaining + 45) + _require(deleted['wall_ms'] >= prepared['lease']['expires_ms'] + and deleted['lease']['epoch'] == prepared['lease']['epoch'] + and deleted['lease']['expires_ms'] == prepared['lease']['expires_ms'], + 'stalled upload did not reach its original lease expiry') + observed.update(prepared=prepared, deleted=deleted) + yield archive[1:] + + response = await client.post(route, headers=headers | {'Authorization': 'Bearer ' + token, + 'Idempotency-Key': key}, content=stalled()) + _require(response.status_code == 503 + and response.json()['error']['code'] == 'service_unavailable' + and response.json()['detail'] == 'pre_submission_attempt_outcome_unresolved', + 'expired upload was not reported as unresolved infrastructure') + cleanup = await node_command(node_root, 'drained') + _require(cleanup['foreign_preserved'] and not any(cleanup['resources'].values()), + 'expired upload cleanup changed foreign custody or retained resources') + machines = assert_launches(node_root, start, creates=1, executions=0, project=False) + assert_launches(node_root, start, creates=0, executions=0) + async with factory() as session: + _require(await session.scalar(select(func.count()).select_from(SelectedPreSubmitRun).where( + SelectedPreSubmitRun.idempotency_key == key)) == 0, + 'expired incomplete upload reserved a selected checker run') + await _assert_no_submission_after_intake_failure(factory, task) + return {'case': 'stalled_upload_original_expiry', 'observed': observed, 'machines': machines, + 'http_status': response.status_code, 'reserved_runs': 0, 'cleanup': cleanup} + + async def public_intake_trials(client, factory, env, trace: Path, node_root: Path): evidence = [] for count in (0, 2): @@ -159,25 +226,35 @@ async def public_intake_trials(client, factory, env, trace: Path, node_root: Pat "X-Contributor-Attestation": "rights_confirmed; no confidential data or credentials."} if count == 0: start = len(_events(trace)) + unauth_runtime_start = len(trace_events(node_root)) response = await client.post(route, headers=headers | {"Idempotency-Key": str(uuid4())}, content=archive) _require(response.status_code == 401, f"unauthenticated upload returned {response.status_code}; expected 401") _require(len(_events(trace)) == start, "unauthenticated upload allocated an intake session") + _require(len(trace_events(node_root)) == unauth_runtime_start, + 'unauthenticated upload allocated native resources') evidence.append(await _cancel_upload( client, factory, route, headers, contributor, archive, trace, task, node_root, )) + if trace is None: + evidence.append(await _expire_upload( + client, factory, route, headers, contributor, archive, task, node_root)) + # The real configured upload deadline may outlive a fixture + # token. Mint a fresh token for the next independent request. + contributor = env.token_for_label('contributor') start = len(_events(trace)) runtime_start = len(trace_events(node_root)) key = str(uuid4()) async def delayed_upload(): yield archive[:1] - ready = await _wait_event(trace, start, "prepare_ready") + ready = await _prepared(trace, start, node_root, runtime_start) evidence.append({"case": "upload_overlap", "ready": ready, "upload_finished_at": time.monotonic()}) yield archive[1:] result = await _json(client, "POST", route, contributor, 202, headers=headers | {"Idempotency-Key": key}, content=delayed_upload()) - await _wait_event(trace, start, "close_confirmed") + session = evidence[-1]['ready']['session'] + await _closed(trace, start, session, node_root, runtime_start) row = await _selected(factory, key) _assert_default_only_pre(row) _assert_project_invocations(row, trace, start, node_root, runtime_start) diff --git a/experiments/pilot04_external_checker_runtime/public_node_faults.py b/experiments/pilot04_external_checker_runtime/public_node_faults.py index 799ca5103..11686ff38 100644 --- a/experiments/pilot04_external_checker_runtime/public_node_faults.py +++ b/experiments/pilot04_external_checker_runtime/public_node_faults.py @@ -7,6 +7,7 @@ import os from pathlib import Path import signal +import stat import subprocess import time from uuid import uuid4 @@ -14,23 +15,56 @@ from experiments.pilot04_external_checker_runtime.public_fault_files import PrivateControlDirectory, private_path -def runtime(config, *args): +HOSTED_IDENTITY = (10001, 10001) + + +def _operator_controls(root): + return PrivateControlDirectory(root, trusted_uids=(10001,) if os.getuid() == 0 else ()) + + +def runtime_control_root(root): + """Resolve only the operator-written, fixed hosted sibling binding.""" + root = Path(root) + binding = root / 'runtime-controls.json' + if not binding.exists() and not binding.is_symlink(): + return root + control = _operator_controls(root) + try: + value = control.read(binding.name) + finally: + control.close() + expected = root.with_name(root.name + '-runtime') + if os.getuid() != 0 or value != {'root': str(expected), 'uid': 10001, 'gid': 10001}: + raise ValueError('runtime control binding differs') + private_path(expected, 0o700, directory=True, uid=10001, gid=10001) + return expected + + +def _runtime_user(identity): + if identity is None: + return {} + if os.getuid() != 0 or identity != HOSTED_IDENTITY: + raise ValueError('hosted runtime commands require the root operator') + return {'user': identity[0], 'group': identity[1], 'extra_groups': []} + + +def runtime(config, *args, identity=None): smol = config['smol'] return json.loads(subprocess.check_output( [smol['binary'], *args], env={'HOME': smol['home'], 'PATH': '/usr/bin:/bin'}, - text=True, timeout=15, + text=True, timeout=15, **_runtime_user(identity), )) -def owned_resources(config, root=None): +def owned_resources(config, root=None, *, identity=None): leases = Path(config['socket_path']).parent / 'smol-leases' owner = (leases.parent / '.workstream-checker-owner').read_text().strip() - machines = [row for row in runtime(config, 'machine', 'ls', '--json') + machines = [row for row in runtime(config, 'machine', 'ls', '--json', identity=identity) if row['labels'].get('workstream.external-checker-owner') == owner] groups = [p.name for p in Path(config['smol']['cgroup_root']).iterdir() if p.is_dir() and p.name.startswith('ws-')] - names = [] if root is None else [json.loads(line)['name'] for line in - (root / 'runtime-trace.jsonl').read_text().splitlines() if json.loads(line)['operation'] == 'create'] + names = [] if root is None else [event['name'] for event in trace_events(root) + if event['operation'] == 'create'] disks = [name for name in names if (Path(config['smol']['home']) / '.cache/smolvm/vms' / hashlib.sha256(name.encode()).hexdigest()[:16]).exists()] return {'disks': disks, 'leases': [p.name for p in leases.iterdir()], @@ -43,7 +77,7 @@ def _trace_require(condition, message): def trace_events(root): - return [json.loads(line) for line in (root / 'runtime-trace.jsonl').read_text().splitlines()] + return [json.loads(line) for line in (runtime_control_root(root) / 'runtime-trace.jsonl').read_text().splitlines()] def assert_launches(root, start, *, creates, executions, project=True, shared_deadline=True): @@ -82,7 +116,7 @@ def assert_launches(root, start, *, creates, executions, project=True, shared_de async def node_command(root, action): """Send only a closed fixture command to the separately owned host supervisor.""" identifier = uuid4().hex - control = PrivateControlDirectory(root) + control = _operator_controls(root) try: control.write(identifier + '.request', {'action': action}) finally: @@ -91,7 +125,7 @@ async def node_command(root, action): deadline = time.monotonic() + 45 while time.monotonic() < deadline: if response.exists(): - control = PrivateControlDirectory(root) + control = _operator_controls(root) try: result = control.read(response.name) finally: @@ -104,24 +138,77 @@ async def node_command(root, action): -def validate_node(config_path, config, unit, manifest_path): +def _config_identity(hosted_runtime): + if hosted_runtime: + if os.getuid() != 0: + raise ValueError('hosted supervisor requires the root operator') + return 10001, 10001, 0o400 + return os.getuid(), os.getgid(), 0o600 + + +def _read_config(config_path, hosted_runtime): + uid, gid, mode = _config_identity(hosted_runtime) + private_path(config_path, mode, uid=uid, gid=gid) + before = config_path.lstat() + descriptor = os.open(config_path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC) + with os.fdopen(descriptor, 'rb') as source: + if not os.path.samestat(before, os.fstat(source.fileno())) or before.st_size > 1024 * 1024: + raise ValueError('node config identity/size differs') + data = source.read(1024 * 1024 + 1) + if len(data) > 1024 * 1024: + raise ValueError('node config exceeds bound') + return data, (before.st_dev, before.st_ino) + + +def write_node_config(config_path, data, identity, *, hosted_runtime=False): + """Write only the validated existing inode, preserving its owner and mode.""" + if len(data) > 1024 * 1024: + raise ValueError('node config exceeds bound') + uid, gid, mode = _config_identity(hosted_runtime) + private_path(config_path, mode, uid=uid, gid=gid) + descriptor = os.open(config_path, os.O_WRONLY | os.O_NOFOLLOW | os.O_NONBLOCK | os.O_CLOEXEC) + with os.fdopen(descriptor, 'wb') as target: + info = os.fstat(target.fileno()) + if ((info.st_dev, info.st_ino) != identity or not stat.S_ISREG(info.st_mode) + or (info.st_uid, info.st_gid, stat.S_IMODE(info.st_mode)) != (uid, gid, mode) + or info.st_nlink != 1 or not os.path.samestat(info, config_path.lstat())): + raise ValueError('node config custody changed before write') + target.seek(0) + target.write(data) + target.truncate() + target.flush() + os.fsync(target.fileno()) + if not os.path.samestat(info, config_path.lstat()): + raise ValueError('node config custody changed during write') + + +def validate_node(config_path, config, unit, manifest_path, *, hosted_runtime=False): if config['smol'].get('maximum_parallel') != 1: raise ValueError('public proof requires a one-slot node') - private_path(config_path, 0o600) - private_path(manifest_path, 0o600) + original, _ = _read_config(config_path, hosted_runtime) + if json.loads(original) != config: + raise ValueError('node config changed during validation') + private_path(manifest_path, 0o600, trusted_uids=(10001,) if hosted_runtime else ()) manifest = json.loads(manifest_path.read_text()) socket = Path(config['socket_path']) - private_path(socket.parent, 0o700, directory=True) + uid, gid, _ = _config_identity(hosted_runtime) + private_path(socket.parent, 0o700, directory=True, uid=uid, gid=gid) owner_path = socket.parent / '.workstream-checker-owner' - private_path(owner_path, 0o600) + private_path(owner_path, 0o600, uid=uid, gid=gid) properties = dict(line.split('=', 1) for line in subprocess.check_output( ['systemctl', 'show', unit, '-p', 'MainPID', '-p', 'ControlGroup'], text=True).splitlines()) pid = int(properties['MainPID']) + if hosted_runtime: + status = dict(line.split(':', 1) for line in Path(f'/proc/{pid}/status').read_text().splitlines()) + if (config.get('sandbox_uid'), config.get('sandbox_gid')) != HOSTED_IDENTITY or any( + [int(value) for value in status[key].split()] != [expected] * 4 + for key, expected in [('Uid', uid), ('Gid', gid)]): + raise ValueError('live hosted node identity differs') executable = str(Path(f'/proc/{pid}/exe').resolve()) argv = Path(f'/proc/{pid}/cmdline').read_bytes().rstrip(b'\0').decode().split('\0') expected = {'unit': unit, 'config': str(config_path), 'socket': str(socket), 'executable': executable, 'owner': owner_path.read_text().strip(), - 'config_sha256': 'sha256:' + hashlib.sha256(config_path.read_bytes()).hexdigest(), + 'config_sha256': 'sha256:' + hashlib.sha256(original).hexdigest(), 'node_config': config, 'service_sha256': 'sha256:' + hashlib.sha256(Path(executable).read_bytes()).hexdigest(), 'smol_sha256': 'sha256:' + hashlib.sha256(Path(config['smol']['binary']).read_bytes()).hexdigest(), @@ -140,18 +227,20 @@ def validate_node(config_path, config, unit, manifest_path): class ForeignControl: """A uniquely named foreign VM and sibling process must survive node recovery.""" - def __init__(self, config, project): + def __init__(self, config, project, *, identity=None): self.config = config self.name = 'smol-proof-foreign-' + uuid4().hex self.group = Path(config['smol']['cgroup_root']) / self.name self.process = None self.created = False self.project = project + self.identity = identity def command(self, *args): smol = self.config['smol'] return subprocess.run([smol['binary'], *args], check=True, capture_output=True, - env={'HOME': smol['home'], 'PATH': '/usr/bin:/bin'}, timeout=30) + env={'HOME': smol['home'], 'PATH': '/usr/bin:/bin'}, timeout=30, + **_runtime_user(getattr(self, 'identity', None))) def start(self): self.command('machine', 'create', '--name', self.name, '--image', self.project['smol']['archive'], @@ -164,7 +253,7 @@ def start(self): self.check() def check(self): - rows = runtime(self.config, 'machine', 'ls', '--json') + rows = runtime(self.config, 'machine', 'ls', '--json', identity=getattr(self, 'identity', None)) matching = [row for row in rows if row['name'] == self.name] if (len(matching) != 1 or matching[0]['labels'].get('workstream.external-checker-owner') != 'foreign-control' or self.process.poll() is not None or not self.group.exists() @@ -178,42 +267,54 @@ def close(self): if self.group.exists(): self.group.rmdir() if self.created: - matching = [row for row in runtime(self.config, 'machine', 'ls', '--json') if row['name'] == self.name] + matching = [row for row in runtime(self.config, 'machine', 'ls', '--json', identity=getattr(self, 'identity', None)) if row['name'] == self.name] if len(matching) != 1 or matching[0]['labels'].get('workstream.external-checker-owner') != 'foreign-control': raise RuntimeError('foreign fixture identity changed before exact cleanup') self.command('machine', 'delete', '--name', self.name, '--force') disk = Path(self.config['smol']['home']) / '.cache/smolvm/vms' / hashlib.sha256(self.name.encode()).hexdigest()[:16] - remaining = [row for row in runtime(self.config, 'machine', 'ls', '--json') if row['name'] == self.name] + remaining = [row for row in runtime(self.config, 'machine', 'ls', '--json', identity=getattr(self, 'identity', None)) if row['name'] == self.name] if remaining or disk.exists() or self.group.exists() or (self.process is not None and self.process.poll() is None): raise RuntimeError('foreign fixture cleanup is incomplete') return {'name': self.name, 'machine_absent': True, 'disk_absent': True, 'cgroup_absent': True, 'process_reaped': True} -def serve(config_path, root, unit, manifest_path): +def serve(config_path, root, unit, manifest_path, *, hosted_runtime=False): """Inject only project-start faults on a dedicated empty node and restore on exit.""" from shutil import copyfile os.umask(0o077) - private_path(config_path, 0o600) - original = config_path.read_bytes() + original, config_identity = _read_config(config_path, hosted_runtime) config = json.loads(original) - binding = validate_node(config_path, config, unit, manifest_path) - if any(owned_resources(config).values()): + binding = validate_node(config_path, config, unit, manifest_path, hosted_runtime=hosted_runtime) + identity = HOSTED_IDENTITY if hosted_runtime else None + if any(owned_resources(config, identity=identity).values()): raise RuntimeError('node must be drained') root.mkdir(mode=0o700, parents=True, exist_ok=True) - control = PrivateControlDirectory(root) + control = _operator_controls(root) if list(root.iterdir()): raise ValueError('fault root must be fresh and empty') control.write('node-binding.json', binding) - wrapper = root / 'runtime-wrapper' + runtime_root = root + runtime_control = control + if hosted_runtime: + runtime_root = root.with_name(root.name + '-runtime') + runtime_root.mkdir(mode=0o700) + os.chown(runtime_root, *HOSTED_IDENTITY) + runtime_control = PrivateControlDirectory(runtime_root, uid=10001, gid=10001) + control.write('runtime-controls.json', {'root': str(runtime_root), 'uid': 10001, 'gid': 10001}) + wrapper = runtime_root / 'runtime-wrapper' copyfile(Path(__file__).with_name('public_runtime_wrapper.py'), wrapper) wrapper.chmod(0o700) + if hosted_runtime: + os.chown(wrapper, *HOSTED_IDENTITY) project = next(c for c in config['cache'] if c['repository'] == 'workstream-test/public-journey') - state = root / 'runtime-mode.json' - control.write(state.name, {'failures': 0, 'kill': False}) - trace = root / 'runtime-trace.jsonl' + state = runtime_root / 'runtime-mode.json' + runtime_control.write(state.name, {'failures': 0, 'kill': False}) + trace = runtime_root / 'runtime-trace.jsonl' trace.touch(mode=0o600) - control.write(wrapper.with_suffix('.json').name, { + if hosted_runtime: + os.chown(trace, *HOSTED_IDENTITY) + runtime_control.write(wrapper.with_suffix('.json').name, { 'binary': config['smol']['binary'], 'project_archive': project['smol']['archive'], 'mode': str(state), 'trace': str(trace), 'unit': unit, 'socket': config['socket_path'], @@ -228,11 +329,12 @@ def restart(): def stop(_signum, _frame): raise KeyboardInterrupt - foreign = ForeignControl(config, project) + foreign = ForeignControl(config, project, identity=identity) signal.signal(signal.SIGTERM, stop) try: foreign.start() - config_path.write_text(json.dumps(changed)) + write_node_config(config_path, json.dumps(changed).encode(), config_identity, + hosted_runtime=hosted_runtime) restart() (root / 'supervisor-ready').write_text('ready') while True: @@ -246,17 +348,17 @@ def stop(_signum, _frame): if action in {'project_fail_once', 'project_fail_all', 'project_kill', 'clear'}: failures = {'project_fail_once': 1, 'project_fail_all': -1, 'project_kill': 0, 'clear': 0}[action] - control.write(state.name, {'failures': failures, 'kill': action == 'project_kill'}) + runtime_control.write(state.name, {'failures': failures, 'kill': action == 'project_kill'}) if action == 'project_kill': result['controller_pid'] = int(subprocess.check_output( ['systemctl', 'show', unit, '-p', 'MainPID', '--value'], text=True)) elif action == 'drained': deadline = time.monotonic() + 25 - while any(owned_resources(config, root).values()): + while any(owned_resources(config, root, identity=identity).values()): if time.monotonic() >= deadline: raise RuntimeError('owned resources not drained') time.sleep(.1) - result['resources'] = owned_resources(config, root) + result['resources'] = owned_resources(config, root, identity=identity) foreign.check() result['foreign_preserved'] = foreign.name elif action == 'restarted': @@ -282,14 +384,14 @@ def stop(_signum, _frame): pass finally: try: - config_path.write_bytes(original) + write_node_config(config_path, original, config_identity, hosted_runtime=hosted_runtime) restart() if foreign.created and foreign.process is not None: foreign.check() finally: try: foreign_cleanup = foreign.close() - resources = owned_resources(config, root) + resources = owned_resources(config, root, identity=identity) if any(resources.values()) or config_path.read_bytes() != original: raise RuntimeError('supervisor restore or owned cleanup is incomplete') control.write('supervisor-cleanup.json', { @@ -298,6 +400,8 @@ def stop(_signum, _frame): }) (root / 'supervisor-ready').unlink(missing_ok=True) finally: + if runtime_control is not control: + runtime_control.close() control.close() @@ -307,5 +411,7 @@ def stop(_signum, _frame): parser.add_argument('--root', type=Path, required=True) parser.add_argument('--unit', required=True) parser.add_argument('--owner-manifest', type=Path, required=True) + parser.add_argument('--hosted-runtime', action='store_true', + help='root operator with the fixed runtime identity 10001:10001') args = parser.parse_args() - serve(args.node_config, args.root, args.unit, args.owner_manifest) + serve(args.node_config, args.root, args.unit, args.owner_manifest, hosted_runtime=args.hosted_runtime) diff --git a/experiments/pilot04_external_checker_runtime/public_probe_support.py b/experiments/pilot04_external_checker_runtime/public_probe_support.py index 351673979..ecae598f6 100644 --- a/experiments/pilot04_external_checker_runtime/public_probe_support.py +++ b/experiments/pilot04_external_checker_runtime/public_probe_support.py @@ -44,6 +44,10 @@ def _assert_input_hashes(row, expected): def _token(env: dict[str, str], actor) -> str: + from experiments.pilot04_external_checker_runtime.public_hosted_attachment import HostedAttachment + + if isinstance(env, HostedAttachment): + return env.token(actor) issuer, audience, secret = flow_settings(env) return issue_flow_token( str(actor.actor_profile_id), [], issuer=issuer, audience=audience, secret=secret @@ -201,13 +205,16 @@ async def _started_public_task(client, *, project_id: str, manager_token: str, async def _public_project(client, factory, env, *, human_review_required, pre_requirements=1): """Public guide activation and submitter grant shared by every runtime trial.""" + from experiments.pilot04_external_checker_runtime.public_hosted_attachment import HostedAttachment + + attachment = env if isinstance(env, HostedAttachment) else None compilation = await public_guide_setup( client, factory, human_review_required, _json, lambda actor: _token(env, actor), - pre_requirements=pre_requirements, + pre_requirements=pre_requirements, attachment=attachment, ) manager = compilation.manager guide_id = str(compilation.guide_id) - contributor = await _seed_contributor(factory) + contributor = attachment.actors['contributor'] if attachment else await _seed_contributor(factory) manager_token, contributor_token = _token(env, manager), _token(env, contributor) project_id = str(compilation.project_id) diff --git a/experiments/pilot04_external_checker_runtime/public_runtime_wrapper.py b/experiments/pilot04_external_checker_runtime/public_runtime_wrapper.py index 0410aa002..508d3065b 100644 --- a/experiments/pilot04_external_checker_runtime/public_runtime_wrapper.py +++ b/experiments/pilot04_external_checker_runtime/public_runtime_wrapper.py @@ -16,6 +16,9 @@ if len(arguments) >= 2 and arguments[0] == 'machine': operation = arguments[1] name = arguments[arguments.index('--name') + 1] if '--name' in arguments else None + final_copy = operation == 'cp' and arguments[-1].endswith(':/.workstream-guest') + if final_copy: + name = arguments[-1].removesuffix(':/.workstream-guest') with Path(settings['trace']).open('a+') as output: fcntl.flock(output, fcntl.LOCK_EX) output.seek(0) @@ -32,6 +35,23 @@ if lease else None}) + '\n') output.flush() os.fsync(output.fileno()) + if final_copy: + # Observe only actual successful native preparation. The service still + # emits and the real API client consumes its own prepare/ok protocol ACK. + result = subprocess.run([settings['binary'], *arguments], check=False) + if result.returncode == 0: + lease_path = Path(settings['socket']).parent / 'smol-leases' / (str(name) + '.json') + lease = json.loads(lease_path.read_text()) if lease_path.exists() else None + with Path(settings['trace']).open('a') as output: + fcntl.flock(output, fcntl.LOCK_EX) + output.write(json.dumps({'operation': 'prepare_completed', 'name': name, + 'project': project, 'monotonic': time.monotonic(), + 'wall_ms': time.time_ns() // 1_000_000, + 'lease': {k: lease[k] for k in ('name', 'owner', 'epoch', 'expires_ms', 'lifetime_ms')} + if lease else None}) + '\n') + output.flush() + os.fsync(output.fileno()) + sys.exit(result.returncode) if project and operation in {'start', 'exec'}: with Path(settings['mode']).open('r+') as state: fcntl.flock(state, fcntl.LOCK_EX) diff --git a/experiments/pilot04_external_checker_runtime/run_public_journey_probe.py b/experiments/pilot04_external_checker_runtime/run_public_journey_probe.py index 9bc035dfd..1d34c0079 100644 --- a/experiments/pilot04_external_checker_runtime/run_public_journey_probe.py +++ b/experiments/pilot04_external_checker_runtime/run_public_journey_probe.py @@ -412,7 +412,7 @@ async def _wait_outbox_due(factory, event_id: UUID) -> None: async def _ready_admission(client, factory, task_id: str, assignment_id: str, contributor_token: str, archive: bytes, summary: str, - attestation: str, predecessor_id: str | None = None) -> str: + attestation: str, predecessor_id: str | None = None, *, hosted=False) -> str: from app.workers.artifacts import verify_object expected_input = _fixture_hashes(archive) @@ -467,7 +467,7 @@ async def _ready_admission(client, factory, task_id: str, assignment_id: str, if now - status_since >= 60: puts, jobs = await _artifact_task_state(factory, task_id) pending = [identifier for identifier, state, _ in jobs if state == "pending"] - if status == "object_confirmed" and len(pending) == 1: + if not hosted and status == "object_confirmed" and len(pending) == 1: verification = await asyncio.to_thread( verify_object.apply_async, args=(pending[0],), ignore_result=False, @@ -497,10 +497,10 @@ async def _ready_admission(client, factory, task_id: str, assignment_id: str, async def _submit(client, factory, *, task_id: str, assignment_id: str, contributor_token: str, archive: bytes, summary: str, - attestation: str, predecessor_id: str | None, before_create=None) -> dict: + attestation: str, predecessor_id: str | None, before_create=None, hosted=False) -> dict: admission_id = await _ready_admission( client, factory, task_id, assignment_id, contributor_token, - archive, summary, attestation, predecessor_id, + archive, summary, attestation, predecessor_id, hosted=hosted, ) if before_create is not None: await before_create() @@ -680,6 +680,7 @@ async def _assert_outcome(factory, *, guide_source, project_id: UUID, task_id: s and row.acceptance_source == "task_post_submit_route", "submitter ContributionRecord lost its exact source or artifact lineage", ) + if expected_awards: award = (await session.execute(text( "SELECT contribution_record_id,contributor_id,contribution_policy_version_id," "instrument_type,unit_code,quantity FROM public.compensation_awards " @@ -789,7 +790,10 @@ async def _one_mode(client, factory, env, *, human_review_required: bool, operator_id: UUID | None, stop_worker: Callable[[], None] | None = None, start_worker: Callable[[], Awaitable[None]] | None = None, fault_root=None, operator_token=None) -> dict: - awards = () if human_review_required else ("project_points",) + from experiments.pilot04_external_checker_runtime.public_hosted_attachment import HostedAttachment + + attachment = env if isinstance(env, HostedAttachment) else None + awards = () if human_review_required or attachment else ("project_points",) from app.workers.outbox import deliver_event compilation, contributor, manager_token, contributor_token = await _public_project( @@ -877,6 +881,7 @@ async def fail_post_launch(): contributor_token=contributor_token, archive=first_archive, summary=summary, attestation=attestation, predecessor_id=None, before_create=fail_post_launch if not human_review_required and fault_root else None, + hosted=attachment is not None, ) submission_path = f"/api/v1/tasks/{task_id}/submissions" history = await _json(client, "GET", submission_path, contributor_token, 200) @@ -910,6 +915,7 @@ async def fail_post_launch(): contributor_token=contributor_token, archive=replacement_archive, summary=summary, attestation=attestation, predecessor_id=first_submission_id, + hosted=attachment is not None, ) expected_inputs[saved["submission_id"]] = _fixture_hashes(replacement_archive) _require(saved["submission_version"] == 2, @@ -930,8 +936,11 @@ async def fail_post_launch(): _require(stop_worker is not None and start_worker is not None, "prefork restart control is absent") stop_worker() - from tests.reviews.lifecycle.transition_support import command_for, transition - await transition(await command_for(operator_id, "live")) + if attachment: + await attachment.transition('live') + else: + from tests.reviews.lifecycle.transition_support import command_for, transition + await transition(await command_for(operator_id, "live")) await _wait_outbox_due(factory, completion_id) await start_worker() await _wait_outbox_acknowledged(factory, completion_id, @@ -960,22 +969,25 @@ async def fail_post_launch(): _require(item["submission_id"] == saved["submission_id"] and item["acceptance_source"] == "task_post_submit_route" and [(award["instrument"], award["unit"], Decimal(award["quantity"])) - for award in item["awards"]] == [("project_points", "PTS", Decimal("7"))], + for award in item["awards"]] == ([("project_points", "PTS", Decimal("7"))] if awards else []), "public contribution source or award set differs") - # A result for this one probe publish proves the prefork child consumed - # the duplicate; the production task still ignores results by default. + # Arm hosted task events before duplicate publication. The local fixture + # has a result backend; neither path changes the production worker. before_delivery = await _outbox_snapshot(factory, completion_id) - duplicate = await asyncio.to_thread( - deliver_event.apply_async, - args=(str(completion_id), project_id), - ignore_result=False, - ) - try: - duplicate_result = await asyncio.to_thread( - duplicate.get, timeout=120, propagate=True, - ) - finally: - await asyncio.to_thread(duplicate.forget) + if attachment: + duplicate_receipt = await attachment.deliver_duplicate(completion_id, project_id) + duplicate_result = duplicate_receipt['result'] + duplicate_task_id = duplicate_receipt['task_id'] + else: + duplicate = await asyncio.to_thread(deliver_event.apply_async, + args=(str(completion_id), project_id)) + duplicate_task_id = duplicate.id + try: + duplicate_result = await asyncio.to_thread( + duplicate.get, timeout=120, propagate=True, + ) + finally: + await asyncio.to_thread(duplicate.forget) _require(duplicate_result == {"status": "delivery_unavailable"}, "prefork child did not refuse duplicate completion delivery") _require(await _outbox_snapshot(factory, completion_id) == before_delivery, @@ -990,7 +1002,7 @@ async def fail_post_launch(): ) _require(again == outcome, "duplicate completion changed governed outcome") late_revocation = None - if not human_review_required: + if not human_review_required and attachment is None: late_revocation = await _late_revocation_trial( client, factory, env, project_id=compilation.project_id, accepted_task_id=task_id, manager_token=manager_token, @@ -1014,6 +1026,10 @@ async def fail_post_launch(): ), "outcome": outcome, "submitted_inputs": expected_inputs, + 'duplicate_delivery': { + 'task_id': duplicate_task_id, + 'observation': 'scoped_worker_events' if attachment else 'result_backend', + }, "late_revocation": late_revocation, } @@ -1249,13 +1265,29 @@ def main() -> int: parser.add_argument("--image-reference") parser.add_argument("--node-config", type=Path, help="Owned provisioned Smol node configuration") parser.add_argument("--broker-url") - parser.add_argument("--expected-custody", type=Path, required=True) + parser.add_argument("--expected-custody", type=Path) parser.add_argument("--work-dir", type=Path, help="Unique private directory retaining API and worker diagnostics") + parser.add_argument('--hosted-attachment', type=Path, + help='Private attachment manifest for an existing hosted API/worker/Beat') + parser.add_argument('--hosted-model-post', type=Path, + help='Private model qualification input; requires hosted attachment and completed actor bootstrap') args = parser.parse_args() if not args.database_url: parser.error("isolated --database-url or WORKSTREAM_TEST_DATABASE_URL is required") - if not args.image_reference or not args.node_config or not args.broker_url: + if args.hosted_model_post and not args.hosted_attachment: + parser.error('model POST qualification requires explicit hosted attachment') + if args.hosted_attachment: + if (not args.broker_url or args.image_reference or args.node_config + or args.work_dir or args.expected_custody): + parser.error('hosted attachment requires broker URL and excludes local launch arguments') + from experiments.pilot04_external_checker_runtime.public_hosted_attachment import HostedAttachment, run as attach + attachment = HostedAttachment.read(args.hosted_attachment) + value = asyncio.run(attach(attachment, args.database_url, args.broker_url, args.hosted_model_post) + if args.hosted_model_post else attach(attachment, args.database_url, args.broker_url)) + print(json.dumps(value, sort_keys=True, separators=(',', ':'))) + return 0 + if not args.image_reference or not args.node_config or not args.broker_url or not args.expected_custody: parser.error("full drill requires image, node config, and broker URL") workspace = (nullcontext(args.work_dir) if args.work_dir else tempfile.TemporaryDirectory(prefix="ws-pilot04-public-")) diff --git a/external_checkers/README.md b/external_checkers/README.md index f9373c47c..1cdb878de 100644 --- a/external_checkers/README.md +++ b/external_checkers/README.md @@ -191,6 +191,12 @@ Hosted Smol composition and its scoped AWS proof are tracked separately in does not establish live-model grading or the full Terminal Systems guide-to-contribution journey. +The [hosted composition](../deploy/hosted/README.md) uses that same native node +for typed intake and project execution, with credentials confined to the trusted +proxy. Its configuration/readiness checks do not establish a live model call or +public lifecycle qualification. PRE model selectors remain unsupported; only +existing approved POST model lineage can issue that grant. + The opt-in [`model_relay_docker.py`](workstream-checker-service/tests/model_relay_docker.py) probe drives a real Docker-development checker from its invocation-local socket, @@ -273,7 +279,7 @@ if that capability still exists. ## Smol microVM engine -The node also accepts the deployment pair `smolvm` / `smol-microvm`. Python still +The hosted node uses the deployment pair `smolvm` / `smol-microvm`. Python still uses the single `external_checker_execution/unix_socket` provider; neither a project policy nor a checker request can choose the engine. The trusted guest receives typed current and optional predecessor material roots and the authorized @@ -286,10 +292,14 @@ Use Linux x86-64 with `/dev/kvm`, cgroup v2 and Smol 1.31.4. The places the controller in its own subgroup; `DelegateSubgroup` requires systemd 254 or later ([systemd documentation](https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#DelegateSubgroup=)). Set `smol.cgroup_root` to that unit's delegated root, not the controller subgroup. -The service refuses startup outside its delegation. Use the ART grant issuer's -Unix identity and a separate private socket directory, Smol HOME and image cache -for each node. The sample unit's CPU, memory and task ceilings are deployment -capacity; individual registered checker limits remain unchanged. +The service refuses startup outside its delegation. Use the ART grant issuer's numeric UID/GID (`10001:10001` in the hosted unit) +and a separate private socket directory, Smol HOME and image cache for each node. The sample unit's CPU, memory and task ceilings are deployment +capacity; individual registered checker limits remain unchanged. Hosted releases +may select one canonical private deployment root and a digest-pinned native +binary path; the same derived absolute scratch/socket paths must appear in the +node JSON and API/worker mounts. See the +[hosted operator sequence](../deploy/hosted/README.md#start-and-inspect) for +migration/activation/proxy prerequisites and native readiness. Keep the canonical Smol HOME path at most 59 filesystem bytes. Trial-enabled nodes also need a separately owned `trial_runtime.child_home_root` at most 18 @@ -407,10 +417,10 @@ Backend API and durable workers require all three settings: - `WORKSTREAM_ARTIFACT_INTAKE_CONFIG_DIGEST`: exact image config digest. The configured socket must serve Smol with the expected trusted image and default -behavior. A Docker/runsc project node can remain separate, but cannot satisfy ART -intake readiness. Missing or mismatched configuration fails closed; there is no -host ZIP/default fallback. Existing Docker-only deployment compositions therefore -need a separately provisioned Smol intake node before accepting uploads or POST. +behavior. The hosted release uses the same native node socket for this typed capability +and project execution; the protocols and privileged intake image remain separate. +Missing or mismatched configuration fails closed. Local Docker-only mechanism +configurations still cannot supply Smol intake readiness. The original artifact deadline bounds upload and intake preparation together. Only known pre-launch infrastructure failures permit at most three preparation diff --git a/external_checkers/deploy/workstream-checker.service b/external_checkers/deploy/workstream-checker.service index 71412f2ec..d975c8b2d 100644 --- a/external_checkers/deploy/workstream-checker.service +++ b/external_checkers/deploy/workstream-checker.service @@ -7,13 +7,13 @@ StartLimitBurst=5 [Service] Type=simple # Use the same Unix identity as the ART scratch/grant issuer. -User=workstream -Group=workstream +User=10001 +Group=10001 SupplementaryGroups=kvm StateDirectory=workstream-checker StateDirectoryMode=0700 UMask=0077 -ExecStart=/opt/workstream/bin/workstream-checker-service /var/lib/workstream-checker/service.json +ExecStart=/opt/workstream/bin/workstream-checker-service /etc/workstream/checker-service.json Delegate=cpu memory pids DelegateSubgroup=controller MemoryMax=4G diff --git a/scripts/test_hosted_release_config.py b/scripts/test_hosted_release_config.py index 4f1d3b1d8..90749308c 100644 --- a/scripts/test_hosted_release_config.py +++ b/scripts/test_hosted_release_config.py @@ -30,6 +30,7 @@ ) assert SPEC is not None and SPEC.loader is not None preflight = importlib.util.module_from_spec(SPEC) +sys.modules[SPEC.name] = preflight SPEC.loader.exec_module(preflight) COMPOSE_SPEC = importlib.util.spec_from_file_location( @@ -125,7 +126,8 @@ def tearDown(self) -> None: def test_valid_configuration_and_rendered_topology(self) -> None: self.assertEqual( - str(preflight.SOCKET_ROOT), "/var/lib/workstream/checker-runtime" + str(preflight.deployment_paths(self.environment).socket_root), + "/var/lib/workstream/checker-runtime", ) preflight.validate_environment(self.environment) preflight.validate_checker_config(self.checker, self.environment) @@ -139,7 +141,6 @@ def test_valid_configuration_and_rendered_topology(self) -> None: "beat", "model-proxy", "guide-document-parser", - "checker-service", *preflight.PROOF_SERVICES, }, ) @@ -154,7 +155,6 @@ def test_valid_configuration_and_rendered_topology(self) -> None: "beat", "model-proxy", "guide-document-parser", - "checker-service", ): service = services[name] self.assertEqual(service["pull_policy"], "always") @@ -204,7 +204,6 @@ def test_valid_configuration_and_rendered_topology(self) -> None: "worker", "beat", "model-proxy", - "checker-service", ) } self.assertIn("/api/v1/health", health_commands["api"]) @@ -214,7 +213,9 @@ def test_valid_configuration_and_rendered_topology(self) -> None: "/var/lib/workstream/model-runtime/model.sock", health_commands["model-proxy"], ) - self.assertIn(".Runtimes.runsc", health_commands["checker-service"]) + self.assertNotIn("checker-service", services) + for service in services.values(): + self.assertNotIn("/var/run/docker.sock", preflight._service_mounts(service)) self.assertEqual(services["api"]["stop_grace_period"], "30s") self.assertIn(services["worker"]["stop_grace_period"], {"7320s", "2h2m0s"}) self.assertIn(services["model-proxy"]["stop_grace_period"], {"7320s", "2h2m0s"}) @@ -256,7 +257,6 @@ def test_valid_configuration_and_rendered_topology(self) -> None: ) else: self.assertNotIn(key, services[name]["environment"]) - self.assertNotIn("secrets", services["checker-service"]) parser = services["guide-document-parser"] self.assertEqual(parser["image"], services["api"]["image"]) self.assertEqual(parser["user"], "0:0") @@ -269,7 +269,7 @@ def test_valid_configuration_and_rendered_topology(self) -> None: { "WORKSTREAM_RELEASE_ROLE": "parser", "WORKSTREAM_GUIDE_DOCUMENT_PARSER_SOCKET": str( - preflight.GUIDE_SOCKET_PATH + preflight.deployment_paths(self.environment).guide_socket ), }, ) @@ -278,20 +278,23 @@ def test_valid_configuration_and_rendered_topology(self) -> None: self.assertEqual(parser["cpus"], 1) self.assertEqual(parser["pids_limit"], 64) self.assertFalse( - preflight._service_mounts(parser)[str(preflight.GUIDE_SOCKET_ROOT)].get( - "read_only", False - ) + preflight._service_mounts(parser)[ + str(preflight.deployment_paths(self.environment).guide_root) + ].get("read_only", False) ) worker_mounts = preflight._service_mounts(services["worker"]) - self.assertTrue(worker_mounts[str(preflight.GUIDE_SOCKET_ROOT)]["read_only"]) + self.assertTrue( + worker_mounts[str(preflight.deployment_paths(self.environment).guide_root)][ + "read_only" + ] + ) for name in ( "api", "beat", - "checker-service", *preflight.PROOF_SERVICES, ): self.assertNotIn( - str(preflight.GUIDE_SOCKET_ROOT), + str(preflight.deployment_paths(self.environment).guide_root), preflight._service_mounts(services[name]), ) for name in ("api", "worker", "model-proxy"): @@ -305,7 +308,7 @@ def test_valid_configuration_and_rendered_topology(self) -> None: ) beat_mounts = preflight._service_mounts(services["beat"]) self.assertEqual(beat_mounts, {}) - model_socket_root = str(preflight.MODEL_SOCKET_ROOT) + model_socket_root = str(preflight.deployment_paths(self.environment).model_root) self.assertTrue(worker_mounts[model_socket_root]["read_only"]) self.assertFalse( preflight._service_mounts(services["model-proxy"])[model_socket_root].get( @@ -317,16 +320,22 @@ def test_valid_configuration_and_rendered_topology(self) -> None: model_socket_root, preflight._service_mounts(services[name]), ) - checker_mounts = preflight._service_mounts(services["checker-service"]) - self.assertTrue(checker_mounts[model_socket_root]["read_only"]) - relay_root = str(preflight.MODEL_RELAY_ROOT) - self.assertEqual(checker_mounts[relay_root]["source"], relay_root) - self.assertFalse(checker_mounts[relay_root].get("read_only", False)) - for name in ("api", "worker", "beat", "model-proxy", *preflight.PROOF_SERVICES): - self.assertNotIn(relay_root, preflight._service_mounts(services[name])) - self.assertNotIn( - str(preflight.ROLES_ANYWHERE_CERTIFICATE_TARGET), checker_mounts - ) + relay_root = str(preflight.deployment_paths(self.environment).model_relays) + for name, service in services.items(): + self.assertNotIn(relay_root, preflight._service_mounts(service)) + for role in ("api", "worker"): + self.assertEqual( + services[role]["environment"][ + "WORKSTREAM_ARTIFACT_INTAKE_SERVICE_SOCKET" + ], + str(preflight.deployment_paths(self.environment).socket), + ) + self.assertEqual( + services[role]["environment"][ + "WORKSTREAM_EXTERNAL_CHECKER_SERVICE_SOCKET" + ], + str(preflight.deployment_paths(self.environment).socket), + ) def test_actual_render_rejects_beat_inheriting_api_volumes(self) -> None: compose_text = preflight.COMPOSE_PATH.read_text(encoding="utf-8").replace( @@ -373,13 +382,13 @@ def test_actual_render_rejects_extra_backend_identity_binds(self) -> None: cases = ( ( "API readiness key", - " depends_on:\n checker-service:\n", + " restart: on-failure:5\n stop_grace_period: 30s\n healthcheck:\n test: [CMD, python, -c,", "WORKSTREAM_ARTIFACT_S3_ROLES_ANYWHERE_READINESS_PRIVATE_KEY_HOST_FILE", "/run/unrelated/readiness-key.pem", ), ( "API negative key", - " depends_on:\n checker-service:\n", + " restart: on-failure:5\n stop_grace_period: 30s\n healthcheck:\n test: [CMD, python, -c,", "WORKSTREAM_ARTIFACT_S3_ROLES_ANYWHERE_NEGATIVE_PRIVATE_KEY_HOST_FILE", "/run/unrelated/negative-key.pem", ), @@ -415,34 +424,28 @@ def test_actual_render_rejects_extra_backend_identity_binds(self) -> None: ): preflight.validate_compose(rendered, self.environment) - def test_actual_render_rejects_checker_hardlink_identity_bind(self) -> None: - private_key_alias = self.root / "runtime-key-checker-alias.pem" + def test_actual_render_rejects_parser_hardlink_identity_bind(self) -> None: + private_key_alias = self.root / "runtime-key-parser-alias.pem" os.link(self.runtime_private_key, private_key_alias) - marker = ( - " healthcheck:\n" - ' test: [CMD-SHELL, "test -S /var/lib/workstream/checker-runtime/checker.sock' - ) + marker = " healthcheck:\n test: [CMD, python, -m, app.adapters.guide_document_tools.service" extra_bind = ( " - type: bind\n" f" source: {private_key_alias}\n" - " target: /run/unrelated/key.pem\n" - " read_only: true\n" + " target: /run/unrelated/key.pem\n read_only: true\n" ) compose_text = preflight.COMPOSE_PATH.read_text(encoding="utf-8").replace( - marker, - extra_bind + marker, - 1, + marker, extra_bind + marker, 1 ) rendered = self._render_modified_compose(compose_text) - - checker_mounts = preflight._service_mounts( - rendered["services"]["checker-service"] + parser_mounts = preflight._service_mounts( + rendered["services"]["guide-document-parser"] ) self.assertEqual( - checker_mounts["/run/unrelated/key.pem"]["source"], - str(private_key_alias), + parser_mounts["/run/unrelated/key.pem"]["source"], str(private_key_alias) ) - with self.assertRaisesRegex(preflight.PreflightError, "checker-service bind"): + with self.assertRaisesRegex( + preflight.PreflightError, "guide-document-parser bind" + ): preflight.validate_compose(rendered, self.environment) def test_actual_render_rejects_secret_file_substitution(self) -> None: @@ -531,13 +534,13 @@ def test_render_and_compose_operation_ignore_hostile_ambient_environment( "deploy/hosted/compose.py", (ROOT / "deploy/hosted/README.md").read_text() ) - def _compose_main(self, operation: str) -> tuple[int, str]: + def _compose_main(self, *arguments: str) -> tuple[int, str]: errors = io.StringIO() with ( patch.object( sys, "argv", - ["compose.py", "--env-file", str(self.environment_path), operation], + ["compose.py", "--env-file", str(self.environment_path), *arguments], ), redirect_stderr(errors), ): @@ -550,7 +553,7 @@ def test_launch_rechecks_checker_selectors_after_preflight(self) -> None: sys, "argv", ["preflight.py", "--env-file", str(self.environment_path)] ), patch.object(preflight, "validate_private_host_inputs"), - patch.object(preflight, "validate_docker_runtime"), + patch.object(preflight, "validate_smol_runtime"), patch.object(preflight, "validate_trusted_intake"), redirect_stdout(io.StringIO()), ): @@ -562,7 +565,7 @@ def test_launch_rechecks_checker_selectors_after_preflight(self) -> None: ) with ( patch.object(compose, "validate_private_host_inputs") as private, - patch.object(compose, "validate_docker_runtime") as runtime, + patch.object(compose, "validate_smol_runtime") as runtime, patch.object(compose, "run_compose") as run, ): result, error = self._compose_main("up") @@ -610,9 +613,11 @@ def launch(path, values, arguments): side_effect=private_check, ), patch.object( - compose, "validate_docker_runtime", side_effect=runtime_check + compose, "validate_smol_runtime", side_effect=runtime_check + ), + patch.object( + compose, "validate_trusted_intake", side_effect=intake_check ), - patch.object(compose, "validate_trusted_intake", side_effect=intake_check), patch.object(compose, "run_compose", side_effect=launch) as run, ): result, error = self._compose_main("up") @@ -625,8 +630,11 @@ def launch(path, values, arguments): self.assertIn("rejected", error) self.assertEqual( order, - {"private": ["private"], "runtime": ["private", "runtime"], - "intake": ["private", "runtime", "intake"]}[failure], + { + "private": ["private"], + "runtime": ["private", "runtime"], + "intake": ["private", "runtime", "intake"], + }[failure], ) run.assert_not_called() @@ -635,7 +643,7 @@ def test_stop_remains_available_with_invalid_checker_configuration(self) -> None with ( patch.object(compose, "load_checker_config") as load, patch.object(compose, "validate_private_host_inputs") as private, - patch.object(compose, "validate_docker_runtime") as runtime, + patch.object(compose, "validate_smol_runtime") as runtime, patch.object(compose, "run_shutdown", return_value=0) as shutdown, ): self.assertEqual(self._compose_main("stop"), (0, "")) @@ -644,7 +652,9 @@ def test_stop_remains_available_with_invalid_checker_configuration(self) -> None runtime.assert_not_called() shutdown.assert_called_once_with(self.environment_path, self.environment) - def test_shutdown_stops_producers_then_worker_then_parser_and_checker(self) -> None: + def test_shutdown_drains_producers_worker_and_proxy_without_stopping_native_node( + self, + ) -> None: completed = subprocess.CompletedProcess([], 0) with patch.object(preflight.subprocess, "run", return_value=completed) as run: self.assertEqual( @@ -658,8 +668,8 @@ def test_shutdown_stops_producers_then_worker_then_parser_and_checker(self) -> N ["stop", "--timeout", "7320", "model-proxy"], ) self.assertEqual( - commands[3][-3:], - ["stop", "guide-document-parser", "checker-service"], + commands[3][-2:], + ["stop", "guide-document-parser"], ) def test_shutdown_failure_keeps_later_dependencies_running(self) -> None: @@ -693,6 +703,7 @@ def test_stop_rejects_caller_selected_service_order(self) -> None: str(self.environment_path), "stop", "worker", + "api", ], ), patch.object(compose, "run_shutdown") as shutdown, @@ -701,8 +712,123 @@ def test_stop_rejects_caller_selected_service_order(self) -> None: self.assertEqual(compose.main(), 1) shutdown.assert_not_called() self.assertIn( - "fixed producer, worker, parser/checker sequence", errors.getvalue() + "fixed shutdown sequence or exactly one worker", errors.getvalue() + ) + + def test_exact_worker_stop_preserves_dependencies_and_start_rechecks_native(self): + self.checker_path.write_text("invalid json", encoding="utf-8") + with ( + patch.object(compose, "load_checker_config") as load, + patch.object(compose, "run_compose", return_value=0) as run, + ): + self.assertEqual(self._compose_main("stop", "worker"), (0, "")) + load.assert_not_called() + run.assert_called_once_with( + self.environment_path, self.environment, ["stop", "worker"] + ) + self.checker_path.write_text(json.dumps(self.checker), encoding="utf-8") + for failure in (None, "private", "native", "intake"): + with ( + self.subTest(failure=failure), + patch.object( + compose, + "validate_private_host_inputs", + side_effect=preflight.PreflightError("private") + if failure == "private" + else None, + ) as private, + patch.object( + compose, + "validate_smol_runtime", + side_effect=preflight.PreflightError("native") + if failure == "native" + else None, + ) as native, + patch.object( + compose, + "validate_trusted_intake", + side_effect=preflight.PreflightError("intake") + if failure == "intake" + else None, + ) as intake, + patch.object(compose, "run_compose", return_value=0) as run, + ): + result, _ = self._compose_main("start", "worker") + self.assertEqual(result, 0 if failure is None else 1) + if failure is None: + private.assert_called_once_with(self.environment) + native.assert_called_once_with(self.checker, self.environment) + intake.assert_called_once_with(self.environment) + run.assert_called_once_with( + self.environment_path, self.environment, ["start", "worker"] + ) + else: + run.assert_not_called() + + def test_worker_operations_have_fixed_grace_readiness_and_management_bounds(self): + cases = ( + (["stop", "worker"], ["stop", "--timeout", "7320", "worker"], 7380), + ( + ["start", "worker"], + [ + "up", + "--detach", + "--no-deps", + "--wait", + "--wait-timeout", + "60", + "worker", + ], + 120, + ), ) + for arguments, expected, timeout in cases: + with ( + self.subTest(arguments=arguments), + patch.object( + preflight.subprocess, + "run", + return_value=subprocess.CompletedProcess([], 0), + ) as run, + ): + self.assertEqual( + preflight.run_compose( + self.environment_path, self.environment, arguments + ), + 0, + ) + self.assertEqual(run.call_args.args[0][-len(expected) :], expected) + self.assertEqual(run.call_args.kwargs["timeout"], timeout) + self.assertEqual( + run.call_args.kwargs["env"]["DOCKER_HOST"], + "unix:///var/run/docker.sock", + ) + with patch.object( + preflight.subprocess, + "run", + side_effect=subprocess.TimeoutExpired("docker", timeout), + ): + with self.assertRaisesRegex( + preflight.PreflightError, "management deadline" + ): + preflight.run_compose( + self.environment_path, self.environment, arguments + ) + for arguments in ( + ["start", "api"], + ["restart", "worker"], + ["stop", "--timeout", "0", "worker"], + ["start", "worker", "api"], + ): + with ( + self.subTest(arguments=arguments), + patch.object(preflight.subprocess, "run") as run, + ): + with self.assertRaises(preflight.PreflightError): + preflight.run_compose( + self.environment_path, self.environment, arguments + ) + run.assert_not_called() def test_proof_wrapper_runs_only_the_fixed_identity_sequence(self) -> None: nonce = "0199f68b-5dc2-7000-8000-000000000001" @@ -720,12 +846,15 @@ def test_proof_wrapper_runs_only_the_fixed_identity_sequence(self) -> None: ], ), patch.object(compose, "load_checker_config", return_value=self.checker), - patch.object(compose, "validate_private_host_inputs"), - patch.object(compose, "validate_docker_runtime"), - patch.object(compose, "validate_trusted_intake"), + patch.object(compose, "validate_private_host_inputs") as private, + patch.object(compose, "validate_smol_runtime") as native, + patch.object(compose, "validate_trusted_intake") as intake, patch.object(compose, "run_proof_group", return_value=0) as proof_group, ): self.assertEqual(compose.main(), 0) + private.assert_called_once_with(self.environment, require_node=False) + native.assert_not_called() + intake.assert_not_called() self.assertEqual(str(proof_group.call_args.args[2]), nonce) for invalid in (["proof"], ["proof", "--group-nonce", "not-a-uuid"]): with ( @@ -871,6 +1000,64 @@ def timeout_then_cleanup(command, **kwargs): self.assertEqual(calls[1], ["docker", "rm", "--force", name]) self.assertFalse(any("aws-proof-coordinate" in command for command in calls)) + def test_artifact_preparation_deadline_is_required_bounded_and_shared(self): + key = "WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS" + for deadline in ("360", "3400"): + values = self.environment | {key: deadline} + with self.subTest(deadline=deadline): + preflight.validate_environment(values) + rendered = preflight.render_compose(self.environment_path, values) + preflight.validate_compose(rendered, values) + for role in ( + *preflight.ROLE_SECRET_FILE_KEYS, + *preflight.PROOF_SERVICES, + ): + self.assertEqual( + rendered["services"][role]["environment"][key], deadline + ) + for mutation in ("missing", "different"): + changed = copy.deepcopy(rendered) + if mutation == "missing": + del changed["services"]["worker"]["environment"][key] + else: + changed["services"]["worker"]["environment"][key] = "600" + with ( + self.subTest(mutation=mutation), + self.assertRaises(preflight.PreflightError), + ): + preflight.validate_compose(changed, values) + changed = copy.deepcopy(rendered) + del changed["services"]["aws-proof-runtime"]["environment"][key] + with self.assertRaisesRegex( + preflight.PreflightError, "preparation deadline" + ): + preflight.validate_compose(changed, values) + changed = copy.deepcopy(rendered) + for role in preflight.ROLE_SECRET_FILE_KEYS: + changed["services"][role]["environment"][key] = "600" + with self.assertRaisesRegex(preflight.PreflightError, key): + preflight.validate_compose(changed, values) + for deadline in ( + "", + "NaN", + "Infinity", + "-Infinity", + "300", + "3600", + "0", + "-1", + "true", + ): + with ( + self.subTest(deadline=deadline), + self.assertRaises(preflight.PreflightError), + ): + preflight.validate_environment(self.environment | {key: deadline}) + missing = dict(self.environment) + del missing[key] + with self.assertRaises(preflight.PreflightError): + preflight.validate_environment(missing) + def test_environment_rejects_mutable_or_development_settings(self) -> None: cases = { "mutable image": ( @@ -1175,25 +1362,390 @@ def test_checker_config_rejects_non_hosted_or_substituted_identity(self) -> None with self.assertRaises(preflight.PreflightError): preflight.validate_checker_config(changed, self.environment) - def test_model_runtime_requires_usable_host_socket(self) -> None: - config = self.checker | {"cache": []} - for arguments in ([], ["--host-uds=none"], ["--host-uds=open", "--host-uds=none"]): - with self.subTest(arguments=arguments), patch.object( - preflight, "_run_json", return_value={"runsc": {"runtimeArgs": arguments}} + def test_custom_deployment_root_preserves_paths_and_refuses_unsafe_custody( + self, + ) -> None: + root = self.root / "owned-data" + root.mkdir(mode=0o700) + values = self.environment | {"WORKSTREAM_DEPLOYMENT_ROOT": str(root)} + paths = preflight.deployment_paths(values) + config = copy.deepcopy(self.checker) + config.update( + socket_path=str(paths.socket), + material_root=str(paths.scratch), + model_proxy_socket=str(paths.model_socket), + model_relay_root=str(paths.model_relays), + ) + preflight.validate_environment(values) + preflight.validate_checker_config(config, values) + rendered = preflight.render_compose(self.environment_path, values) + preflight.validate_compose(rendered, values) + for role in ("api", "worker"): + environment = rendered["services"][role]["environment"] + self.assertEqual( + environment["WORKSTREAM_ARTIFACT_SCRATCH_ROOT"], str(paths.scratch) + ) + self.assertEqual( + environment["WORKSTREAM_EXTERNAL_CHECKER_SERVICE_SOCKET"], + str(paths.socket), + ) + self.assertEqual( + environment["WORKSTREAM_ARTIFACT_INTAKE_SERVICE_SOCKET"], + str(paths.socket), + ) + mounts = preflight._service_mounts(rendered["services"][role]) + self.assertEqual(mounts[str(paths.scratch)]["source"], str(paths.scratch)) + self.assertTrue(mounts[str(paths.socket_root)]["read_only"]) + proxy_mounts = preflight._service_mounts(rendered["services"]["model-proxy"]) + self.assertEqual( + proxy_mounts[str(paths.model_root)]["source"], str(paths.model_root) + ) + preflight._private_metadata( + root, kind="directory", uid=os.getuid(), gid=os.getgid(), mode=0o700 + ) + with self.assertRaisesRegex(preflight.PreflightError, "owner"): + preflight._private_metadata( + root, kind="directory", uid=os.getuid() + 1, gid=os.getgid(), mode=0o700 + ) + alias = self.root / "root-alias" + alias.symlink_to(root, target_is_directory=True) + for unsafe in ("/", str(alias), str(root / ".." / "owned-data")): + with self.subTest(root=unsafe), self.assertRaises(preflight.PreflightError): + preflight.validate_environment( + values | {"WORKSTREAM_DEPLOYMENT_ROOT": unsafe} + ) + with self.assertRaises(preflight.PreflightError): + preflight.validate_environment( + values | {"WORKSTREAM_CHECKER_SERVICE_BINARY": str(root / "service")} + ) + for field, path in ( + ("guest_binary", root / "guest"), + ("binary", root / "engine"), + ): + changed = copy.deepcopy(config) + changed["smol"][field] = str(path) + with self.assertRaisesRegex(preflight.PreflightError, "outside deployment"): + preflight.validate_checker_config(changed, values) + credential_values = values | { + "WORKSTREAM_MODEL_PROXY_OPENROUTER_API_KEY_FILE": str(root / "credential") + } + with self.assertRaisesRegex(preflight.PreflightError, "outside deployment"): + preflight.validate_checker_config(config, credential_values) + self.assertTrue(root.is_dir()) + + def test_native_socket_rejects_foreign_identity_permissions_and_links(self) -> None: + path = self.root / "native.sock" + listener = socket.socket(socket.AF_UNIX) + listener.bind(str(path)) + self.addCleanup(listener.close) + path.chmod(0o600) + with self.subTest(path=path): + preflight._validate_native_socket(path, os.getuid(), os.getgid()) + for uid, gid in ( + (os.getuid() + 1, os.getgid()), + (os.getuid(), os.getgid() + 1), ): - with self.assertRaisesRegex(preflight.PreflightError, "host-uds"): - preflight.validate_docker_runtime(config, self.environment) - with patch.object( - preflight, "_run_json", - return_value={"runsc": {"runtimeArgs": ["--host-uds=open"]}}, + with self.assertRaisesRegex(preflight.PreflightError, "identity"): + preflight._validate_native_socket(path, uid, gid) + path.chmod(0o660) + with self.assertRaises(preflight.PreflightError): + preflight._validate_native_socket(path, os.getuid(), os.getgid()) + path.chmod(0o600) + alias = self.root / "native-alias" + os.link(path, alias) + with self.assertRaises(preflight.PreflightError): + preflight._validate_native_socket(path, os.getuid(), os.getgid()) + alias.unlink() + preflight._validate_native_socket(path, os.getuid(), os.getgid()) + + def test_trial_manifest_is_paired_shared_and_confined_to_the_proxy(self) -> None: + manifest = self.root / "trial-authority.json" + manifest.write_text("fixture pinned authority", encoding="utf-8") + manifest.chmod(0o400) + self.environment.update( + { + preflight.TRIAL_KEYS[0]: str(manifest), + preflight.TRIAL_KEYS[1]: "sha256:" + + hashlib.sha256(manifest.read_bytes()).hexdigest(), + } + ) + preflight.validate_environment(self.environment) + rendered = preflight.render_compose(self.environment_path, self.environment) + preflight.validate_compose(rendered, self.environment) + self.assertEqual( + preflight._service_mounts(rendered["services"]["model-proxy"])[ + str(preflight.TRIAL_MANIFEST_TARGET) + ]["source"], + str(manifest), + ) + for role in ("api", "worker", "beat"): + self.assertFalse( + set(preflight.TRIAL_KEYS) + & set(rendered["services"][role]["environment"]) + ) + for mutation in ("writable", "foreign-source", "worker-selector"): + changed = copy.deepcopy(rendered) + mount = next( + item + for item in changed["services"]["model-proxy"]["volumes"] + if item["target"] == str(preflight.TRIAL_MANIFEST_TARGET) + ) + if mutation == "writable": + mount["read_only"] = False + elif mutation == "foreign-source": + mount["source"] = "/tmp/foreign-manifest" + else: + changed["services"]["worker"]["environment"][ + preflight.TRIAL_KEYS[0] + ] = str(preflight.TRIAL_MANIFEST_TARGET) + with self.assertRaises(preflight.PreflightError): + preflight.validate_compose(changed, self.environment) + native = copy.deepcopy(self.checker) + native.update( + trial_authority_manifest_path=str(manifest), + trial_authority_manifest_digest=self.environment[preflight.TRIAL_KEYS[1]], + ) + native["trial_runtime"] = { + **{ + key: str(self.root / key) + for key in ( + "relay_root", + "scratch_root", + "child_home_root", + "offline_cache_root", + "trusted_launcher_path", + "filesystem_tool_path", + ) + }, + **{ + key: "sha256:" + "f" * 64 + for key in ( + "outer_runtime_config_digest", + "trusted_launcher_digest", + "filesystem_tool_digest", + ) + }, + "builder_image_id": native["cache"][0]["config_image_id"], + "deployment_owner": "hosted-native", + "maximum_candidate_archive_bytes": 1024, + } + native["trial_runtime"]["child_home_root"] = self.enterContext( + tempfile.TemporaryDirectory(prefix="ws", dir="/tmp") + ) + preflight.validate_checker_config(native, self.environment) + for root, valid in ( + ("/" + "a" * 17, True), + ("/" + "é" * 8 + "a", True), + ("/" + "a" * 18, False), + ("/" + "é" * 9, False), ): - preflight.validate_docker_runtime(config, self.environment) - with patch.object( - preflight, "_run_json", return_value={"runsc": {"runtimeArgs": []}} + changed = copy.deepcopy(native) + changed["trial_runtime"]["child_home_root"] = root + with self.subTest(child_home_root=root): + if valid: + preflight.validate_checker_config(changed, self.environment) + else: + with self.assertRaisesRegex(preflight.PreflightError, "18-byte"): + preflight.validate_checker_config(changed, self.environment) + substituted = native | {"trial_authority_manifest_digest": "sha256:" + "0" * 64} + with self.assertRaisesRegex(preflight.PreflightError, "trial authority"): + preflight.validate_checker_config(substituted, self.environment) + for key in preflight.TRIAL_KEYS: + changed = dict(self.environment) + del changed[key] + with self.assertRaises(preflight.PreflightError): + preflight.validate_environment(changed) + with self.assertRaisesRegex(preflight.PreflightError, "trial authority"): + preflight.validate_checker_config(self.checker, self.environment) + + def test_smol_home_socket_budget_counts_filesystem_bytes(self): + for root, valid in ( + ("/" + "a" * 58, True), + ("/" + "é" * 29, True), + ("/" + "a" * 59, False), + ("/" + "é" * 30, False), + ): + changed = copy.deepcopy(self.checker) + changed["smol"]["home"] = root + with self.subTest(home=root): + if valid: + preflight.validate_checker_config(changed, self.environment) + else: + with self.assertRaisesRegex(preflight.PreflightError, "59-byte"): + preflight.validate_checker_config(changed, self.environment) + + def _native_inputs(self): + config = copy.deepcopy(self.checker) + values = self.environment | { + "WORKSTREAM_RUNTIME_UID": str(os.getuid()), + "WORKSTREAM_RUNTIME_GID": str(os.getgid()), + } + smol = config["smol"] + home = self.root / "native-home" + home.mkdir(mode=0o700) + group = self.root / "delegated" + group.mkdir(mode=0o700) + (group / "controller").mkdir(mode=0o700) + for key in ("cgroup.controllers", "cgroup.subtree_control"): + (group / key).write_text("cpu memory pids", encoding="ascii") + smol.update(home=str(home), cgroup_root=str(group)) + for key, digest_key in ( + ("binary", "binary_sha256"), + ("guest_binary", "guest_sha256"), + ): + path = self.root / key + raw = key.encode() + path.write_bytes(raw) + path.chmod(0o400) + smol[key] = str(path) + smol[digest_key] = "sha256:" + hashlib.sha256(raw).hexdigest() + for index, entry in enumerate(config["cache"]): + for key, digest_key in ( + ("archive", "archive_sha256"), + ("image_config", "config_image_id"), + ): + path = self.root / f"cache-{index}-{key}" + raw = f"static native custody fixture {index} {key}".encode() + path.write_bytes(raw) + path.chmod(0o400) + entry["smol"][key] = str(path) + target = entry["smol"] if key == "archive" else entry + target[digest_key] = "sha256:" + hashlib.sha256(raw).hexdigest() + values[preflight.INTAKE_DIGEST_KEYS[1]] = config["cache"][1]["config_image_id"] + service = self.root / "service-binary" + service.write_bytes(b"pinned installed native service") + service.chmod(0o400) + values["WORKSTREAM_CHECKER_SERVICE_BINARY"] = str(service) + values["WORKSTREAM_CHECKER_SERVICE_BINARY_SHA256"] = ( + "sha256:" + hashlib.sha256(service.read_bytes()).hexdigest() + ) + health = { + "runtime": "smolvm", + "isolation_mode": "smol-microvm", + "operating_system": "linux", + "architecture": "amd64", + "sandbox_uid": os.getuid(), + "sandbox_gid": os.getgid(), + "cached_platform_manifests": len(config["cache"]), + } + return config, values, service, health + + def test_native_asset_refusals_preserve_operator_diagnostics(self) -> None: + for failure, reason in ( + ("path", "path is not canonical"), + ("custody", "custody is unsafe"), + ("deadline", "verification deadline exceeded"), + ("digest", "digest or identity differs"), + ("missing", "is unavailable"), + ): + with self.subTest(failure=failure): + asset = self.root / f"asset-{failure}" + asset.write_bytes(b"pinned asset") + asset.chmod(0o400) + expected = "sha256:" + hashlib.sha256(asset.read_bytes()).hexdigest() + deadline = float("inf") + if failure == "path": + alias = self.root / "asset-alias" + alias.symlink_to(asset) + asset = alias + elif failure == "custody": + asset.chmod(0o620) + elif failure == "deadline": + deadline = 0 + elif failure == "digest": + expected = "sha256:" + "0" * 64 + else: + asset.unlink() + with self.assertRaisesRegex( + preflight.PreflightError, f"^native Smol asset {reason}$" + ): + preflight._verify_smol_asset(asset, expected, os.getuid(), deadline) + + def test_native_readiness_binds_static_assets_delegation_and_typed_health( + self, + ) -> None: + config, values, service, health = self._native_inputs() + with ( + patch.object(preflight, "_run_json", return_value=health) as run, + ): + preflight.validate_smol_runtime(config, values) + command = run.call_args.args[0] + self.assertIn("configured_external_checker_execution(s).health()", command[-1]) + self.assertIn("--network", command) + self.assertEqual(command[command.index("--network") + 1], "none") + self.assertIn("readonly", command[command.index("--mount") + 1]) + self.assertNotIn("--env", command) + self.assertNotIn("--env-file", command) + self.assertNotIn("/var/run/docker.sock", " ".join(command)) + for changed_health in ( + health | {"runtime": "runsc"}, + health | {"sandbox_uid": os.getuid() + 1}, + health | {"cached_platform_manifests": 1}, ): - preflight.validate_docker_runtime( - config, self.environment | {"WORKSTREAM_MODEL_PROXY_ENABLED": "false"} + with ( + patch.object(preflight, "_run_json", return_value=changed_health), + ): + with self.assertRaisesRegex( + preflight.PreflightError, "readiness identity" + ): + preflight.validate_smol_runtime(config, values) + for mutate in ("cache", "binary", "delegation"): + changed = copy.deepcopy(config) + if mutate == "cache": + changed["cache"][0]["smol"]["archive_sha256"] = "sha256:" + "f" * 64 + elif mutate == "binary": + changed["smol"]["guest_sha256"] = "sha256:" + "f" * 64 + else: + Path(changed["smol"]["cgroup_root"]).joinpath( + "cgroup.subtree_control" + ).write_text("cpu memory", encoding="ascii") + with ( + patch.object(preflight, "_run_json") as run, + ): + with self.assertRaises(preflight.PreflightError): + preflight.validate_smol_runtime(changed, values) + run.assert_not_called() + + def test_cold_start_prerequisites_never_release_application_or_require_node( + self, + ) -> None: + for arguments in (("migrate",), ("up", "-d", "model-proxy")): + with ( + self.subTest(arguments=arguments), + patch.object(compose, "validate_private_host_inputs") as private, + patch.object(compose, "validate_smol_runtime") as native, + patch.object(compose, "validate_trusted_intake") as intake, + patch.object(compose, "run_compose", return_value=0) as run, + ): + self.assertEqual(self._compose_main(*arguments), (0, "")) + private.assert_called_once_with(self.environment, require_node=False) + native.assert_not_called() + intake.assert_not_called() + run.assert_called_once_with( + self.environment_path, self.environment, list(arguments) + ) + completed = subprocess.CompletedProcess([], 0) + with patch.object(preflight.subprocess, "run", return_value=completed) as run: + self.assertEqual( + preflight.run_compose( + self.environment_path, self.environment, ["migrate"] + ), + 0, ) + self.assertEqual( + run.call_args.args[0][-9:], + [ + "run", + "--rm", + "--no-deps", + "api", + "python", + "-m", + "alembic", + "upgrade", + "head", + ], + ) def test_checker_model_paths_require_enabled_exact_proxy_topology(self) -> None: for key in ("model_proxy_socket", "model_relay_root"): @@ -1211,28 +1763,13 @@ def test_checker_model_paths_require_enabled_exact_proxy_topology(self) -> None: del ordinary["model_relay_root"] preflight.validate_checker_config(ordinary, disabled) - def test_rendered_checker_model_relay_cannot_lose_custody(self) -> None: + def test_rendered_topology_rejects_a_second_checker_controller(self) -> None: rendered = preflight.render_compose(self.environment_path, self.environment) - for target, read_only in ( - (str(preflight.MODEL_SOCKET_ROOT), False), - (str(preflight.MODEL_RELAY_ROOT), True), - ): - for change in ("remove", "mode", "source"): - with self.subTest(target=target, change=change): - changed = copy.deepcopy(rendered) - mounts = changed["services"]["checker-service"]["volumes"] - selected = next(m for m in mounts if m["target"] == target) - if change == "remove": - mounts.remove(selected) - elif change == "mode": - selected["read_only"] = read_only - else: - selected["source"] = "/tmp/substituted" - with self.assertRaises(preflight.PreflightError): - preflight.validate_compose(changed, self.environment) changed = copy.deepcopy(rendered) - del changed["services"]["checker-service"]["depends_on"]["model-proxy"] - with self.assertRaises(preflight.PreflightError): + changed["services"]["checker-service"] = copy.deepcopy( + rendered["services"]["worker"] + ) + with self.assertRaisesRegex(preflight.PreflightError, "bounded topology"): preflight.validate_compose(changed, self.environment) def test_rendered_topology_rejects_privilege_and_path_drift(self) -> None: @@ -1248,34 +1785,14 @@ def test_rendered_topology_rejects_privilege_and_path_drift(self) -> None: "registry.example/workstream/backend@sha256:" + "9" * 64 ) cases.append(("different guide parser image", different_parser_image)) - networked = copy.deepcopy(rendered) - networked["services"]["checker-service"]["network_mode"] = "default" - cases.append(("networked checker", networked)) - privileged = copy.deepcopy(rendered) - privileged["services"]["checker-service"]["cap_drop"] = [] - cases.append(("checker capabilities", privileged)) - writable_material = copy.deepcopy(rendered) - for mount in writable_material["services"]["checker-service"]["volumes"]: - if mount["target"] == str(preflight.SCRATCH_ROOT): - mount["read_only"] = False - cases.append(("writable checker material", writable_material)) - volatile_owner = copy.deepcopy(rendered) - for mount in volatile_owner["services"]["checker-service"]["volumes"]: - if mount["target"] == str(preflight.SOCKET_ROOT): - mount["source"] = "/run/workstream-checker" - cases.append(("boot-volatile checker custody", volatile_owner)) for role in ("api", "worker"): writable_owner = copy.deepcopy(rendered) for mount in writable_owner["services"][role]["volumes"]: - if mount["target"] == str(preflight.SOCKET_ROOT): + if mount["target"] == str( + preflight.deployment_paths(self.environment).socket_root + ): mount["read_only"] = False cases.append((f"{role} writable checker custody", writable_owner)) - root_controller = copy.deepcopy(rendered) - root_controller["services"]["checker-service"]["user"] = "0:0" - cases.append(("root checker controller", root_controller)) - extra_group = copy.deepcopy(rendered) - extra_group["services"]["checker-service"]["group_add"].append("10002") - cases.append(("extra checker group", extra_group)) expanded_worker = copy.deepcopy(rendered) expanded_worker["services"]["worker"]["environment"]["UNREVIEWED"] = "true" cases.append(("worker environment drift", expanded_worker)) @@ -1340,11 +1857,6 @@ def test_rendered_topology_rejects_privilege_and_path_drift(self) -> None: cases.append( ("hardlink runtime identity exposed to Beat", hardlink_beat_identity) ) - checker_identity = copy.deepcopy(rendered) - checker_identity["services"]["checker-service"]["volumes"].append( - copy.deepcopy(rendered["services"]["api"]["volumes"][-1]) - ) - cases.append(("runtime identity exposed to checker", checker_identity)) parser_secret = copy.deepcopy(rendered) parser_secret["services"]["guide-document-parser"]["secrets"] = [ copy.deepcopy(rendered["services"]["worker"]["secrets"][-1]) @@ -1416,7 +1928,7 @@ def test_rendered_topology_rejects_privilege_and_path_drift(self) -> None: proof_parser_socket = copy.deepcopy(rendered) proof_parser_socket["services"]["aws-proof-coordinate"]["environment"][ "WORKSTREAM_GUIDE_DOCUMENT_PARSER_SOCKET" - ] = str(preflight.GUIDE_SOCKET_PATH) + ] = str(preflight.deployment_paths(self.environment).guide_socket) cases.append(("guide parser socket exposed to proof", proof_parser_socket)) proof_provider_selector = copy.deepcopy(rendered) proof_provider_selector["services"]["aws-proof-coordinate"]["environment"][ @@ -1489,33 +2001,30 @@ def test_private_metadata_rejects_empty_broad_and_linked_files(self) -> None: mode=0o400, ) - def test_private_host_inputs_check_exact_socket_group_without_reading_secrets( + def test_private_host_inputs_check_socket_identity_without_reading_secrets( self, ) -> None: scratch = self.root / "scratch" - socket_root = self.root / "socket" - model_socket_root = self.root / "model-socket" + socket_root = self.root / "checker-runtime" + model_socket_root = self.root / "model-runtime" model_relay_root = self.root / "model-relays" - guide_socket_root = self.root / "guide-socket" - intake_root = self.root / "intake" + guide_socket_root = self.root / "guide-runtime" + intake_root = self.root / "checker-runtime" intake_root.mkdir(mode=0o700) intake_path = intake_root / "checker.sock" intake_listener = socket.socket(socket.AF_UNIX) intake_listener.bind(str(intake_path)) intake_path.chmod(0o600) self.addCleanup(intake_listener.close) - for field, value in (("INTAKE_SOCKET_ROOT", intake_root), ("INTAKE_SOCKET_PATH", intake_path)): - owned_patch = patch.object(preflight, field, value) - owned_patch.start() - self.addCleanup(owned_patch.stop) scratch.mkdir(mode=0o700) - socket_root.mkdir(mode=0o700) - model_socket_root.mkdir(mode=0o700) + socket_root.mkdir(mode=0o700, exist_ok=True) + model_socket_root.mkdir(mode=0o700, exist_ok=True) model_relay_root.mkdir(mode=0o700) guide_socket_root.mkdir(mode=0o750) host_uid = os.getuid() host_gid = os.getgid() environment = dict(self.environment) + environment["WORKSTREAM_DEPLOYMENT_ROOT"] = str(self.root) environment["WORKSTREAM_RUNTIME_UID"] = str(host_uid) environment["WORKSTREAM_RUNTIME_GID"] = str(host_gid) checker = self.root / "private-checker.json" @@ -1533,25 +2042,7 @@ def test_private_host_inputs_check_exact_socket_group_without_reading_secrets( listener = socket.socket(socket.AF_UNIX) listener.bind(str(docker_socket)) try: - environment["WORKSTREAM_DOCKER_SOCKET_GID"] = str( - docker_socket.stat().st_gid - ) with ( - patch.object(preflight, "SCRATCH_ROOT", scratch), - patch.object(preflight, "SOCKET_ROOT", socket_root), - patch.object(preflight, "MODEL_SOCKET_ROOT", model_socket_root), - patch.object(preflight, "MODEL_RELAY_ROOT", model_relay_root), - patch.object( - preflight, - "MODEL_SOCKET_PATH", - model_socket_root / "model.sock", - ), - patch.object(preflight, "GUIDE_SOCKET_ROOT", guide_socket_root), - patch.object( - preflight, - "GUIDE_SOCKET_PATH", - guide_socket_root / "guide.sock", - ), patch.object(preflight, "GUIDE_PARSER_UID", host_uid), patch.object(preflight, "GUIDE_PARSER_GID", host_gid), patch.object(preflight, "DOCKER_SOCKET", docker_socket), @@ -1591,14 +2082,6 @@ def test_private_host_inputs_check_exact_socket_group_without_reading_secrets( with self.assertRaisesRegex(preflight.PreflightError, "wrong mode"): preflight.validate_private_host_inputs(environment) guide_socket_root.chmod(0o750) - environment["WORKSTREAM_DOCKER_SOCKET_GID"] = str( - docker_socket.stat().st_gid + 1 - ) - with self.assertRaises(preflight.PreflightError): - preflight.validate_private_host_inputs(environment) - environment["WORKSTREAM_DOCKER_SOCKET_GID"] = str( - docker_socket.stat().st_gid - ) scratch_secret = scratch / "project-agent-secret" scratch_secret.write_text("private", encoding="utf-8") scratch_secret.chmod(0o400) @@ -1608,21 +2091,6 @@ def test_private_host_inputs_check_exact_socket_group_without_reading_secrets( scratch_alias / scratch_secret.name ) with ( - patch.object(preflight, "SCRATCH_ROOT", scratch), - patch.object(preflight, "SOCKET_ROOT", socket_root), - patch.object(preflight, "MODEL_SOCKET_ROOT", model_socket_root), - patch.object(preflight, "MODEL_RELAY_ROOT", model_relay_root), - patch.object( - preflight, - "MODEL_SOCKET_PATH", - model_socket_root / "model.sock", - ), - patch.object(preflight, "GUIDE_SOCKET_ROOT", guide_socket_root), - patch.object( - preflight, - "GUIDE_SOCKET_PATH", - guide_socket_root / "guide.sock", - ), patch.object(preflight, "GUIDE_PARSER_UID", host_uid), patch.object(preflight, "GUIDE_PARSER_GID", host_gid), patch.object(preflight, "DOCKER_SOCKET", docker_socket), @@ -1643,21 +2111,6 @@ def test_private_host_inputs_check_exact_socket_group_without_reading_secrets( original_negative_key = environment[negative_key] environment[negative_key] = str(socket_identity) with ( - patch.object(preflight, "SCRATCH_ROOT", scratch), - patch.object(preflight, "SOCKET_ROOT", socket_root), - patch.object(preflight, "MODEL_SOCKET_ROOT", model_socket_root), - patch.object(preflight, "MODEL_RELAY_ROOT", model_relay_root), - patch.object( - preflight, - "MODEL_SOCKET_PATH", - model_socket_root / "model.sock", - ), - patch.object(preflight, "GUIDE_SOCKET_ROOT", guide_socket_root), - patch.object( - preflight, - "GUIDE_SOCKET_PATH", - guide_socket_root / "guide.sock", - ), patch.object(preflight, "GUIDE_PARSER_UID", host_uid), patch.object(preflight, "GUIDE_PARSER_GID", host_gid), patch.object(preflight, "DOCKER_SOCKET", docker_socket), @@ -1675,21 +2128,6 @@ def test_private_host_inputs_check_exact_socket_group_without_reading_secrets( os.link(secret_paths[-1], secret_alias) environment[runtime_key] = str(secret_alias) with ( - patch.object(preflight, "SCRATCH_ROOT", scratch), - patch.object(preflight, "SOCKET_ROOT", socket_root), - patch.object(preflight, "MODEL_SOCKET_ROOT", model_socket_root), - patch.object(preflight, "MODEL_RELAY_ROOT", model_relay_root), - patch.object( - preflight, - "MODEL_SOCKET_PATH", - model_socket_root / "model.sock", - ), - patch.object(preflight, "GUIDE_SOCKET_ROOT", guide_socket_root), - patch.object( - preflight, - "GUIDE_SOCKET_PATH", - guide_socket_root / "guide.sock", - ), patch.object(preflight, "GUIDE_PARSER_UID", host_uid), patch.object(preflight, "GUIDE_PARSER_GID", host_gid), patch.object(preflight, "DOCKER_SOCKET", docker_socket), @@ -1701,21 +2139,6 @@ def test_private_host_inputs_check_exact_socket_group_without_reading_secrets( os.link(checker, checker_alias) environment[negative_key] = str(checker_alias) with ( - patch.object(preflight, "SCRATCH_ROOT", scratch), - patch.object(preflight, "SOCKET_ROOT", socket_root), - patch.object(preflight, "MODEL_SOCKET_ROOT", model_socket_root), - patch.object(preflight, "MODEL_RELAY_ROOT", model_relay_root), - patch.object( - preflight, - "MODEL_SOCKET_PATH", - model_socket_root / "model.sock", - ), - patch.object(preflight, "GUIDE_SOCKET_ROOT", guide_socket_root), - patch.object( - preflight, - "GUIDE_SOCKET_PATH", - guide_socket_root / "guide.sock", - ), patch.object(preflight, "GUIDE_PARSER_UID", host_uid), patch.object(preflight, "GUIDE_PARSER_GID", host_gid), patch.object(preflight, "DOCKER_SOCKET", docker_socket), @@ -1733,21 +2156,6 @@ def test_private_host_inputs_check_exact_socket_group_without_reading_secrets( } ) with ( - patch.object(preflight, "SCRATCH_ROOT", scratch), - patch.object(preflight, "SOCKET_ROOT", socket_root), - patch.object(preflight, "MODEL_SOCKET_ROOT", model_socket_root), - patch.object(preflight, "MODEL_RELAY_ROOT", model_relay_root), - patch.object( - preflight, - "MODEL_SOCKET_PATH", - model_socket_root / "model.sock", - ), - patch.object(preflight, "GUIDE_SOCKET_ROOT", guide_socket_root), - patch.object( - preflight, - "GUIDE_SOCKET_PATH", - guide_socket_root / "guide.sock", - ), patch.object(preflight, "GUIDE_PARSER_UID", host_uid), patch.object(preflight, "GUIDE_PARSER_GID", host_gid), patch.object(preflight, "DOCKER_SOCKET", docker_socket), @@ -1757,7 +2165,6 @@ def test_private_host_inputs_check_exact_socket_group_without_reading_secrets( finally: listener.close() - def test_trusted_intake_requires_pinned_settings_and_readonly_mounts(self) -> None: for key in preflight.INTAKE_DIGEST_KEYS: for value in (None, "latest"): @@ -1766,21 +2173,29 @@ def test_trusted_intake_requires_pinned_settings_and_readonly_mounts(self) -> No del changed[key] else: changed[key] = value - with self.subTest(key=key, value=value), self.assertRaises(preflight.PreflightError): + with ( + self.subTest(key=key, value=value), + self.assertRaises(preflight.PreflightError), + ): preflight.validate_environment(changed) rendered = preflight.render_compose(self.environment_path, self.environment) for role in ("api", "worker"): changed = copy.deepcopy(rendered) for mount in changed["services"][role]["volumes"]: - if mount["target"] == str(preflight.INTAKE_SOCKET_ROOT): + if mount["target"] == str( + preflight.deployment_paths(self.environment).socket_root + ): mount["read_only"] = False with self.subTest(role=role), self.assertRaises(preflight.PreflightError): preflight.validate_compose(changed, self.environment) def test_trusted_intake_health_uses_typed_factory_without_credentials(self) -> None: - health = {"runtime": "smolvm", "isolation_mode": "smol-microvm", - "image_digest": self.environment[preflight.INTAKE_DIGEST_KEYS[0]], - "config_digest": self.environment[preflight.INTAKE_DIGEST_KEYS[1]]} + health = { + "runtime": "smolvm", + "isolation_mode": "smol-microvm", + "image_digest": self.environment[preflight.INTAKE_DIGEST_KEYS[0]], + "config_digest": self.environment[preflight.INTAKE_DIGEST_KEYS[1]], + } with patch.object(preflight, "_run_json", return_value=health) as run: preflight.validate_trusted_intake(self.environment) command = run.call_args.args[0] @@ -2039,6 +2454,26 @@ def test_loader_loads_only_role_secrets_and_rejects_missing_or_unknown_role( self.assertEqual(unknown.stdout, "") self.assertIn("release role is unavailable or unsupported", unknown.stderr) + def test_api_entrypoint_executes_without_node_proxy_or_model_credentials( + self, + ) -> None: + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + missing_socket = root / "not-started.sock" + with patch.dict( + os.environ, + { + "WORKSTREAM_ARTIFACT_INTAKE_SERVICE_SOCKET": str(missing_socket), + "WORKSTREAM_EXTERNAL_CHECKER_SERVICE_SOCKET": str(missing_socket), + "WORKSTREAM_CHECKER_MODEL_PROXY_SOCKET": str(missing_socket), + }, + ): + completed = self._run_loader(root, role="api", provider_files=()) + self.assertEqual(completed.returncode, 0, completed.stderr) + self.assertEqual(completed.stdout, "loader-ok\n") + self.assertEqual(completed.stderr, "") + self.assertFalse(missing_socket.exists()) + def test_loader_binds_runtime_to_sdk_marker_before_provider_secret(self) -> None: with tempfile.TemporaryDirectory() as temporary: root = Path(temporary) @@ -2145,7 +2580,5 @@ def test_loader_rejects_empty_missing_directory_and_symlink_without_disclosure( self.assertNotIn("do-not-print", completed.stderr) - - if __name__ == "__main__": unittest.main() diff --git a/scripts/test_hosted_release_images.py b/scripts/test_hosted_release_images.py index d0ac76bef..5abfc7117 100644 --- a/scripts/test_hosted_release_images.py +++ b/scripts/test_hosted_release_images.py @@ -5,6 +5,7 @@ import json from pathlib import Path import re +import sys import tempfile import unittest from unittest.mock import patch @@ -479,6 +480,7 @@ def test_evidence_is_closed_and_never_overwrites_a_symlink_target(self): ) } evidence.update( + proof_scope="all-release-images", source_sha="b" * 40, tree_sha="c" * 40, backend_agents_image_id="sha256:" + "d" * 64, @@ -500,6 +502,95 @@ def test_evidence_is_closed_and_never_overwrites_a_symlink_target(self): proof.write_evidence(linked, evidence) self.assertEqual(json.loads(output.read_text()), evidence) + def test_native_hosted_proves_both_backends_without_controller_image(self): + head = "a" * 40 + for checker_image in (False, True): + arguments = [ + "verify_hosted_release_images.py", + "--scope", + "native-hosted", + "--expected-head", + head, + "--container-owner", + "123-1", + "--backend-agents-image", + "agents", + "--backend-omnicore-image", + "omnicore", + "--evidence", + "/unused/evidence.json", + ] + if checker_image: + arguments.extend(["--checker-image", "controller"]) + + def checked(command): + if command[:3] == ["git", "status", "--porcelain"]: + return b"" + return (head + "\n").encode() + + omnicore = self.backend | {"sdk_environment": "omnicore"} + with ( + self.subTest(checker_image=checker_image), + patch.object(sys, "argv", arguments), + patch.object(proof, "checked", side_effect=checked), + patch.object(proof, "source_inventory", return_value=self.source), + patch.object(proof, "expected_packages", return_value=self.packages), + patch.object(proof, "file_hash", return_value=self.loader), + patch.object( + proof, "image_metadata", side_effect=lambda image, role: image + ) as metadata, + patch.object( + proof, "inspect_image", side_effect=[self.backend, omnicore] + ) as inspect, + patch.object(proof, "verify_backend_entrypoint") as entrypoint, + patch.object(proof, "verify_checker") as checker, + patch.object(proof, "write_evidence") as write, + ): + if checker_image: + with self.assertRaisesRegex(ValueError, "checker image"): + proof.main() + metadata.assert_not_called() + write.assert_not_called() + continue + proof.main() + self.assertEqual(metadata.call_args_list[0].args, ("agents", "backend")) + self.assertEqual( + metadata.call_args_list[1].args, ("omnicore", "backend") + ) + self.assertEqual(inspect.call_count, 2) + self.assertEqual(entrypoint.call_count, 2) + checker.assert_not_called() + evidence = write.call_args.args[1] + self.assertEqual(evidence["proof_scope"], "native-hosted") + self.assertFalse(any(key.startswith("checker_") for key in evidence)) + + def test_native_hosted_evidence_cannot_claim_controller_or_node_attestation(self): + evidence = { + "proof_scope": "native-hosted", + "source_sha": "a" * 40, + "tree_sha": "b" * 40, + "backend_agents_image_id": "sha256:" + "c" * 64, + "backend_omnicore_image_id": "sha256:" + "d" * 64, + **{ + key: "e" * 64 + for key in ( + "backend_source_sha256", + "backend_lock_sha256", + "loader_sha256", + "roles_anywhere_helper_sha256", + ) + }, + } + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "evidence.json" + proof.write_evidence(path, evidence) + self.assertEqual(json.loads(path.read_text()), evidence) + for key in ("checker_image_id", "native_service_sha256", "guest_sha256"): + with self.subTest(key=key), self.assertRaisesRegex(ValueError, "shape"): + proof.write_evidence( + Path(directory) / "bad.json", evidence | {key: "f" * 64} + ) + if __name__ == "__main__": unittest.main() diff --git a/scripts/test_public_hosted_attachment.py b/scripts/test_public_hosted_attachment.py new file mode 100644 index 000000000..b6011c35b --- /dev/null +++ b/scripts/test_public_hosted_attachment.py @@ -0,0 +1,528 @@ +"""Hosted attachment refuses substitute services and observes real native copies.""" + +import copy +import asyncio +import hashlib +import json +from itertools import count as sequence +from pathlib import Path +import shutil +import subprocess +import sys +import tempfile +import threading +import unittest +from unittest.mock import AsyncMock, patch +from types import SimpleNamespace +from uuid import uuid4 + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) +sys.path[:0] = [str(ROOT / 'backend'), str(ROOT / 'backend/scripts'), str(ROOT / 'backend/tests')] +from experiments.pilot04_external_checker_runtime.public_hosted_attachment import ( # noqa: E402 + validate_container, + HostedAttachment, manager_rules, deliver_duplicate, +) + + +class HostedDuplicateEvents(unittest.TestCase): + def observe(self, event_change=None, *, already_enabled=False, restore_reply=None): + from celery import Celery + from kombu.transport.memory import Channel + + app = Celery('hosted-duplicate-proof', broker='memory://') + self.addCleanup(app.close) + hostname = 'worker@owned-fixture' + stopped = threading.Event() + errors = [] + commands = [] + published = [] + listeners = [] + receiver_type = app.events.Receiver + + def receiver(connection, **options): + listeners.append(connection) + return receiver_type(connection, **options) + + def heartbeats(): + try: + with app.connection_for_write() as connection: + with app.events.Dispatcher(connection, hostname=hostname) as dispatcher: + while not stopped.is_set(): + dispatcher.send('worker-heartbeat') + stopped.wait(.01) + except Exception as error: + errors.append(error) + + def control(command, **options): + if command == 'enable_events': + # The in-memory transport's queue registry is process-global; + # finish its heartbeat producer before queue deletion. Real Redis + # separately qualifies concurrent worker heartbeats and control. + stopped.set() + heartbeat_thread.join(timeout=2) + self.assertFalse(heartbeat_thread.is_alive()) + self.assertEqual(len(listeners), 1) + self.assertIsNot(options['connection'], listeners[0], + 'Redis event subscription must not share pidbox control connection') + self.assertEqual(options['destination'], [hostname]) + self.assertTrue(options['reply']) + self.assertEqual(options['timeout'], 5) + commands.append(command) + reply = ({'ok': 'task events already enabled'} if already_enabled + else {'ok': 'task events enabled'}) if command == 'enable_events' else ( + restore_reply or {'ok': 'task events disabled'}) + return [{hostname: reply}] + + def publish(**options): + published.append(options) + self.assertEqual(commands, ['enable_events']) + self.assertFalse(options['retry']) + self.assertTrue(options['ignore_result']) + identifier = options['task_id'] + queue = app.conf.event_queue_prefix + '.ws-proof-' + identifier + self.assertIn(queue, Channel.queues) + events = [ + ('task-received', {'uuid': str(uuid4()), 'name': 'unrelated'}), + ('task-received', {'uuid': identifier, 'name': task.name}), + ('task-succeeded', {'uuid': identifier, 'result': "{'status': 'delivery_unavailable'}"}), + ] + if event_change: + event_change(events, identifier) + with app.connection_for_write() as connection: + for kind, fields in events: + worker = fields.pop('_worker', hostname) + with app.events.Dispatcher(connection, hostname=worker) as dispatcher: + dispatcher.send(kind, **fields) + + task = SimpleNamespace(app=app, name='workstream.outbox.deliver_event', apply_async=publish) + heartbeat_thread = threading.Thread(target=heartbeats, daemon=True) + heartbeat_thread.start() + try: + with patch.object(app.events, 'Receiver', side_effect=receiver), \ + patch.object(app.control, 'broadcast', side_effect=control): + result = deliver_duplicate(task, hostname, 'event-id', 'project-id') + self.assertEqual(result, {'task_id': published[0]['task_id'], + 'result': {'status': 'delivery_unavailable'}}) + self.assertEqual(published[0]['args'], ('event-id', 'project-id')) + return result + finally: + stopped.set() + heartbeat_thread.join(timeout=2) + self.assertFalse(heartbeat_thread.is_alive()) + self.assertFalse(errors) + self.assertEqual(commands, ['enable_events'] if already_enabled + else ['enable_events', 'disable_events']) + for item in published: + self.assertNotIn(app.conf.event_queue_prefix + '.ws-proof-' + item['task_id'], Channel.queues) + + def test_real_event_queue_is_armed_before_dispatch_and_then_removed(self): + self.observe() + self.observe(already_enabled=True) + + def test_wrong_task_result_failure_and_terminal_before_received_are_refused(self): + def wrong_name(events, identifier): + events[1][1]['name'] = 'another.task' + def wrong_result(events, identifier): + events[2][1]['result'] = "{'status': 'delivered'}" + def wrong_worker(events, identifier): + events[2][1]['_worker'] = 'worker@another-fixture' + def failed(events, identifier): + events[2] = ('task-failed', {'uuid': identifier}) + def reversed_order(events, identifier): + events[1], events[2] = events[2], events[1] + for change in (wrong_name, wrong_result, wrong_worker, failed, reversed_order): + with self.subTest(change=change.__name__), self.assertRaises(RuntimeError): + self.observe(change) + + def test_absolute_completion_deadline_is_enforced(self): + def elapsed(events, identifier): + timer = patch('experiments.pilot04_external_checker_runtime.public_hosted_attachment.time.monotonic', + side_effect=lambda: next(ticks)) + ticks = sequence(step=121) + timer.start() + self.addCleanup(timer.stop) + with self.assertRaisesRegex(RuntimeError, 'completion was not observed'): + self.observe(elapsed) + + def test_failed_event_restore_cannot_claim_success(self): + with self.assertRaisesRegex(RuntimeError, 'not restored'): + self.observe(restore_reply={'error': 'not restored'}) + + +class HostedContainerCustody(unittest.TestCase): + def setUp(self): + self.rendered = { + 'image': 'example/backend@sha256:' + 'a' * 64, 'user': '10001:10001', + 'command': ['python', '-m', 'celery', '-A', 'app.workers.celery_app', 'worker', '--pool=prefork'], + 'environment': {'WORKSTREAM_ENVIRONMENT': 'production', + 'WORKSTREAM_PROJECT_AGENT_RUNTIME': 'manager_authored'}, + 'volumes': [{'type': 'bind', 'source': '/owned/scratch', 'target': '/owned/scratch'}], + 'secrets': [{'source': 'database', 'target': 'database_url'}], + 'networks': {'default': None}, + } + self.image = {'Id': 'sha256:' + 'b' * 64, 'Config': { + 'Entrypoint': ['/usr/local/bin/workstream-release-entrypoint'], 'Env': ['PATH=/usr/bin']}} + self.container = {'Id': 'exact-worker', 'Image': self.image['Id'], + 'State': {'Running': True, 'Health': {'Status': 'healthy'}}, + 'Config': {'Image': self.rendered['image'], 'User': self.rendered['user'], + 'Cmd': self.rendered['command'], + 'Entrypoint': self.image['Config']['Entrypoint'], + 'Env': ['PATH=/usr/bin'] + [k + '=' + v for k, v in self.rendered['environment'].items()]}, + 'HostConfig': {'ReadonlyRootfs': True, 'Privileged': False, + 'CapDrop': ['ALL'], 'CapAdd': None, + 'SecurityOpt': ['no-new-privileges:true']}, + 'NetworkSettings': {'Networks': {'owned_default': {}}}, + 'Mounts': [{'Type': 'bind', 'Source': '/owned/scratch', + 'Destination': '/owned/scratch', 'RW': True}, + {'Type': 'bind', 'Source': '/private/db', + 'Destination': '/run/secrets/database_url', 'RW': False}]} + + def check(self, container=None): + validate_container('worker', self.rendered, container or self.container, + self.image, {'database': {'file': '/private/db'}}, + {'default': {'name': 'owned_default'}}) + + def test_exact_production_service_is_accepted(self): + self.check() + + def test_recorded_worker_host_api_and_hmac_are_refused(self): + for command in ( + ['python', '-m', 'celery', '-A', 'experiments.pilot04_external_checker_runtime.guide_worker', 'worker'], + ['python', '-m', 'uvicorn', 'experiments.pilot04_external_checker_runtime.public_intake_observer:app'], + ): + changed = copy.deepcopy(self.container) + changed['Config']['Cmd'] = command + with self.assertRaisesRegex(RuntimeError, 'executable differs'): + self.check(changed) + for environment in ('WORKSTREAM_FLOW_AUTH_LOCAL_HMAC_SECRET=fixture-secret', + 'WORKSTREAM_CELERY_TASK_ALWAYS_EAGER=true', + 'WORKSTREAM_PROBE_INTAKE_TRACE=/tmp/fake'): + changed = copy.deepcopy(self.container) + changed['Config']['Env'].append(environment) + with self.assertRaisesRegex(RuntimeError, 'extra environment'): + self.check(changed) + + def test_same_labels_cannot_hide_foreign_image_mount_or_entrypoint(self): + changes = ( + ('Image', 'sha256:' + 'c' * 64), + ('Mounts', self.container['Mounts'] + [{'Type': 'bind', 'Source': '/var/run/docker.sock', + 'Destination': '/var/run/docker.sock', 'RW': True}]), + ('Config', dict(self.container['Config'], Entrypoint=['/bin/sh'])), + ('HostConfig', dict(self.container['HostConfig'], Privileged=True)), + ('State', {'Running': True, 'Health': {'Status': 'starting'}}), + ('NetworkSettings', {'Networks': {'foreign': {}}}), + ('HostConfig', dict(self.container['HostConfig'], PortBindings={ + '8000/tcp': [{'HostIp': '0.0.0.0', 'HostPort': '8000'}]})), + ) + for key, value in changes: + with self.subTest(key=key), self.assertRaises(RuntimeError): + self.check(dict(self.container, **{key: value})) + + def test_tmpfs_must_match_rendered_targets_and_options(self): + self.rendered['tmpfs'] = ['/tmp:mode=0700,uid=10001,gid=10001'] + self.container['HostConfig']['Tmpfs'] = {'/tmp': 'gid=10001,mode=0700,uid=10001'} + self.check() + for mutation in ('code-cover', 'extra-record', 'missing', 'wrong-owner', 'extra-option'): + changed = copy.deepcopy(self.container) + if mutation == 'code-cover': + changed['HostConfig']['Tmpfs']['/app'] = 'rw,uid=10001,gid=10001' + changed['Mounts'].append({'Type': 'tmpfs', 'Destination': '/app', 'RW': True}) + elif mutation == 'extra-record': + changed['Mounts'].append({'Type': 'tmpfs', 'Destination': '/app', 'RW': True}) + elif mutation == 'missing': + changed['HostConfig']['Tmpfs'] = {} + elif mutation == 'wrong-owner': + changed['HostConfig']['Tmpfs']['/tmp'] = 'mode=0700,uid=0,gid=10001' + else: + changed['HostConfig']['Tmpfs']['/tmp'] += ',exec' + with self.subTest(mutation=mutation), self.assertRaisesRegex(RuntimeError, 'tmpfs differs'): + self.check(changed) + + def test_parser_uses_its_exact_declared_isolation_and_resource_profile(self): + self.rendered.update( + user='0:0', command=['python', '-m', 'app.adapters.guide_document_tools.service'], + environment={'WORKSTREAM_RELEASE_ROLE': 'parser'}, network_mode='none', + cap_add=['KILL', 'SETGID', 'SETUID'], mem_limit='536870912', + memswap_limit='536870912', cpus=1.0, pids_limit=64, + tmpfs=['/tmp:mode=0700,uid=10002,gid=10001,size=268435456,noexec,nosuid,nodev'], + ) + del self.rendered['networks'] + self.container['Config'].update(User='0:0', Cmd=self.rendered['command'], + Env=['PATH=/usr/bin', 'WORKSTREAM_RELEASE_ROLE=parser']) + self.container['HostConfig'].update( + CapAdd=['CAP_KILL', 'CAP_SETGID', 'CAP_SETUID'], NetworkMode='none', + Memory=536870912, MemorySwap=536870912, NanoCpus=1_000_000_000, PidsLimit=64, + Tmpfs={'/tmp': self.rendered['tmpfs'][0].split(':', 1)[1]}, + ) + self.container['NetworkSettings']['Networks'] = {'none': {}} + self.check() + for key, value in (('CapAdd', ['CAP_SYS_ADMIN']), ('NetworkMode', 'host'), + ('Memory', 0), ('MemorySwap', -1), ('NanoCpus', 0), ('PidsLimit', -1)): + changed = copy.deepcopy(self.container) + changed['HostConfig'][key] = value + with self.subTest(key=key), self.assertRaises(RuntimeError): + self.check(changed) + + +class NativePreparationObservation(unittest.TestCase): + def test_only_successful_final_copy_records_completion_without_altering_stream(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + wrapper = root / 'wrapper' + shutil.copyfile(ROOT / 'experiments/pilot04_external_checker_runtime/public_runtime_wrapper.py', wrapper) + binary = root / 'smol' + binary.write_text('#!/bin/sh\nprintf "actual-native-output\\n"\nexit "$RESULT"\n') + binary.chmod(0o700) + trace = root / 'trace' + leases = root / 'smol-leases' + leases.mkdir() + lease = {'name': 'intake-one', 'owner': 'owned', 'epoch': 'original', + 'expires_ms': 100000, 'lifetime_ms': 90000} + (leases / 'intake-one.json').write_text(json.dumps(lease)) + wrapper.with_suffix('.json').write_text(json.dumps({ + 'trace': str(trace), 'binary': str(binary), 'project_archive': '/project.tar', + 'socket': str(root / 'node.sock')})) + for status, destination, completed in ((7, 'intake-one:/.workstream-guest', False), + (0, 'intake-one:/unrelated', False), + (0, 'intake-one:/.workstream-guest', True)): + trace.write_text(json.dumps({'operation': 'create', 'name': 'intake-one', + 'project': False, 'lease': lease}) + '\n') + result = subprocess.run([sys.executable, str(wrapper), 'machine', 'cp', '--mode', '0555', + '/guest', destination], env={'RESULT': str(status)}, + capture_output=True, timeout=10) + self.assertEqual(result.returncode, status) + self.assertEqual(result.stdout, b'actual-native-output\n') + events = [json.loads(line) for line in trace.read_text().splitlines()] + self.assertEqual(len(events), 2 if completed else 1) + if completed: + self.assertEqual(events[-1]['operation'], 'prepare_completed') + self.assertEqual(events[-1]['name'], 'intake-one') + self.assertEqual(events[-1]['lease'], lease) + self.assertFalse(events[-1]['project']) + + +class HostedFixtureContracts(unittest.TestCase): + def test_fresh_setup_grants_operator_then_registers_exact_supplied_baseline_capabilities(self): + import httpx + from datetime import UTC, datetime + from app.core.identifiers import new_record_id + from app.modules.checkers.api.external import ExternalCheckerRegistrySpec + from experiments.pilot04_external_checker_runtime.run_selected_end_to_end_probe import _locked_spec + + capabilities = {'pre_submit': {'capability_id': 'probe.hosted_pre', 'capability_version': 'v1'}, + 'post_submit': {'capability_id': 'probe.hosted_post', 'capability_version': 'v1'}} + attachment = HostedAttachment('http://127.0.0.1:8000', Path('/release'), Path('/issuer'), + Path('/node'), capabilities, 'sha256:fixture') + attachment.actors = {label: SimpleNamespace(actor_profile_id=new_record_id()) + for label in ('access-admin', 'operator', 'manager')} + image = {'repository': 'workstream-test/public-journey', 'platform_manifest_digest': 'sha256:' + 'a' * 64} + attachment.native_binding = {'node_config': {'cache': [image]}} + operations, registered_specs = [], [] + + async def respond(request): + body = json.loads(request.content) + if request.url.path == '/api/v1/admin-role-grants': + operations.append('grant:' + body['role']) + return httpx.Response(201, json={'granted': True}) + self.assertIn('grant:operator', operations) + self.assertEqual(request.url.path, '/api/v1/operations/external-checker-registry') + self.assertEqual(request.headers['Authorization'], 'Bearer token:operator') + spec = ExternalCheckerRegistrySpec.model_validate_json(request.content) + operations.append('registry:' + spec.phase) + registered_specs.append(spec) + receipt = body | {'registry_entry_id': str(new_record_id()), 'entry_digest': spec.spec_digest, + 'registration_operation_id': request.headers['Idempotency-Key'], + 'registered_by_actor_profile_id': str(attachment.actors['operator'].actor_profile_id), + 'authorization_decision_event_id': str(new_record_id()), 'created_at': datetime.now(UTC).isoformat()} + return httpx.Response(201, json=receipt) + + async def services(client): + operations.append('services') + + async def transition(phase): + operations.append(phase) + + async def exercise(): + async with httpx.AsyncClient(base_url=attachment.origin, transport=httpx.MockTransport(respond)) as client: + await attachment.prepare_actors(client, None) + + with patch.object(attachment, 'admit_actors', new=AsyncMock()), \ + patch.object(attachment, 'setup_command', side_effect=lambda *args: operations.append('bootstrap')), \ + patch.object(attachment, 'provision_services', side_effect=services), \ + patch.object(attachment, 'transition', side_effect=transition), \ + patch.object(attachment, 'token_for_label', side_effect=lambda label: 'token:' + label): + asyncio.run(exercise()) + self.assertEqual(operations, ['bootstrap', 'services', 'grant:operator', 'registry:pre_submit', + 'registry:post_submit', 'grant:project_manager', 'shadow']) + self.assertEqual(len(registered_specs), 2) + for spec in registered_specs: + expected = _locked_spec(spec.phase, image).model_dump(mode='json') | capabilities[spec.phase] + self.assertEqual(spec.model_dump(mode='json'), expected) + + def test_baseline_registration_refuses_unvalidated_cache_and_substituted_receipts(self): + import httpx + from datetime import UTC, datetime + from app.core.identifiers import new_record_id + from app.modules.checkers.api.external import ExternalCheckerRegistrySpec + + capabilities = {phase: {'capability_id': 'probe.' + phase, 'capability_version': 'v1'} + for phase in ('pre_submit', 'post_submit')} + attachment = HostedAttachment('http://127.0.0.1:8000', Path('/release'), Path('/issuer'), + Path('/node'), capabilities, 'sha256:fixture') + operator_id = new_record_id() + attachment.actors['operator'] = SimpleNamespace(actor_profile_id=operator_id) + image = {'repository': 'workstream-test/public-journey', 'platform_manifest_digest': 'sha256:' + 'a' * 64} + requested = [] + + async def refuse(request): + requested.append(request.url.path) + raise AssertionError('invalid cache must refuse before registration') + + async def empty_or_ambiguous(): + async with httpx.AsyncClient(base_url=attachment.origin, transport=httpx.MockTransport(refuse)) as client: + for images in ([], [image, image], [dict(image, repository='workstream/terminal-systems-post')]): + attachment.native_binding = {'node_config': {'cache': images}} + with self.assertRaisesRegex(RuntimeError, 'absent or ambiguous'): + await attachment.register_baseline_capabilities(client) + + asyncio.run(empty_or_ambiguous()) + self.assertEqual(requested, []) + attachment.native_binding = {'node_config': {'cache': [image]}} + + async def exercise(fault): + async def respond(request): + body = json.loads(request.content) + if fault == 'image': + body['image_digest'] = 'sha256:' + 'f' * 64 + spec = ExternalCheckerRegistrySpec.model_validate(body) + receipt = body | {'registry_entry_id': str(new_record_id()), 'entry_digest': spec.spec_digest, + 'registration_operation_id': request.headers['Idempotency-Key'], + 'registered_by_actor_profile_id': str(operator_id if fault != 'actor' else new_record_id()), + 'authorization_decision_event_id': str(new_record_id()), 'created_at': datetime.now(UTC).isoformat()} + if fault == 'operation': + receipt['registration_operation_id'] = str(uuid4()) + return httpx.Response(201, json=receipt) + async with httpx.AsyncClient(base_url=attachment.origin, transport=httpx.MockTransport(respond)) as client: + with self.assertRaisesRegex(RuntimeError, 'substituted specification or authority'): + await attachment.register_baseline_capabilities(client) + + with patch.object(attachment, 'token_for_label', return_value='token:operator'): + for fault in ('image', 'actor', 'operation'): + asyncio.run(exercise(fault)) + + def test_service_setup_uses_public_owner_endpoint_and_stable_exact_replay_keys(self): + import httpx + from provision_local_service_actors import JOURNEY_SERVICE_IDENTITIES + + attachment = HostedAttachment('http://127.0.0.1:8000', Path('/release'), Path('/issuer'), + Path('/node'), {}, 'sha256:fixture') + attachment.actors['access-admin'] = SimpleNamespace(actor_profile_id=uuid4()) + observed = [] + + async def respond(request): + observed.append((request.method, request.url.path, request.headers['Authorization'], + request.headers['Idempotency-Key'], json.loads(request.content))) + return httpx.Response(201, json={'created': True}) + + async def exercise(): + async with httpx.AsyncClient(base_url=attachment.origin, transport=httpx.MockTransport(respond)) as client: + await attachment.provision_services(client) + await attachment.provision_services(client) + + with patch.object(attachment, 'token_for_label', return_value='scoped-rs256-token'): + asyncio.run(exercise()) + count = len(JOURNEY_SERVICE_IDENTITIES) + self.assertEqual(observed[:count], observed[count:]) + self.assertEqual([item[4]['service_identity'] for item in observed[:count]], list(JOURNEY_SERVICE_IDENTITIES)) + self.assertTrue(all(item[:3] == ('POST', '/api/v1/service-actors', 'Bearer scoped-rs256-token') + for item in observed)) + self.assertEqual(len({item[3] for item in observed}), count) + + def test_scoped_fixture_token_uses_rs256_without_embedded_product_roles(self): + import jwt + from experiments.hosted_drill_identity.fixture import initialize, read_private, ISSUER, AUDIENCE + + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) / 'issuer' + initialize(root) + attachment = HostedAttachment('http://127.0.0.1:8000', root / 'release.env', root, + root / 'node', {}, 'sha256:unused') + token = attachment.token_for_label('operator') + key = jwt.PyJWK(json.loads(read_private(root / 'jwks.json'))['keys'][0]).key + claims = jwt.decode(token, key, algorithms=['RS256'], issuer=ISSUER, audience=AUDIENCE) + self.assertEqual(claims['scope'], 'workstream:access') + self.assertEqual(claims['subject_kind'], 'human') + self.assertEqual(claims['exp'] - claims['iat'], 900) + self.assertNotIn('roles', claims) + self.assertEqual(list(root.glob('token-*')), []) + with self.assertRaises(jwt.InvalidAlgorithmError): + jwt.decode(token, key, algorithms=['HS256']) + + def test_manager_rules_compile_with_production_dto_and_original_citation(self): + from app.adapters.project_agents.manager_authored import _compile_rules + + original = b'Explicit synthetic hosted runtime guide.' + document = SimpleNamespace(source_item_id=uuid4(), ingest_id=uuid4(), + sha256='sha256:' + hashlib.sha256(original).hexdigest()) + rules = SimpleNamespace(source_item_id=uuid4(), ingest_id=uuid4(), sha256='sha256:' + 'f' * 64) + context = SimpleNamespace(agent_version='v1', material=SimpleNamespace(documents=[document, rules])) + capabilities = {stage: {'capability_id': 'probe.' + stage, 'capability_version': 'v1'} + for stage in ('pre_submit', 'post_submit')} + for count in (0, 1, 2): + authored = json.loads(manager_rules(original, capabilities, count)) + result = _compile_rules(context, [(rules, authored)]) + self.assertEqual(result.status, 'draft_ready') + self.assertEqual(len(result.pre_submit_bindings), count) + self.assertEqual(len(result.post_submit_bindings), 1) + self.assertEqual(len(result.requirements), count + 1) + self.assertEqual(result.findings[0].evidence_refs[0].sha256, document.sha256) + + def test_hosted_cli_never_calls_local_launcher(self): + from experiments.pilot04_external_checker_runtime import run_public_journey_probe as probe + from experiments.pilot04_external_checker_runtime import public_hosted_attachment as hosted + + with patch.object(sys, 'argv', ['proof', '--hosted-attachment', '/private/attach.json', + '--database-url', 'postgresql+asyncpg://isolated', '--broker-url', 'redis://isolated']), \ + patch.object(hosted.HostedAttachment, 'read', return_value='attached'), \ + patch.object(hosted, 'run', new=AsyncMock(return_value={'attached': True})) as attached, \ + patch.object(probe, 'run') as launch, patch('builtins.print'): + self.assertEqual(probe.main(), 0) + attached.assert_awaited_once_with('attached', 'postgresql+asyncpg://isolated', 'redis://isolated') + launch.assert_not_called() + with patch.object(sys, 'argv', ['proof', '--hosted-attachment', '/private/attach.json', + '--database-url', 'postgresql+asyncpg://isolated', '--broker-url', 'redis://isolated', + '--node-config', '/local/node.json']), patch.object(probe, 'run') as launch: + with self.assertRaises(SystemExit): + probe.main() + launch.assert_not_called() + + def test_stalled_upload_refuses_early_cleanup_or_reset_deadline(self): + import time + from experiments.pilot04_external_checker_runtime import public_intake_trials as intake + + class StreamingClient: + async def post(self, route, *, headers, content): + async for _ in content: + pass + raise AssertionError('invalid expiry should fail before completing request') + + expires = time.time_ns() // 1_000_000 + 1000 + prepared = {'operation': 'prepare_completed', 'name': 'exact-intake', 'project': False, + 'lease': {'epoch': 'original', 'expires_ms': expires}} + for deletion in ( + {'wall_ms': expires - 1, 'lease': prepared['lease']}, + {'wall_ms': expires + 1, 'lease': {'epoch': 'original', 'expires_ms': expires + 1000}}, + {'wall_ms': expires + 1, 'lease': {'epoch': 'restarted', 'expires_ms': expires}}, + ): + with self.subTest(deletion=deletion), patch.object(intake, 'trace_events', return_value=[]), \ + patch.object(intake, '_native_event', new=AsyncMock(side_effect=[prepared, deletion])), \ + patch.object(intake, 'node_command', new=AsyncMock()) as cleanup: + with self.assertRaisesRegex(RuntimeError, 'original lease expiry'): + asyncio.run(intake._expire_upload(StreamingClient(), None, '/public', {}, 'token', b'zip', + 'task', Path('/owned/control'))) + cleanup.assert_not_awaited() + + +if __name__ == '__main__': + unittest.main() diff --git a/scripts/test_public_hosted_model_post.py b/scripts/test_public_hosted_model_post.py new file mode 100644 index 000000000..6e455fa6f --- /dev/null +++ b/scripts/test_public_hosted_model_post.py @@ -0,0 +1,441 @@ +"""Bounded hosted model proof: public authority, no second paid project or fake grading.""" + +import asyncio +import copy +from datetime import UTC, datetime +from io import BytesIO +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +from types import SimpleNamespace +import unittest +from unittest.mock import AsyncMock, Mock, patch +from uuid import UUID +from zipfile import ZipFile + +ROOT = Path(__file__).resolve().parents[1] +sys.path[:0] = [str(ROOT), str(ROOT / 'backend'), str(ROOT / 'backend/scripts'), str(ROOT / 'backend/tests')] +from app.core.identifiers import new_record_id # noqa: E402 +from app.modules.model_usage.api import ProjectModelProfile # noqa: E402 +from experiments.pilot04_external_checker_runtime.public_hosted_model_post import ( # noqa: E402 + CEILING, CHECK, MODEL, ModelPostQualification, canonical, digest, fixture_archive, + profile_spec, registry_spec, require_completed_effect, specification, POST_IMAGE, POST_CONFIG, POST_ARCHIVE, + account_snapshot, validate_runtime_window, observe_model_post, paid_custody, model_grant_snapshot, +) + + +class HostedModelAuthority(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name) + + def qualification(self): + value = ModelPostQualification(POST_IMAGE, POST_CONFIG, POST_ARCHIVE, self.root / 'proof.json', + 'earlier setup effect remains unresolved in its original account') + value.claim() + self.addCleanup(value.close) + return value + + def test_planned_model_window_fits_both_actual_custody_and_socket_limits(self): + def check(artifact, socket, *, driver_artifact=None): + values = {'WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS': str(artifact), + 'WORKSTREAM_EXTERNAL_CHECKER_SERVICE_TIMEOUT_SECONDS': str(socket)} + settings = SimpleNamespace( + artifact_preparation_total_deadline_seconds=artifact if driver_artifact is None else driver_artifact, + external_checker_service_timeout_seconds=socket, + ) + return validate_runtime_window(values, settings) + + self.assertEqual(check(3400, 3660), 720000) + self.assertEqual(check(3400.0, 3660.0), 720000) + self.assertEqual(check(720, 660), 720000) + for artifact in (360, 719): + with self.assertRaisesRegex(RuntimeError, 'ART custody window'): + check(artifact, 3660) + with self.assertRaisesRegex(RuntimeError, 'checker socket window'): + check(3400, 659) + with self.assertRaisesRegex(RuntimeError, 'differ from inspected hosted roles'): + check(3400, 3660, driver_artifact=360) + + def test_short_rehearsal_window_refuses_before_claim_or_public_work(self): + from experiments.pilot04_external_checker_runtime import public_hosted_attachment as hosted + + attachment = SimpleNamespace(inspect=Mock(), node_root=self.root, values={ + 'WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS': '360', + 'WORKSTREAM_EXTERNAL_CHECKER_SERVICE_TIMEOUT_SECONDS': '3660', + }) + settings = SimpleNamespace(artifact_preparation_total_deadline_seconds=360, + external_checker_service_timeout_seconds=3660) + with patch('app.core.config.get_settings', return_value=settings), \ + patch.object(ModelPostQualification, 'read_and_claim') as claim, \ + patch.object(hosted, '_run', new_callable=AsyncMock) as public_work: + with self.assertRaisesRegex(RuntimeError, 'ART custody window'): + asyncio.run(hosted.run(attachment, 'unused', 'unused', self.root / 'input.json')) + attachment.inspect.assert_called_once_with() + claim.assert_not_called() + public_work.assert_not_called() + self.assertFalse((self.root / 'model-post-qualification.json').exists()) + + def test_hosted_socket_timeout_uses_the_canonical_default_when_not_overridden(self): + from app.core.config import Settings + + values = {'WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS': '3400'} + settings = Settings.model_construct(artifact_preparation_total_deadline_seconds=3400) + self.assertEqual(validate_runtime_window(values, settings), 720000) + settings.external_checker_service_timeout_seconds -= 1 + with self.assertRaisesRegex(RuntimeError, 'differ from inspected hosted roles'): + validate_runtime_window(values, settings) + + def test_prepared_effect_recovery_retains_one_exact_predecessor_grant(self): + def exercise(change=None, *, history=True): + project, submission, prior_run_id, run_id, registry_id, profile_id = (new_record_id() for _ in range(6)) + now = datetime.now(UTC) + effect_id, request_hash = str(new_record_id()), 'sha256:' + 'a' * 64 + def grant(identifier): + return SimpleNamespace(id=str(new_record_id()), checker_run_id=str(identifier), + registry_entry_id=registry_id, token_hash='sha256:' + 'b' * 64, + external_request_digest='sha256:' + 'c' * 64, + execution_facts={'run': str(identifier)}, paid_slot={'submission': str(submission)}, + authorization_evidence_id=str(new_record_id()), issued_at=now, expires_at=now) + previous, current = grant(prior_run_id), grant(run_id) + def call(): + return SimpleNamespace(call_index=0, attempt_id=effect_id, + request_sha256=request_hash, bound_at=now) + prior_call, current_call = call(), call() + snapshot = model_grant_snapshot(previous, [prior_call]) + slot = SimpleNamespace(id=str(new_record_id()), submission_id=str(submission), requirement_id=CHECK, + profile_id=str(profile_id), profile_digest='profile', registry_entry_digest='registry', + first_request_sha256=request_hash) + prior_run = SimpleNamespace(id=str(prior_run_id), status='infrastructure_failed', + project_id=str(project), submission_id=str(submission)) + run = SimpleNamespace(project_id=str(project), submission_id=str(submission), + supersedes_checker_run_id=str(prior_run_id)) + grants = [previous, current] + if change == 'history_mutated': + previous.execution_facts = {'changed': True} + elif change == 'new_effect': + current_call.attempt_id = str(new_record_id()) + elif change == 'unrelated_run': + run.supersedes_checker_run_id = str(new_record_id()) + elif change == 'extra_grant': + grants.append(grant(new_record_id())) + elif change == 'initial': + grants = [current] + session = AsyncMock() + session.__aenter__.return_value = session + session.scalars.side_effect = [Mock(all=Mock(return_value=rows)) for rows in + (grants, [slot], [current_call], [prior_call])] + session.get.side_effect = [prior_run, run] + qualification = SimpleNamespace(state={'prior_model_grant': snapshot} if history else {}, + registry={'registry_entry_id': str(registry_id), 'entry_digest': 'registry'}, + profile=SimpleNamespace(profile_id=profile_id, profile_digest='profile')) + with patch('app.modules.checkers.execution_repository.stored_request', + return_value=SimpleNamespace(policy='same immutable policy')): + return asyncio.run(paid_custody(Mock(return_value=session), str(project), run_id, + str(submission), SimpleNamespace(request_digest=current.external_request_digest), qualification)) + + receipt = exercise() + self.assertEqual(receipt['effect_id'], receipt['prior_model_grant']['effect_id']) + self.assertNotIn('prior_model_grant', exercise('initial', history=False)) + for change in ('history_mutated', 'new_effect', 'unrelated_run', 'extra_grant'): + with self.subTest(change=change), self.assertRaises(RuntimeError): + exercise(change) + with self.assertRaisesRegex(RuntimeError, 'unexpected grant/paid-slot count'): + exercise(history=False) + + def test_observation_refuses_a_run_from_another_project_task_or_submission(self): + project, task, submission = (str(new_record_id()) for _ in range(3)) + for field in ('project_id', 'task_id', 'submission_id'): + run = SimpleNamespace(project_id=project, task_id=task, submission_id=submission) + setattr(run, field, str(new_record_id())) + session = AsyncMock() + session.__aenter__.return_value = session + session.get.return_value = run + with self.subTest(field=field), \ + patch('app.modules.checkers.execution_repository.stored_request') as request: + with self.assertRaisesRegex(RuntimeError, 'differs from the retained project and Submission'): + asyncio.run(observe_model_post( + None, None, Mock(return_value=session), None, + project=SimpleNamespace(project_id=project), contributor=None, + task=task, assignment=None, archive=b'', summary=None, attestation=None, + saved={'submission_id': submission}, event_id=new_record_id(), run_id=new_record_id(), + )) + request.assert_not_called() + + def test_observation_checks_real_policy_budget_with_float_settings(self): + from app.modules.checkers.api.external_policy import LockedExternalPostSubmitPolicy + + policy = LockedExternalPostSubmitPolicy(entries=(), policy_sha256=digest(canonical({ + 'schema_version': 'external_post_submit_policy.v1', 'entries': [], + }))) + project, task, submission = (str(new_record_id()) for _ in range(3)) + session = AsyncMock() + session.__aenter__.return_value = session + session.get.return_value = SimpleNamespace(project_id=project, task_id=task, submission_id=submission) + attachment = SimpleNamespace(values={ + 'WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS': '3400.0', + 'WORKSTREAM_EXTERNAL_CHECKER_SERVICE_TIMEOUT_SECONDS': '3660.0', + }) + settings = SimpleNamespace(artifact_preparation_total_deadline_seconds=3400.0, + external_checker_service_timeout_seconds=3660.0) + with patch('app.core.config.get_settings', return_value=settings), \ + patch('app.modules.checkers.execution_repository.stored_request', + return_value=SimpleNamespace(policy=policy)): + with self.assertRaisesRegex(RuntimeError, 'locked model qualification execution budget differs'): + asyncio.run(observe_model_post( + attachment, None, Mock(return_value=session), None, + project=SimpleNamespace(project_id=project), contributor=None, + task=task, assignment=None, archive=b'', summary=None, attestation=None, + saved={'submission_id': submission}, event_id=new_record_id(), run_id=new_record_id(), + )) + + def test_cache_requires_exact_qualified_production_image_not_a_substitute(self): + qualification = self.qualification() + binding = {'node_config': {'cache': [{'repository': 'workstream/terminal-systems-post', + 'platform_manifest_digest': POST_IMAGE.rsplit('@', 1)[1], 'config_image_id': POST_CONFIG, + 'smol': {'archive_sha256': POST_ARCHIVE}}]}} + (self.root / 'node-binding.json').write_bytes(canonical(binding)) + attachment = SimpleNamespace(node_root=self.root, values={'WORKSTREAM_MODEL_PROXY_ENABLED': 'true'}, + custody={'model-proxy': {'image_id': 'verified-by-attachment'}}) + qualification.verify_cache(attachment) + qualification.image_config_digest = 'sha256:' + '0' * 64 + with self.assertRaisesRegex(RuntimeError, 'image identity differs'): + qualification.verify_cache(attachment) + qualification.image_config_digest = POST_CONFIG + binding['node_config']['cache'][0]['smol']['archive_sha256'] = 'sha256:' + '0' * 64 + (self.root / 'node-binding.json').write_bytes(canonical(binding)) + with self.assertRaisesRegex(RuntimeError, 'absent or substituted'): + qualification.verify_cache(attachment) + + def test_one_literal_profile_reserves_full_context_within_the_total_authorization(self): + profile = profile_spec(new_record_id(), new_record_id()) + self.assertEqual((profile.provider, profile.api, profile.model_snapshot), ('openrouter', 'chat_completions', MODEL)) + self.assertEqual(profile.maximum_input_tokens, 1050000) + self.assertEqual(profile.maximum_calls, 1) + self.assertEqual(profile.allowed_tools, ()) + self.assertEqual(profile.maximum_reservation_nanodollars, 5_323_728_000) + self.assertLessEqual(profile.maximum_reservation_nanodollars, CEILING) + self.assertEqual(profile.maximum_dispatch_reservation_nanodollars, profile.maximum_reservation_nanodollars) + with self.assertRaises(ValueError): + type(profile).model_validate(profile.model_dump(mode='python') | {'maximum_response_bytes': 200703}) + + def test_existing_claim_or_alias_cannot_create_another_paid_project(self): + value = self.qualification() + value.checkpoint(project_id=str(new_record_id()), phase='authority_published') + retained = value.proof_state.read_bytes() + mode = value.proof_state.stat().st_mode + with self.assertRaises(FileExistsError): + self.qualification() + self.assertEqual(value.proof_state.read_bytes(), retained) + self.assertEqual(value.proof_state.stat().st_mode, mode) + alias = self.root / 'alias' + alias.symlink_to(value.proof_state) + other = ModelPostQualification(value.image_reference, value.image_config_digest, value.image_archive_sha256, + alias, 'unresolved') + with self.assertRaisesRegex(RuntimeError, 'canonical'): + other.claim() + self.assertEqual(value.proof_state.read_bytes(), retained) + + def test_provider_override_is_rejected_before_claim_or_mutation(self): + config = self.root / 'input.json' + config.write_bytes(canonical({'image_reference': 'workstream/terminal-systems-post@sha256:' + 'a' * 64, + 'image_config_digest': 'sha256:' + 'b' * 64, 'image_archive_sha256': 'sha256:' + 'c' * 64, + 'proof_state': str(self.root / 'claim'), 'historical_uncertain_reference': 'unresolved', + 'provider': 'openai'})) + config.chmod(0o600) + with self.assertRaisesRegex(RuntimeError, 'fields differ'): + ModelPostQualification.read_and_claim(config, self.root) + self.assertFalse((self.root / 'claim').exists()) + + def test_claim_creation_flushes_its_parent_before_any_public_mutation(self): + import stat + + flushed = [] + original = os.fsync + + def fsync(descriptor): + flushed.append('directory' if stat.S_ISDIR(os.fstat(descriptor).st_mode) else 'file') + original(descriptor) + + with patch('os.fsync', side_effect=fsync): + self.qualification() + self.assertEqual(flushed, ['file', 'directory']) + + def test_changing_the_state_path_cannot_reset_the_deployment_authorization(self): + config = self.root / 'input.json' + config.write_bytes(canonical({'image_reference': POST_IMAGE, 'image_config_digest': POST_CONFIG, + 'image_archive_sha256': POST_ARCHIVE, 'proof_state': str(self.root / 'another-claim'), + 'historical_uncertain_reference': 'unresolved'})) + config.chmod(0o600) + with self.assertRaisesRegex(RuntimeError, 'single fixed state path'): + ModelPostQualification.read_and_claim(config, self.root) + self.assertFalse((self.root / 'another-claim').exists()) + + def test_public_authority_publication_and_real_rules_bind_the_exact_profile(self): + import httpx + from app.adapters.project_agents.manager_authored import _compile_rules + from app.core.hashing import canonical_json_hash + + qualification = self.qualification() + project_id = str(new_record_id()) + requests = [] + actor_id, profile_id = new_record_id(), new_record_id() + + async def handler(request): + body = json.loads(request.content) + requests.append((request.method, request.url.path, request.headers['Idempotency-Key'], body)) + if request.url.path.endswith('/model-profiles'): + spec = type(profile_spec(UUID(project_id), new_record_id())).model_validate_json(request.content) + response = ProjectModelProfile(operation_id=UUID(request.headers['Idempotency-Key']), + actor_profile_id=actor_id, spec=spec, profile_digest=canonical_json_hash(body), + profile_id=profile_id, authorization_evidence_id=new_record_id(), created_at=datetime.now(UTC)).model_dump(mode='json') + elif request.url.path.endswith('/model-budget-configurations'): + response = {'spec': body, 'configuration_id': str(new_record_id()), + 'lifetime_reserved_nanodollars': 0, 'lifetime_settled_nanodollars': 0} + else: + spec = registry_spec(body['image_digest']) + self.assertEqual(body, spec.model_dump(mode='json')) + response = {**body, 'entry_digest': spec.spec_digest, 'registry_entry_id': str(new_record_id())} + return httpx.Response(201, json=response) + + attachment = SimpleNamespace(token_for_label=lambda label: 'scoped-rs256-fixture') + + async def exercise(): + async with httpx.AsyncClient(base_url='http://127.0.0.1:8000', transport=httpx.MockTransport(handler)) as client: + await qualification.bind_project(client, project_id, attachment) + with self.assertRaisesRegex(RuntimeError, 'already owns one project'): + await qualification.bind_project(client, str(new_record_id()), attachment) + + asyncio.run(exercise()) + self.assertEqual([item[1] for item in requests], [ + f'/api/v1/operations/projects/{project_id}/model-profiles', + f'/api/v1/operations/projects/{project_id}/model-budget-configurations', + '/api/v1/operations/external-checker-registry']) + self.assertTrue(all(item[0] == 'POST' for item in requests)) + budget = requests[1][3] + self.assertEqual(budget['authorized_total_nanodollars'], CEILING) + self.assertEqual(budget['lifetime_limit_nanodollars'], CEILING) + self.assertEqual(budget['configuration_version'], 1) + self.assertEqual(qualification.state['historical_charge']['status'], 'unresolved_separate') + original = qualification.guide_bytes() + capabilities = {stage: {'capability_id': 'fixture.' + stage, 'capability_version': 'v1'} + for stage in ('pre_submit', 'post_submit')} + rules = json.loads(qualification.rules(original, capabilities, 0)) + binding = rules['post_submit_bindings'][0] + configuration = {item['name']: item['value'] for item in binding['parameters']} + registry_spec('sha256:' + 'a' * 64).configuration_schema.validate_instance(configuration) + self.assertEqual(configuration['check'], CHECK) + self.assertEqual(configuration['model_profile_id'], str(profile_id)) + self.assertEqual(rules['pre_submit_bindings'], []) + doc = SimpleNamespace(source_item_id=new_record_id(), ingest_id=new_record_id(), sha256=digest(original)) + authored = SimpleNamespace(source_item_id=new_record_id(), ingest_id=new_record_id(), sha256=digest(canonical(rules))) + result = _compile_rules(SimpleNamespace(agent_version='v1', material=SimpleNamespace(documents=[doc, authored])), [(authored, rules)]) + self.assertEqual(result.requirements[0].check_specification, specification()[0]) + self.assertEqual(result.requirements[0].check_specification.runtime_dependencies[0].provisioning, 'model_proxy') + with self.assertRaisesRegex(RuntimeError, 'PRE model'): + qualification.rules(original, capabilities, 1) + + def test_ambiguous_effect_or_outstanding_reservation_cannot_claim_model_proof(self): + good = {'reserved_nanodollars': 0, 'effects': [{'state': 'completed', 'provider_response_known': True, + 'request_binding_id': 'request', 'response_binding_id': 'response'}]} + require_completed_effect(good) + for state in ('prepared', 'dispatching', 'uncertain', 'known_failed', 'known_no_effect'): + changed = copy.deepcopy(good) + changed['effects'][0]['state'] = state + with self.assertRaisesRegex(RuntimeError, 'ambiguous paid effect'): + require_completed_effect(changed) + for changed in ({**good, 'reserved_nanodollars': 1}, {**good, 'effects': []}, + {**good, 'effects': good['effects'] * 2}): + with self.assertRaises(RuntimeError): + require_completed_effect(changed) + + def test_account_snapshot_joins_real_checker_and_setup_slot_owners_without_mutation(self): + from sqlalchemy.dialects import postgresql + + statements = [] + + class ReadOnlySession: + async def __aenter__(self): + return self + + async def __aexit__(self, *args): + return None + + async def get(self, model, identifier): + return SimpleNamespace(lifetime_reserved_nanodollars=0, lifetime_settled_nanodollars=123, + current_configuration_id='retained-budget') + + async def scalars(self, statement): + statements.append(str(statement.compile(dialect=postgresql.dialect()))) + return SimpleNamespace(all=lambda: []) + + result = asyncio.run(account_snapshot(ReadOnlySession, str(new_record_id()))) + self.assertEqual(result['settled_nanodollars'], 123) + self.assertEqual(result['reserved_nanodollars'], 0) + self.assertFalse(result['provider_invoice_qualified']) + self.assertIn('LEFT OUTER JOIN checker_model_usage_slots', statements[0]) + self.assertIn('LEFT OUTER JOIN setup_model_usage_slots', statements[0]) + self.assertNotIn('model_usage_effects.project_id', statements[0]) + + def test_fixture_is_deterministic_and_its_disclosed_commands_are_coherent(self): + first = fixture_archive() + self.assertEqual(first, fixture_archive()) + with ZipFile(BytesIO(first)) as archive: + self.assertTrue(all(not name.startswith('/') and '..' not in Path(name).parts for name in archive.namelist())) + archive.extractall(self.root) + before = subprocess.run(['sh', 'tests/run.sh'], cwd=self.root, capture_output=True, timeout=5) + self.assertNotEqual(before.returncode, 0) + subprocess.run(['sh', 'solution/apply.sh'], cwd=self.root, capture_output=True, check=True, timeout=5) + subprocess.run(['sh', 'tests/run.sh'], cwd=self.root, capture_output=True, check=True, timeout=5) + self.assertIn(b'no execution or trial measurement is claimed', (self.root / 'evidence/results').read_bytes()) + + def test_hosted_stalled_admission_does_not_inject_a_verifier_task(self): + from app.core.config import get_settings + + with patch.dict(os.environ, {'WORKSTREAM_CELERY_BROKER_URL': 'redis://127.0.0.1:6379/15'}): + # Earlier proof imports may have cached settings without a broker. + get_settings.cache_clear() + try: + from experiments.pilot04_external_checker_runtime import run_public_journey_probe as probe + from app.workers.artifacts import verify_object + + result = {'pre_submission_results': [{}], 'submission_bundle_preparation_status': 'object_confirmed', 'admission_id': None} + ticks = iter(range(0, 1000, 31)) + with patch.object(probe, '_fixture_hashes', return_value={}), \ + patch.object(probe, '_scan_audit_count', new=AsyncMock(return_value=0)), \ + patch.object(probe, '_json', new=AsyncMock(return_value=result)), \ + patch.object(probe, '_artifact_task_state', new=AsyncMock(return_value=([], [('job', 'pending', None)]))), \ + patch.object(probe.time, 'monotonic', side_effect=lambda: next(ticks)), \ + patch.object(probe.asyncio, 'sleep', new=AsyncMock()), patch.object(verify_object, 'apply_async') as dispatch: + with self.assertRaisesRegex(RuntimeError, 'remained object_confirmed'): + asyncio.run(probe._ready_admission(None, None, 'task', 'assignment', 'token', b'zip', 'summary', 'attestation', hosted=True)) + dispatch.assert_not_called() + finally: + get_settings.cache_clear() + + def test_model_stage_requires_and_uses_the_existing_hosted_attachment(self): + from experiments.pilot04_external_checker_runtime import run_public_journey_probe as probe + from experiments.pilot04_external_checker_runtime import public_hosted_attachment as hosted + + args = ['proof', '--database-url', 'postgresql+asyncpg://isolated', '--broker-url', 'redis://isolated', + '--hosted-model-post', '/private/model.json'] + with patch.object(sys, 'argv', args), patch.object(probe, 'run') as local: + with self.assertRaises(SystemExit): + probe.main() + local.assert_not_called() + with patch.object(sys, 'argv', args + ['--hosted-attachment', '/private/attachment.json']), \ + patch.object(hosted.HostedAttachment, 'read', return_value='inspected-later'), \ + patch.object(hosted, 'run', new=AsyncMock(return_value={'scope': 'hosted'})) as attach, \ + patch.object(probe, 'run') as local, patch('builtins.print'): + self.assertEqual(probe.main(), 0) + attach.assert_awaited_once_with('inspected-later', 'postgresql+asyncpg://isolated', 'redis://isolated', + Path('/private/model.json')) + local.assert_not_called() + + +if __name__ == '__main__': + unittest.main() diff --git a/scripts/test_public_smol_proof.py b/scripts/test_public_smol_proof.py index 50b92e9ed..59e0466f8 100644 --- a/scripts/test_public_smol_proof.py +++ b/scripts/test_public_smol_proof.py @@ -2,7 +2,10 @@ import hashlib import json +from contextlib import nullcontext +import os from pathlib import Path +import stat import subprocess import sys import tempfile @@ -13,6 +16,7 @@ sys.path.insert(0, str(ROOT)) from experiments.pilot04_external_checker_runtime.public_fault_files import ( # noqa: E402 PrivateControlDirectory, + private_path, ) from experiments.pilot04_external_checker_runtime import public_node_faults as faults # noqa: E402 from experiments.pilot04_external_checker_runtime.public_node_faults import validate_node # noqa: E402 @@ -53,6 +57,154 @@ def test_preexisting_broad_or_aliased_control_directory_is_refused(self): with self.assertRaises(ValueError): PrivateControlDirectory(alias) + def test_root_sticky_ancestor_requires_private_anchor_and_rejects_foreign_parent(self): + with tempfile.TemporaryDirectory(prefix='ws-sticky-proof-', dir='/tmp') as name: + directory = Path(name) + private_path(directory, 0o700, directory=True) + directory.chmod(0o755) + with self.assertRaises(ValueError): + private_path(directory, 0o700, directory=True) + parent = self.root / 'parent' + parent.mkdir(mode=0o700) + directory = parent / 'control' + directory.mkdir(mode=0o700) + parent.chmod(0o777) + with self.assertRaises(ValueError): + PrivateControlDirectory(directory) + parent.chmod(0o755) + original = Path.lstat + + def foreign(path): + info = original(path) + if path == parent: + fields = list(info) + fields[4] = os.getuid() + 20000 + return os.stat_result(fields) + return info + + with patch.object(Path, 'lstat', foreign): + with self.assertRaises(ValueError): + PrivateControlDirectory(directory) + parent.chmod(0o1777) + with patch.object(Path, 'lstat', foreign): + with self.assertRaises(ValueError): + PrivateControlDirectory(directory) + + def test_control_hardlinks_and_foreign_bindings_are_refused(self): + control = PrivateControlDirectory(self.root) + self.addCleanup(control.close) + control.write('original', {'safe': True}) + os.link(self.root / 'original', self.root / 'linked') + with self.assertRaises(ValueError): + control.read('linked') + with self.assertRaises(ValueError): + control.write('original', {}) + self.assertEqual(json.loads((self.root / 'original').read_text()), {'safe': True}) + control.write('runtime-controls.json', {'root': '/unrelated', 'uid': 10002, 'gid': 10002}) + with self.assertRaises(ValueError): + faults.runtime_control_root(self.root) + + def test_control_fifo_is_refused_without_waiting_for_a_writer(self): + os.mkfifo(self.root / 'pipe', 0o600) + program = ( + 'import sys; from experiments.pilot04_external_checker_runtime.public_fault_files ' + 'import PrivateControlDirectory; c=PrivateControlDirectory(sys.argv[1]); ' + '\ntry: c.read("pipe")\nexcept ValueError: pass\nelse: raise RuntimeError("FIFO accepted")' + '\nfinally: c.close()\n' + ) + result = subprocess.run([sys.executable, '-c', program, str(self.root)], cwd=ROOT, + capture_output=True, text=True, timeout=5) + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + + def test_hosted_config_writes_preserve_readonly_custody_and_reject_replacement(self): + directory = self.root / 'runtime-config' + directory.mkdir(mode=0o700) + config = directory / 'service.json' + config.write_bytes(b'original') + write_descriptor = os.open(config, os.O_WRONLY) + self.addCleanup(os.close, write_descriptor) + config.chmod(0o400) + actual_root = os.getuid() == 0 + uid, gid = (10001, 10001) if actual_root else (os.getuid(), os.getgid()) + if actual_root: + os.chown(directory, uid, gid) + os.chown(config, uid, gid) + # EC2 runs this case as the root operator with real UID10001 files. + # The local unprivileged run retains the real inode/mode refusal probes. + identity_context = nullcontext() if actual_root else patch.object( + faults, '_config_identity', return_value=(uid, gid, 0o400)) + original_open = os.open + + def operator_open(path, flags, *args, **kwargs): + if path == config and flags & os.O_ACCMODE == os.O_WRONLY: + # A retained descriptor supplies root's write authority locally; + # production inode/mode/owner checks still inspect the real file. + return os.dup(write_descriptor) + return original_open(path, flags, *args, **kwargs) + + open_context = nullcontext() if actual_root else patch.object(os, 'open', operator_open) + with identity_context, open_context: + original, identity = faults._read_config(config, True) + faults.write_node_config(config, b'changed', identity, hosted_runtime=True) + info = config.stat() + self.assertEqual((info.st_uid, info.st_gid, stat.S_IMODE(info.st_mode)), (uid, gid, 0o400)) + self.assertEqual((info.st_dev, info.st_ino), identity) + faults.write_node_config(config, original, identity, hosted_runtime=True) + self.assertEqual(config.read_bytes(), b'original') + config.chmod(0o600) + with self.assertRaises(ValueError): + faults.write_node_config(config, b'bad', identity, hosted_runtime=True) + config.chmod(0o400) + config.rename(directory / 'retained-original') + config.write_bytes(b'replacement must survive') + config.chmod(0o400) + if actual_root: + os.chown(config, uid, gid) + with self.assertRaisesRegex(ValueError, 'custody changed'): + faults.write_node_config(config, b'bad', identity, hosted_runtime=True) + self.assertEqual(config.read_bytes(), b'replacement must survive') + self.assertEqual((directory / 'retained-original').read_bytes(), b'original') + + def test_hosted_runtime_identity_is_fixed_and_drops_supplementary_groups(self): + with patch.object(os, 'getuid', return_value=1000): + with self.assertRaises(ValueError): + faults._config_identity(True) + with patch.object(os, 'getuid', return_value=0): + self.assertEqual(faults._config_identity(True), (10001, 10001, 0o400)) + with self.assertRaises(ValueError): + faults._runtime_user((10002, 10002)) + with patch.object(subprocess, 'check_output', return_value='{}') as command: + faults.runtime({'smol': {'binary': '/pinned/smol', 'home': '/private/home'}}, + 'machine', 'ls', '--json', identity=(10001, 10001)) + self.assertEqual({key: command.call_args.kwargs[key] + for key in ('user', 'group', 'extra_groups')}, + {'user': 10001, 'group': 10001, 'extra_groups': []}) + + def test_runtime_sibling_binding_has_distinct_operator_and_runtime_custody(self): + runtime = self.root.with_name(self.root.name + '-runtime') + runtime.mkdir(mode=0o700) + self.addCleanup(runtime.rmdir) + control = PrivateControlDirectory(self.root) + self.addCleanup(control.close) + control.write('runtime-controls.json', {'root': str(runtime), 'uid': 10001, 'gid': 10001}) + original_getuid, original_getgid = os.getuid, os.getgid + # Validate the expected-owner call independently of local chown authority. + original_path = faults.private_path + + def validate(path, mode, **kwargs): + if path == runtime: + self.assertEqual(kwargs, {'directory': True, 'uid': 10001, 'gid': 10001}) + return + return original_path(path, mode, **kwargs) + + with patch.object(faults, '_operator_controls', return_value=control), \ + patch.object(control, 'close'), patch.object(os, 'getuid', return_value=0), \ + patch.object(faults, 'private_path', validate): + # Control.read still checks the actual local operator identity. + self.assertEqual(faults.runtime_control_root(self.root), runtime) + self.assertEqual(os.getuid(), original_getuid()) + self.assertEqual(os.getgid(), original_getgid()) + def test_launch_trace_rejects_extra_execution_and_reset_deadline(self): trace = self.root / 'runtime-trace.jsonl' lease = {'epoch': 'original', 'expires_ms': 1000} diff --git a/scripts/verify_hosted_release_images.py b/scripts/verify_hosted_release_images.py index 09eea1b4a..c16efe137 100644 --- a/scripts/verify_hosted_release_images.py +++ b/scripts/verify_hosted_release_images.py @@ -452,24 +452,30 @@ def verify_backend_entrypoint(image: str, sdk_environment: str, owner: str) -> N def write_evidence(path: Path, evidence: dict) -> None: + scope = evidence.get("proof_scope") + require(scope in {"all-release-images", "native-hosted"}, "evidence scope differs") + keys = { + "proof_scope", + "source_sha", + "tree_sha", + "backend_agents_image_id", + "backend_omnicore_image_id", + "backend_source_sha256", + "backend_lock_sha256", + "loader_sha256", + "roles_anywhere_helper_sha256", + } + if scope == "all-release-images": + keys.update( + {"checker_image_id", "checker_lock_sha256", "checker_binary_sha256"} + ) require( - set(evidence) - == { - "source_sha", - "tree_sha", - "backend_agents_image_id", - "backend_omnicore_image_id", - "checker_image_id", - "backend_source_sha256", - "backend_lock_sha256", - "checker_lock_sha256", - "loader_sha256", - "checker_binary_sha256", - "roles_anywhere_helper_sha256", - }, + set(evidence) == keys, "evidence shape differs", ) for key, value in evidence.items(): + if key == "proof_scope": + continue pattern = ( r"sha256:[a-f0-9]{64}" if key.endswith("_image_id") @@ -499,6 +505,11 @@ def main() -> None: parser.add_argument("--backend-agents-image") parser.add_argument("--backend-omnicore-image") parser.add_argument("--checker-image") + parser.add_argument( + "--scope", + choices=("all-release-images", "native-hosted"), + default="all-release-images", + ) parser.add_argument("--evidence", type=Path) args = parser.parse_args() if args.cleanup_owner: @@ -511,6 +522,14 @@ def main() -> None: ) ) return + require( + bool(args.backend_agents_image) and bool(args.backend_omnicore_image), + "both backend SDK images are required", + ) + require( + bool(args.checker_image) == (args.scope == "all-release-images"), + "checker image is required only for all-release-images scope", + ) owner_label(args.container_owner or "") root = Path(__file__).resolve().parents[1] head = checked(["git", "rev-parse", "HEAD"]).decode().strip() @@ -522,7 +541,6 @@ def main() -> None: loader = file_hash(root / "docker/backend/release-entrypoint.sh") backend_agents_id = image_metadata(args.backend_agents_image, "backend") backend_omnicore_id = image_metadata(args.backend_omnicore_image, "backend") - checker_id = image_metadata(args.checker_image, "checker") backend_agents = inspect_image(backend_agents_id, "backend", args.container_owner) verify_backend(backend_agents, inventory, agents_packages, loader, "agents") verify_backend_entrypoint(backend_agents_id, "agents", args.container_owner) @@ -531,8 +549,28 @@ def main() -> None: ) verify_backend(backend_omnicore, inventory, omnicore_packages, loader, "omnicore") verify_backend_entrypoint(backend_omnicore_id, "omnicore", args.container_owner) - checker = inspect_image(checker_id, "checker", args.container_owner) - verify_checker(checker) + evidence = { + "proof_scope": args.scope, + "source_sha": head, + "tree_sha": checked(["git", "rev-parse", "HEAD^{tree}"]).decode().strip(), + "backend_agents_image_id": backend_agents_id, + "backend_omnicore_image_id": backend_omnicore_id, + "backend_source_sha256": hashlib.sha256( + json.dumps(inventory, sort_keys=True).encode() + ).hexdigest(), + "backend_lock_sha256": inventory["uv.lock"], + "loader_sha256": loader, + "roles_anywhere_helper_sha256": backend_agents["roles_anywhere_helper"], + } + if args.scope == "all-release-images": + checker_id = image_metadata(args.checker_image, "checker") + checker = inspect_image(checker_id, "checker", args.container_owner) + verify_checker(checker) + evidence.update( + checker_image_id=checker_id, + checker_lock_sha256=file_hash(root / "external_checkers/Cargo.lock"), + checker_binary_sha256=checker["binary"], + ) require( checked(["git", "rev-parse", "HEAD"]).decode().strip() == head, "proof checkout changed during inspection", @@ -540,25 +578,10 @@ def main() -> None: require( not checked(["git", "status", "--porcelain"]), "proof checkout became dirty" ) - write_evidence( - args.evidence, - { - "source_sha": head, - "tree_sha": checked(["git", "rev-parse", "HEAD^{tree}"]).decode().strip(), - "backend_agents_image_id": backend_agents_id, - "backend_omnicore_image_id": backend_omnicore_id, - "checker_image_id": checker_id, - "backend_source_sha256": hashlib.sha256( - json.dumps(inventory, sort_keys=True).encode() - ).hexdigest(), - "backend_lock_sha256": inventory["uv.lock"], - "checker_lock_sha256": file_hash(root / "external_checkers/Cargo.lock"), - "loader_sha256": loader, - "checker_binary_sha256": checker["binary"], - "roles_anywhere_helper_sha256": backend_agents["roles_anywhere_helper"], - }, + write_evidence(args.evidence, evidence) + print( + f"{args.scope} image source, packages, metadata and bounded smoke checks passed." ) - print("Release image source, packages, metadata and bounded smoke checks passed.") if __name__ == "__main__":