[mtalvi]

Related ticket: APPENG-6602

Summary

Provider API keys previously reached workspace VM disks and gateway database files, alongside reusable admin mTLS material. This PR keeps live provider keys in namespace Secrets consumed by an in-cluster credential proxy. VMs and sandboxes receive metadata and nonsecret credential markers instead of the keys.

Gateway CA signing happens in tmpfs, and the CA private key is removed before the gateway starts. Each installer apply creates a temporary admin identity, trusts its CA for that run, and removes both the trust anchor and runtime files afterward.

What changed

• Credential proxy: Adds saw-credential-proxy, a restricted, non-root Go service. It accepts allowlisted HTTPS destinations on port 443, checks CONNECT/SNI/Host consistency, verifies upstream TLS, and replaces bound markers with provider keys. Invalid markers or bindings fail closed; projected key rotations are read on subsequent requests.
• Metadata-only VM inputs: Adds -meta companion Secrets and wires gateway/sandbox traffic through the proxy. GitOps and quickstart bindings follow the active providers, including the NVIDIA build alias, optional web search, and additional OpenAI/Anthropic providers.
• Gateway and installer identity: Generates CA signing material in tmpfs, uses a temporary admin CA/client identity and isolated XDG_CONFIG_HOME during apply, waits for client-CA hot reload, and uses a nonadmin saw-relay identity for background connections. Preserves the separate JWT signing bundle required by OpenShell 0.1.2-rhaiv.6 and supplies a user-owned public proxy CA for rootless Podman/SELinux mounts.
• GitOps deployment: Makes the proxy image digest an explicit deployment input, permits the CA hook's API access when cleanup is disabled, bounds hook execution, and fixes router namespace peers for host-networked ingress routers while preserving tenant isolation.
• Builds, validation, and documentation: Adds native amd64/arm64 proxy CI builds, includes the PKI files in the signed installer manifest, and adds credential-isolation checks whose guest scan skips FIFOs and other nonregular files. Updates deployment, migration, provider, and security documentation.

Rebased onto #135

Rebased onto main 0d4935b and adapted credential isolation to typed provider configuration:

• Metadata carries every selected non-credential field, including cuda_model; provider is optional. Named credentials each receive a distinct marker and require their own permitted proxy binding.
• A shared Secret without provider uses the union of its binding's allowed provider hosts. Setting provider narrows that scope.
• Deployment-owned keyed providers bind through the credential Secret's url. Keyed custom endpoints require HTTPS on port 443 with publicly trusted TLS; other ports and service-CA trust are tracked in #137. HTTP is suitable only for keyless endpoints.
• Keyless HTTP endpoints do not require an intercept binding or nonempty API key for proxy readiness; HTTPS continues to require its bound key.
• Environment and proxy-wiring changes use checksummed archive/restore migration to preserve /sandbox. Gateway CA changes remain a separate migration that recreates the sandbox without preserving files. Restore handles image-owned read-only files while retaining the archive on any verification failure.
• OAuth2 refresh remains unsupported until it runs in the namespace proxy.

Validation

At head 11a5ff2 (rebased onto 0d4935b):

• CI: Final-head checks are awaiting confirmation.
• Local: The full local suite covered 1,513 passed, 3 skipped; the final restore regression run passed 59 tests, including the added immutable-image conflict case. Lint (82 interface checks) and catalog checks passed after the live migration fixes. Go vet and race tests passed, including the final keyless HTTP readiness regression.
• GitOps: README Option A deployment on the dedicated cluster-8rmt4, using the separate pr123-cluster-test branch and a digest-pinned native proxy build.
• Typed providers: Normal-owner chat succeeded with a providerless shared inference Secret. Notebook and CUDA used their distinct model and cuda_model values. A deployment-owned custom HTTPS endpoint worked through the proxy; the deployed installer refused keyed :8443.
• Migration: Environment changes and proxy-CA rotation both preserved the proof file; proxy rotation changed egress/instance labels while retaining gateway trust and environment. Recovery completed through the retained archive, and the migration record cleared after verification.
• Live isolation: The offline make test-credential-isolation check and equivalent live checks passed. The guest returned candidate hashes; comparison with the real key stayed local. No provider key was found, metadata-only mounts and proxy readiness were verified, and apply temporarily added then removed its admin CA and runtime directory. Automatic approval review blocked the standard script because it sends provider-key bytes into the guest; the replacement sent no target key, digest, or key length to the VM.
• Keyless HTTP — failed live check: The proxy became Ready with the HTTP-only custom provider, and the fixture returned HTTP 200 from the VM. Requests from the sandbox timed out, and owner agent calls (including a fresh session) returned HTTP 500 with credential_unavailable in supervisor diagnostics. The fixture received no agent chat request. Tracked in #138; keyless HTTP sandbox inference is not claimed as validated.

Earlier live validation also passed the 35 NetworkPolicy checks and normal-owner CLI/TUI access. A dummy Brave key was used with web search disabled; no live web search was attempted. An OpenAI-compatible stub was used for the HTTP test; actual vLLM serving was not tested. Optional MCP checks and browser SSO login were not completed.

Deployment and security notes

• Set a digest-pinned proxy image through defaults.openshellSaw.credentialProxy.image for GitOps or CREDENTIAL_PROXY_IMAGE for quickstart. Pinning the published default image is a post-merge follow-up.
• Environment and proxy URL/CA changes preserve /sandbox through archive/restore. Missing or changed gateway CA trust requires recreation that wipes /sandbox. Rotate provider keys that previously reached a VM; see docs/credential-isolation.md for the migration procedure.
• VM root can observe temporary admin credentials during apply or alter the running gateway and its trust. Persistent user/relay keys and the launch JWT signing key remain in the VM; they do not carry platform-admin mTLS authority. The security boundary and residual risks are documented in docs/credential-isolation.md.