From 20010285841d12478462bfeb320936319e2e6a20 Mon Sep 17 00:00:00 2001 From: vthwang Date: Wed, 14 Oct 2026 22:34:57 -0700 Subject: [PATCH 1/4] docs: plan pooled VTA provisioning and claim lifecycle Define event handover with permanent resource isolation and optional Portal ownership binding. Signed-off-by: vthwang --- docs/pooled-vta-claim-design.md | 430 ++++++++++++++++++++++++++++++++ 1 file changed, 430 insertions(+) create mode 100644 docs/pooled-vta-claim-design.md diff --git a/docs/pooled-vta-claim-design.md b/docs/pooled-vta-claim-design.md new file mode 100644 index 0000000..afadcb6 --- /dev/null +++ b/docs/pooled-vta-claim-design.md @@ -0,0 +1,430 @@ +# Pooled VTA: Event Pre-provisioning and Claim Design + +- Status: Draft for implementation. Event flow, retention, and permanent per-slot infrastructure ownership are confirmed; the Portal binding wire contract remains proposed below. +- Date: 2026-10-14. +- Goal: Prepare VTAs before an event so attendees can claim them through Keyring with minimal waiting and no mandatory registration. +- Reference: [Pooled VTA Claim Flow](https://claude.ai/artifact/5JGCqdwwWjT5Pc11ugp3rV), reviewed against this repository. The page references `~/Projects/one-step-vta.md`; that file was unavailable at the stated local path and has not been reviewed. +- Existing contracts: [Mobile Integration](mobile-connection-app-api.md) and [VTA Claim Design](vta-claim-implementation-design.md). New endpoints, tables, and policies below are proposals, not existing API behavior. + +## 1. Confirmed Product Requirements + +The prototype parks each `vta_only` slot at `vta_setup_complete`: its seed, PVC, configuration, DNS, and published DID are prepared, but no VTA Deployment exists. The attendee confirms the claim before the Farm imports the administrator DID and starts the VTA. + +The product owner confirmed this parked model. There is no idle VTA Deployment and therefore no per-slot running VTA CPU/memory request. PVCs and other persistent resources remain; build Jobs and shared infrastructure still consume compute. After confirmation, DID-host ACL setup, the import Job, Deployment startup, and readiness checks still take time. + +The primary event flow is a **staff-operated station**: an admin displays a claim QR on a computer, the attendee scans it with Keyring, and the admin compares the attendee's check with the station's check before confirming. A fresh QR is used for the next attendee. + +The intended operating rule is one VTA per attendee, supervised by staff. A phone or wallet is not a unique-person credential: Keyring can create fresh keys, so this is not a cryptographic one-person limit. + +After the event, admins can batch-clean unused inventory. Successfully claimed VTAs remain running and are excluded from that cleanup, even if their holders never enroll in the Portal. Holders can later prove ownership and attach their VTA to a Farm Portal account. Until then, Farm admins manage the deployment. + +Each pre-built VTA gets its own internal resource owner immediately. Create its namespace, PVC, and Vault secrets under that owner from the first build; keep these locations unchanged through claim and later Portal attachment. Do not stage attendee resources under the platform user, migrate secrets at claim time, or turn the resource owner into a login account. The platform's mediator and DID-hosting services remain shared. + +## 2. Proposed Initial Scope + +- Support `vta_only` using a permanent Farm-managed domain and the existing platform stack's mediator and DID host. Give each slot its own owner, namespace, and seed. +- Let admins pre-build and inspect the pool, configure an event allocation and quota, open a staffed station, and disable further distribution. +- At the station, the admin selects Next attendee to reserve one slot. No email, Farm account, or passkey is required on the attendee's phone. +- Keyring claims the displayed QR. The station hides the QR after the first successful claim and displays the comparison check; the admin confirms or rejects it. +- Reuse existing claim/redeem Trust Tasks unchanged. Later Portal ownership verification is a separate, explicitly scoped ceremony requiring Keyring integration. +- Initially allocate ready slots only and show retryable unavailability when empty. A self-service event-link page and FIFO waiting queue are deferred; neither is necessary for the staffed-station flow. +- Retain handed-over VTAs after the event and provide both new-account enrollment and existing-account linking for Portal management. + +Full-stack pools, custom domains, post-claim DID-hostname changes, and unbounded automatic expansion are outside the proposed initial scope. This document does not implement the feature. + +## 3. Existing Implementation and Gaps + +| Source | Reusable behavior | Required pool work | +| --- | --- | --- | +| [`internal/setup/orchestrator.go`](../internal/setup/orchestrator.go) | VTA-only setup, DID publication, administrator import, Deployment readiness | Explicit pool readiness, build recovery, bounded batch concurrency | +| [`internal/connection/store.go`](../internal/connection/store.go) | Five-minute QR and confirmation windows, claim checks, session locks, claim/confirm/reject transitions | Atomic coordination with event quotas, slot ownership, and reclamation | +| [`internal/handler/mobile_connection.go`](../internal/handler/mobile_connection.go) | Trigger links and browser state views | Existing authorization requires the signed-in owner's `user_id`; station operations require admin authority instead | +| [`internal/handler/trust_task.go`](../internal/handler/trust_task.go), [`internal/trusttask`](../internal/trusttask) | Wallet proof validation, signed replies, replay handling, redeem long polling | Pool transaction integration, failure handling, and handover protection | +| [`internal/setup/initial_provision.go`](../internal/setup/initial_provision.go) | Persistent provisioning state, cross-replica advisory locks, cancellation during deletion | Recover confirmed claims before provisioning is queued; persist pool build/reaper work independently of process memory | +| [`internal/handler/capacity_service.go`](../internal/handler/capacity_service.go) | Resource estimates | Account for parked stock, concurrent activation, and retained VTAs; estimates do not reserve capacity atomically | +| [Provisioning Load Test](vta-provisioning-load-test.md) | Batch and cleanup patterns | Load tests share a system owner and use an administrator key that cannot be handed over; their sessions are not attendee inventory | +| [`internal/model/user.go`](../internal/model/user.go), [`internal/handler/user.go`](../internal/handler/user.go) | Auto-increment user IDs and an admin `system` flag | Add explicit account classification; current system detection only recognizes platform/load-test names; separate human counts from resource-owner counts | +| [`internal/handler/setup.go`](../internal/handler/setup.go), [`migrations/000003_create_setup_sessions.up.sql`](../migrations/000003_create_setup_sessions.up.sql) | Session teardown and user FK | External teardown is partly best-effort; deleting a user cascades session rows without cleaning external resources; owner removal must be guarded and last | +| [`internal/handler/signup.go`](../internal/handler/signup.go), [`internal/handler/passkey.go`](../internal/handler/passkey.go) | Passkey login/enrollment and user accounts | Email is unverified and signup resumes accounts without passkeys; pooled ownership must never be exposed through that email-only resume path | + +Two implementation gaps directly affect the prototype: + +1. **`vta_setup_complete` does not prove claim readiness.** The orchestrator writes this status before publishing the DID. Its comments identify a crash window that can leave the DID unpublished. Set `pool_slots.ready` only after publication and DID-resolution checks succeed, with recoverable publication or quarantine after interruption. On a publication conflict, verify the existing document and ownership; neither overwrite blindly nor treat every conflict as success. +2. **The current claim flow has no first-DIDComm-connection evidence.** An older migration contains `connected_at`, but the current `MobileConnection` model and claim/redeem flow do not maintain it. `released` means provisioning completed, not that the wallet received the response or connected over DIDComm. The prototype's cleanup rule cannot simply query `connected_at`. + +The prototype also includes `didLogUrl` in its success response. Current `redeemOutcome` returns the handle, VTA DID, operation, and optional mediator DID. Follow the generated protocol schema; do not add response fields without confirming protocol and Keyring support. + +## 4. QR Codes, Station Flow, and Competing Scans + +```text +Admin pre-builds -> slot ready -> admin opens event station + -> Next attendee -> reserve one slot -> display a fresh trigger QR + -> attendee's Keyring claims -> admin compares and confirms + -> administrator import and VTA startup -> redeem succeeds -> Keyring uses VTA + -> admin selects Next attendee -> different slot and fresh QR +``` + +### What the Prototype Says, and What Changes + +| QR or link | Reuse | Meaning | +| --- | --- | --- | +| Prototype event link | Shared across attendees | Opens a Farm browser page, which creates a separate reservation and trigger link for each attendee | +| VTA trigger QR | One reservation and one winning claimant | Contains a fresh `_id` handle and expiry; this is what Keyring opens | +| Proposed station page | Same computer/page across attendees | Admin-controlled UI; each Next attendee action produces a new reservation and trigger QR | + +The prototype does not give everyone the same reusable VTA-claim QR. The requested station replaces its attendee-browser confirmation with staff confirmation while retaining the per-reservation trigger link. A static event-link poster is not the station's claim QR, and the existing VTA-claim flow must never be reused to add successive attendees to an already-running VTA through `grant_acl`. + +Two phones can scan a visible QR, but the atomic claim transition admits only the first valid claimant. The other receives `alreadyClaimed`. The first scan has only locked an empty slot: it has not installed a key or received the VTA. Staff compare the check on the intended attendee's phone with the station. On mismatch or an unexpected claimant, staff reject and issue a new handle. Never confirm because someone merely scanned successfully. + +Once confirmed, that slot is permanently removed from allocation. A later attendee gets another slot and handle. Reject/expiry before confirmation may safely reuse the untouched slot, but never the old handle. QR photographs can cause a temporary reservation race, not a second successful handover; the comparison and staff confirmation are essential. + +### Station Authorization and Continuity + +All station mutations require the existing admin login, event/station scope, Origin validation, and CSRF protection. The event ID, reservation ID, QR handle, wallet proof, and attendee email cannot authorize station confirmation. The attendee-facing display contains only the QR, expiry, progress, and comparison code; it never exposes admin credentials or creates a Portal login on the shared computer. + +Persist the station's current reservation and display generation. Refreshing resumes it; concurrent Next attendee clicks are idempotent. Confirm/reject requests name the reservation and expected generation, preventing a stale tab from confirming the next attendee. Allow at most one unconfirmed reservation per station. After confirmation commits, staff can move to the next attendee while the previous activation continues; previous work remains visible in event history and Keyring continues redeem polling. Do not rotate a claimed QR automatically during comparison. + +Multiple stations allocate from the same pool transactionally and never share one active handle. An expired admin session requires staff reauthentication; it does not transfer confirmation authority to the attendee or discard a confirmed activation. Leaving the station must lock its controls. + +Keep trigger links, cookies, and claim payloads out of access logs, panic dumps, analytics, error tracking, and Referer headers. Set `Cache-Control: no-store` and `Referrer-Policy: no-referrer`, omit third-party tracking, and check CDN/Ingress and outer API loggers. Store new browser onboarding secrets as hashes; existing claim-handle storage remains governed by the current contract and restricted server access. + +Staff enforce the expected one-person-one-VTA practice. No device fingerprinting or phone identifier is proposed. A stricter guarantee would require individual admission tickets or another agreed eligibility mechanism. + +## 5. Data Model and State + +These are logical entities. Implementation should use additive migrations and existing relationships without reserving migration numbers in this draft. + +| Entity | Proposed fields and invariants | +| --- | --- | +| Pool configuration | Target ready count, total inventory/retention ceilings, build and activation concurrency, pinned image, provider snapshot, pause flag | +| `users.account_kind` | Proposed non-null classification: `person`, `platform`, `load_test`, or `pool_owner`; internal owners cannot authenticate or acquire personal credentials | +| `pool_build_batches` | Requested count, creating admin, optional event allocation, frozen profile, request digest/idempotency key, admitted capacity, progress counts; each accepted member has a stable ordinal | +| `pool_slots` | Unique live `setup_session_id` and `resource_owner_user_id`, unique `(batch_id, member_ordinal)`, optional allocated event, state, generation, readiness/handover timestamps, failure reason, worker lease and cleanup progress | +| `pool_events` | ID, label, maximum uses, held/committed counters, expiry/disabled timestamps, creating admin; public event tokens are unnecessary for staffed distribution | +| `pool_stations` | Event, creating/operator admin, current reservation, display generation, enabled state; at most one unconfirmed reservation | +| `pool_reservations` | ID, event/slot/station/mobile-connection IDs, state, deadlines, confirming admin, quota-settlement marker, idempotency-key digest | +| `setup_sessions.portal_user_id` | Nullable FK to the real Portal user; immutable infrastructure ownership remains in `user_id` | +| Pool ownership entitlement | Immutable session/reservation/VTA DID/claimant DID relationship, handover timestamp, revoked/linked state; survives short-lived QR/replay cleanup | +| Portal binding attempts | Entitlement, target account or provisional enrollment ID, browser-secret hash, nonce, purpose, expiry, verified/consumed state | +| Audit records | Actor, event, slot/reservation/session IDs, timestamps, transitions; no bearer credentials or private keys | +| Cleanup manifest/tombstone | Immutable original owner/session IDs and exact external resource identities, per-step results, deletion actor/reason, retry state; survives removal of live rows without cascading away | + +### Infrastructure Owner Versus Portal Owner + +Create one internal non-login `users` row per slot, explicitly classified as a pool resource owner, with a unique generated ID, no email, and no passkey. Use its numeric ID in `setup_sessions.user_id`; retain it for that deployment's lifetime. Namespace, ServiceAccount, Vault policy, and seed paths continue to use this value. Never use the admin's ID, one shared platform user, the wallet DID string, or a guessed email as this owner. + +Store the attendee's real account in a separate nullable `setup_sessions.portal_user_id`. This permits an existing account to manage its newly claimed VTA without moving resources or changing namespace identity. For example: + +```text +Pre-build: session.user_id = 4102; portal_user_id = NULL +Claim + running: session.user_id = 4102; portal_user_id = NULL +Portal binding: session.user_id = 4102; portal_user_id = 87 +``` + +Before binding, only Farm admins can manage this session through Portal APIs; the attendee already controls the VTA through Keyring. After binding, user 87 can manage it, while Farm admins retain their existing operator access. Admin UI should distinguish infrastructure owner, claimant DID, and Portal account. Portal enrollment is optional for keeping the VTA. + +This separation requires an authorization migration, not just a new column. Backfill `portal_user_id = user_id` for existing ordinary user sessions, leave platform/load-test/pool system sessions unbound, and set both IDs for newly created ordinary sessions. User-facing authorization must use `portal_user_id` exclusively after cutover; no fallback for NULL pooled owners. Update list/detail/delete, configuration/export/resources/logs, administrator grants, mobile connections, upgrade creation/status/cancellation, and user quota/profile queries. Keep namespace selection, Vault access, last-session cleanup, and resource-owner deletion keyed by the original `session.UserID`. Audit direct `user_id` queries as well as shared helpers. + +Pool resource owners are never login principals and must be excluded from signup, passkey enrollment, and account-recovery issuance. Portal-account deletion cannot cascade-delete or silently return a deployment to the pool. Retain bound deployments for explicit operator disposition. Initial binding is unique and atomic; already-bound sessions cannot be reassigned by repeating a wallet proof. + +Attaching an already-provisioned VTA must not be blocked by the normal two-VTA creation limit: binding allocates no new infrastructure. Count it toward the account's subsequent creation limit without granting `fullstack_access`; event quota continues to govern event issuance. + +### Permanent Resource Placement + +The following example uses resource owner `4102`, setup session `730`, and eventual Portal user `87`. The IDs are illustrative, not values to reserve or configure. + +| Resource or identity | Pre-build placement | After claim and Portal binding | +| --- | --- | --- | +| Infrastructure owner | `users.id = 4102`, `account_kind = pool_owner` | Unchanged; still a non-login owner | +| Session ownership | `user_id = 4102`, `portal_user_id = NULL` | `user_id = 4102`, `portal_user_id = 87` | +| Kubernetes namespace | `-4102`, normally `vtafarm-user-4102` | Unchanged | +| Workload identity | ServiceAccount `vta` in that namespace | Same namespace-scoped identity | +| Persistent data | PVC `vta-data-730` in that namespace | Same PVC and data | +| Vault policy/auth role | `vta-user-4102`, bound to that namespace and ServiceAccount | Unchanged | +| Vault KV prefix | `/vta/user-4102/session-730/` | Unchanged; never moved under `user-87` | +| Master seed | `master-seed` beneath that prefix | Same seed; no regeneration or copying during claim | +| VTA hostname and DID | Created once from the session name/provider snapshot | Unchanged | + +Vault paths above describe logical KV locations, not raw KV v2 HTTP API routes. Reuse `UserNamespace`, `VtaPVCName`, `UserName`, `VtaPrefix`, and `SeedPath`; do not introduce a second naming algorithm for pooled sessions. A dedicated owner means the existing per-user Vault policy cannot overlap another attendee's or the platform user's prefixes. Reusing the shared mediator/DID host does not grant access to their Vault keys. Namespace separation does not, by itself, assert network isolation or a separate Vault server per attendee. + +Build sequence after database admission: + +1. Load the committed slot, owner, session, and frozen provider/profile. Record expected external resource identities in the cleanup manifest before issuing side effects. +2. Ensure the owner's namespace, ServiceAccounts, and RBAC. On an existing resource, verify ownership before reuse; `AlreadyExists` alone is not ownership evidence. Record Kubernetes UIDs where available. +3. Ensure the owner's Vault policy and Kubernetes-auth role. Render setup configuration with that exact owner's seed path and role. +4. Provision session DNS and the PVC/configuration/setup Job in that namespace, then publish and verify the DID through the snapshotted shared provider. Never fall back to the platform namespace or seed path on failure. +5. Mark the slot ready only after all readiness checks complete and setup work no longer consumes running compute. Keep the PVC and required configuration; no VTA Deployment runs while parked. +6. On staff confirmation, run administrator import and start the Deployment against the same PVC and owner. Portal binding later updates management authorization only. + +Persist the cluster/environment identity, namespace name/UID, Vault KV/auth mounts and logical prefix/role, DNS record IDs, and DID-host provider identifiers needed for recovery. These are protected operational metadata, not user-visible secrets. A later configuration change must not redirect cleanup to a different cluster, mount, or namespace prefix; fail closed if the original target cannot be resolved. Do not store seed values or access credentials in the manifest. + +### User ID Allocation and a 50-Slot Batch + +`users.id` is currently a PostgreSQL `BIGSERIAL` primary key. Personal accounts, the platform account, and pool owners use the same database-generated sequence. Creating an event alone allocates no owners; accepting its explicit 50-slot build batch creates 50 real owner rows, not an application-side reservation of an ID range. + +For illustration, if the last allocated value is 20, no other inserts occur, and no values are skipped, the 50 owners receive 21 through 70; a later ordinary signup receives 71. Twenty existing users does not necessarily mean the sequence is at 20. With concurrent signup or another batch, IDs may interleave; transaction rollback or conflicts can leave gaps. Never promise consecutive IDs or derive batch membership from an ID interval. PostgreSQL allocates sequence values atomically across concurrent callers and does not generally reclaim aborted values. See [PostgreSQL sequence semantics](https://www.postgresql.org/docs/18/functions-sequence.html). + +Implementation rules: + +- Omit `id` on inserts and use the generated value returned by the database/ORM. Do not calculate `MAX(id) + 1`, manually insert a reserved range, reset the sequence, decrement it after cleanup, or use sequence values as an attendee count. +- Keep the numeric ID immutable. Deleted owner IDs are never intentionally reused for new inventory. An actual sequence misconfiguration after an import/restore requires a separate maintenance procedure, not a fallback in signup or pool creation. +- Generate `users.unique_id` independently within its existing 12-character column limit, enforce its unique index, and retry collisions. Account kind is stored explicitly, not encoded in a long prefix or inferred from an ID range. +- Persist exact owner/session/slot associations. Use a batch member ordinal for progress such as `17 of 50`; gaps in primary keys must not affect ordering, cleanup selection, or metrics. +- Allocate IDs only as part of committed ownership records before creating external resources. A rolled-back insert may consume a number, but it must not leave a namespace or Vault path based on an uncommitted owner. + +### Atomic Admission, Build Retries, and Account Classification + +For a bounded 50-slot request, resolve an existing idempotency outcome before attempting new admission. For a new request, validate the profile/event and reserve capacity under a shared database lock within the admission transaction. Persist the idempotency record, batch, 50 owner rows, 50 initial setup-session rows, 50 queued slots, and durable build-work records together. Commit before responding `202` and before workers make external calls. If existing setup creation requires DNS before its row insert, split database admission from external provisioning for this path; do not run the public signup or ordinary create handler 50 times. Existing startup recovery must not bypass the pool scheduler or activate queued/parked sessions; integrate both paths under the same work ownership and deletion fencing. + +The `202` guarantee is **50 durable members admitted**, not 50 ready VTAs. Cap the request count so admission stays bounded; larger inventories use multiple explicit batches. A database failure commits no partial batch, although unused sequence values may remain. The same actor/idempotency key and payload return the same batch; a changed payload conflicts. An HTTP timeout after commit must not create another 50 owners on retry. + +Workers update each existing member independently. A temporary build failure retries the same owner/session and inspects existing resources before continuing; it does not create another user. If the original identity cannot be safely recovered, quarantine and explicitly retire it before creating a separately tracked replacement with new owner/session/name/seed. Report failed and retired members separately from ready inventory. Periodically reconcile queued, failed, and retiring records; an empty namespace or old timestamp alone is not permission to delete an owner. + +Add `account_kind` using an additive, reviewed backfill: identify the platform owner and legacy load-test owners from existing provenance, classify existing ordinary accounts as `person`, and create future pooled owners as `pool_owner`. Do not classify by `email IS NULL`, because legitimate personal accounts can have no email. Preserve the existing admin `system` response as a derived compatibility field and add kind/filter support without silently changing the list's default semantics. Show human-account and internal-owner totals separately; derive personal quotas from Portal ownership, not total `users` rows. + +Enforce non-login status server-side across login/session issuance, passkey registration, invitation/account adoption, profile changes, recovery-link creation/consumption, and administrative personal-account controls. A pool owner always has no email, no personal passkeys, and no fullstack-access grant. Use database checks where possible and service-level checks for cross-table invariants. Do not upgrade it into a person at claim or Portal enrollment. The real Portal account gets a separate row or is an existing authenticated person account. + +Each pooled owner has exactly one lifetime slot association and at most one live setup session. Enforce unique live owner/slot and owner/session associations with constraints and a transactional cross-table guard; prevent ordinary creation paths from attaching another session or custom domain to that owner. Archive original numeric IDs in tombstones; live owner/session FKs may be cleared only in final retirement. Use restrictive FKs/guards so deleting an owner cannot cascade away a live slot or cleanup manifest. Check all references, not only a zero session count. + +### Stable DNS Naming (Proposed Default) + +Use the existing `[dev-]vta-.` shape with a generated lowercase random name, for example `vta-k7m2x9p4q8.`. Reserve the session name transactionally with collision retries and retain retired names in a registry checked by every name-allocation path, including ordinary setup. Keep the existing `-vta` DID path suffix. Do not include an email, wallet DID, numeric owner ID, event label, or temporary `pool` marker. A future display name may change without renaming DNS, the session route identifier, or the DID. The exact generated-name format remains an implementation default, not an already-shipped behavior. + +### State Responsibilities + +```text +slot: queued -> building -> ready -> reserved -> claimed -> activating -> handed_over + +reserved/claimed -> ready + only after rejection, cancellation, or expiry, before confirmation/key import + +uncertain state or possible partial key import -> quarantined -> retiring -> retired +``` + +Preserve existing `mobile_connections` states: `open -> claimed -> confirmed -> provisioning -> released`, plus rejected/lapsed and existing terminal states. `handed_over` is a pool allocation state, not a new Trust Task state. `released` means available for handover, never available for reuse in the pool. + +- The five-minute QR clock starts when a ready slot is assigned and a handle is created. Claim starts a separate five-minute confirmation window, using existing constants and database time. +- Before confirmation, reject/expiry/cancel atomically terminates the reservation and settles quota. Return the slot only after verifying that no administrator key was installed. Every subsequent reservation gets a new handle; old handles never revive. +- Confirm must serialize with reject, expiry, and cleanup. Once confirmation commits, the slot cannot be reassigned, even before the import Job starts. +- QR and comparison deadlines no longer cancel confirmed work. An HTTP timeout is not provisioning failure and must not instruct the wallet to delete its key. +- Retry with the same reservation, slot generation, and claimant. Inspect Job/ACL/Deployment state after interruption or partial success. Quarantine or rebuild a slot if key-import state is uncertain; do not return it directly to ready. + +### Quota and Idempotency + +Proposed accounting: `held + committed <= max_uses`. Allocation increments held; confirmation transfers held to committed. An unconfirmed reject/expiry refunds once. Compensating a confirmed failure requires evidence that it cannot leave a usable or still-starting VTA, plus an auditable settlement operation. + +Allocate the slot, create the reservation/mobile connection, reserve event quota, and save the idempotency outcome in one database transaction. Define one lock order compatible with the existing session-first claim/confirm operations; adding event locks in inconsistent order can deadlock. Use unique constraints to prevent multiple active reservations for a slot. Commit before waking workers. + +Retries for the same station generation return the existing reservation. Scope `Idempotency-Key` to actor, event, station generation, and operation; return 409 when the same key carries a different payload. Lost responses must be recoverable. Repeated confirmation of the same already-confirmed claim returns current progress without restarting the Job or incorrectly reporting expiration. + +## 6. Claiming Portal Management After the Event + +### Recommended Product Flow + +Use **wallet ownership verification followed by account enrollment**, not email-only ownership lookup and not a mandatory new general-purpose wallet-login system. + +For a new Portal user: + +1. On their own phone, open Farm Portal from the claimed VTA in Keyring, carrying only its public VTA DID as a hint. On a computer, select Connect my claimed VTA and enter the VTA DID shown in Keyring. The DID is not a credential. The Farm creates a short-lived onboarding attempt bound to that browser, the selected VTA, and a provisional new-account ID. It displays a separate Portal-binding QR or app link, clearly labeled as Portal access, not another VTA claim. +2. Keyring selects the previously claimed VTA, verifies the Farm request, displays the operation and browser comparison check, and asks the holder to approve. It signs the fresh ownership challenge with the retained claimant key. The Farm matches that proof against the successful event entitlement. +3. Only that browser receives a limited onboarding session for the verified VTA. It can collect an email and register a passkey, but cannot list other users' resources, obtain a general user JWT, or modify the VTA yet. +4. The attendee enters an unused email and creates a passkey. Atomically save the new account's email/passkey, set `portal_user_id`, mark the entitlement linked, and consume the onboarding attempt. Only then issue the ordinary Farm user session and show the VTA in My VTAs. +5. Later login uses the passkey. Email may be an account hint/profile label, but typing it alone cannot authenticate or find claimable VTAs. + +For an existing Portal user, first authenticate with their passkey, then select Connect my claimed VTA and approve the wallet challenge. Bind the challenge to that exact account and show its identifying label in Keyring. Require a fresh passkey assertion for the binding, not merely a cookie issued by the existing public signup flow. After approval and browser confirmation, atomically attach the VTA to the authenticated account without changing the infrastructure owner. + +If the attendee never visits the Portal, keep `portal_user_id = NULL`. Their VTA keeps running, Keyring keeps working, and Farm admins retain management access. Portal enrollment is neither another allocation nor a condition for retention. + +### What the Proof Establishes + +At successful event handover, persist an entitlement containing the session ID, VTA DID, original claim reservation, claimant Ed25519 `did:key`, and confirming admin. Commit the entitlement, pool handover timestamp, and mobile release transition consistently in one transaction after readiness succeeds. A recovery worker must reconcile this even if both browsers disconnect; do not depend on a later status GET. Private keys remain in Keyring. This record is longer-lived than QR expiry and replay caches. + +The Portal challenge proves control of the private key recorded for that entitlement. Knowing a DID string, VTA DID, email, six-character check, or old trigger handle is insufficient. A saved claim/redeem signature cannot be replayed as a Portal-binding approval: purpose, audience, challenge, and browser binding are different. + +Only a completed handover entitlement may bind. Pending/failed/rejected claims cannot. Binding is single-assignment: the same account's retry returns its existing result; a different target receives a conflict. A later transfer or recovery is a separate audited action, not another claim. + +### Key Rotation and Long-Term Authority + +Do not assume the initial administrator DID remains the runtime administrator forever. Existing Farm code documents PNM rotation on first connection, and client onboarding supports replacing the ephemeral ACL identity. A cached ACL snapshot is not current ownership evidence, and refreshing that snapshot currently stops the VTA. + +For the initial Portal-binding design, require Keyring to retain the event claimant private key **as an unbound Farm ownership credential** until Portal attachment succeeds, even if it rotates its VTA runtime ACL key. This extends the existing mobile guide's key-retention requirement with an explicit purpose. Keyring must store the Farm DID, VTA DID, and original claimant relationship durably and sign only a Farm-verified, user-approved binding challenge. Verify this behavior in the actual mobile Keyring; browser-extension/PNM behavior is not evidence of mobile support. + +The Farm entitlement deliberately represents the right to attach the original event allocation, not current runtime ACL membership. Granting another VTA administrator does not transfer Farm hosting ownership. A runtime key swap does not silently revoke or transfer this entitlement. Provide an admin revocation/ownership-recovery procedure for a lost or compromised claim credential; key loss is never repaired by accepting an email assertion. + +Once Portal binding completes, that key cannot bind a second account, sign into the existing account, replace passkeys, or recover it. The Portal's passkey/recovery policy becomes authoritative for hosting management. Do not delete a key from Keyring merely because Portal binding succeeded if the key is still used for VTA administration or redeem recovery. + +If the actual Keyring discards the original key during onboarding, automated Portal binding is blocked until it either retains an appropriate Farm binding credential or implements a verified successor-key registration. A successor registration must authenticate the old entitlement key and prove possession of the new key, bind the Farm/VTA/purpose, and revoke the old binding credential atomically. A wallet-supplied new DID or an arbitrary current ACL administrator is not a substitute. This is a Keyring integration requirement, not a claim that it is already implemented. + +### Binding Ceremony Contract + +Keep `provision/vta/claim/0.1` and `provision/vta/redeem/0.1` unchanged. Define the Portal-binding ceremony separately before implementation. Its proposed semantic contract is: + +- Server-generated random nonce with at least 128 bits of entropy, five-minute expiry, single-use consumption, and server-side state. +- Explicit version, purpose (`enroll_new_portal_owner` or `bind_existing_portal_owner`), Farm DID/audience, exact Portal origin, attempt ID, browser-binding ID, VTA DID, claimant DID, target account/provisional ID, and deadline. Existing-account display identity is frozen for the attempt; changing the target requires a new challenge. +- A Farm-signed request verified by Keyring, and an Ed25519 proof over the complete canonical challenge under the recorded claimant key. Freeze the canonicalization, proof fields, and closed schema in the implementation contract; reuse existing cryptographic helpers only where their validation semantics match. +- Explicit wallet approval and a browser comparison check. The signing UI states that this grants Farm hosting management, including configuration/deletion rights, and does not grant staff access to the attendee's wallet. +- A random HttpOnly browser capability, hashed at rest and independent of the wallet-visible QR. Only its originating browser can poll private results and finish enrollment/linking. A wallet response never receives or forwards a Portal JWT to the station or another browser. +- A separate short-lived request-read handle in the QR/app link lets Keyring fetch the signed challenge without the browser cookie. It exposes only the request needed for informed approval, cannot finish enrollment, and is also redacted from logs. Polling/completion still require the private browser capability. +- Verification and final linking each recheck entitlement state, expiry, target account, and deletion/revocation. Linking and challenge consumption share one transaction. Replayed proofs, changed targets, and stale attempts cannot transfer ownership. + +This is a proposed Farm HTTPS ceremony, not a registered Trust Task or an assertion that Keyring already recognizes a new trigger-link type. Its app-link dispatch and signed schema are implementation dependencies. Do not put a Portal-binding request into a VTA-claim QR and expect existing handlers to interpret it. + +The existing [auth/oob/identify specification](https://github.com/trustoverip/dtgwg-trust-tasks-tf/blob/main/specs/auth/oob/identify/0.1/spec.md) verifies an already recognized member and participates in a larger approval flow; it is not, by itself, proof that the member owns this Farm allocation. Therefore generic VTI sign-in is not assumed to solve initial Portal attachment. It may be integrated later once its subject is securely mapped to Farm accounts and entitlements. + +### Email, Passkey, and Interrupted Enrollment + +The current Farm sends no verification email. Email is a self-declared label; passkeys authenticate accounts. Preserve that meaning in this feature. Email OTP/magic-link login would be a separate authentication and delivery-system change, not implied by this design. + +There is a concrete integration hazard: `SignupHandler.resumeOrReject` issues a full user session for an email whose account has no passkeys. Never write a pooled owner's email or attach their VTA to such an account before credential setup. Otherwise another person could enter the same email and resume an account that already controls the VTA. + +Keep new onboarding accounts isolated with no public email mapping or ordinary login session until successful WebAuthn completion. Use a purpose-limited registration context rather than the generic signup-issued JWT. Commit email uniqueness, passkey persistence, account activation, ownership binding, and attempt consumption together. If registration or the final transaction fails, leave the VTA unbound and recoverable through a fresh wallet challenge; it must not become reclaimable event inventory. + +If the requested email already exists, require authentication to that existing account and a fresh challenge targeting it, or choose another unused email. Never merge based on the email. An existing email-only account that cannot provide a passkey requires the ordinary recovery/onboarding resolution without attaching the VTA; do not trust its old signup cookies. Concurrent enrollment for the same email must have one database winner and no partial binding. Expired pending accounts can be cleaned independently of VTA resources. + +## 7. Proposed API + +The primary frontend is an admin event/station page. REST paths below are under `/api/v1`. The existing wallet endpoint remains `POST /trust-task/trust-tasks`. Public event-link/self-service allocation routes are deferred. + +| Method/path under `/api/v1` | Authority | Behavior | +| --- | --- | --- | +| `GET /admin/pool` | Admin | Inventory, active work, quarantine, retained count, budget, dependency state | +| `POST /admin/pool/slots` | Admin | `{count,event_id?}` with the configured pool profile and required idempotency key; 202 only after all requested owner/session/slot rows and work records commit; batch ID, admitted count, and status URL | +| `GET /admin/pool/batches/:id` | Admin | Requested/admitted/queued/building/ready/failed/retired counts and paginated member details; exact member identities, not an owner-ID range | +| `GET /admin/pool/slots` | Admin | Paginated inventory with state/batch/event filters; no bearer credentials | +| `POST /admin/pool/events` | Admin | `{label,max_uses,expires_at}`; 201; create a bounded distribution event | +| `GET /admin/pool/events` | Admin | Paginated events, quota, expiry, and disabled state | +| `PUT /admin/pool/events/:id/status` | Admin | `{enabled:false}` stops new reservations; does not delete resources or cancel confirmed delivery | +| `POST /admin/pool/events/:id/stations` | Admin | Create an event station; return station ID and page URL | +| `GET /admin/pool/stations/:id` | Admin with station scope | Current reservation/generation, QR while open, comparison check after claim, and progress | +| `POST /admin/pool/stations/:id/reservations` | Admin with station scope and CSRF protection | Next attendee; expected generation and idempotency key; 201 new or 200 resumed reservation | +| `POST /admin/pool/reservations/:id/confirm` | Admin with station scope and CSRF protection | Confirm the frozen reservation/generation; 202 activation; do not accept a replacement admin DID | +| `POST /admin/pool/reservations/:id/reject` | Admin with station scope and CSRF protection | 200 rejection before confirmation; next issuance gets a fresh handle | +| `DELETE /admin/pool/reservations/:id` | Admin with station scope and CSRF protection | 204 cancellation before claim; 409 for later states | +| `GET /admin/pool/events/:id/cleanup-preview` | Admin | Eligible/protected resources and a short-lived versioned plan ID | +| `POST /admin/pool/events/:id/cleanup` | Admin | Submit plan ID; recheck eligibility; 409 for stale plans, otherwise 202 with cleanup-job URL | +| `GET /admin/pool/cleanup-jobs/:id` | Admin | Progress, item failures, and retry status | +| `GET /admin/users` (extension) | Admin | Preserve the existing list contract; expose account kind and optional kind filtering, with internal owners marked `system: true` | +| `POST /portal-bindings` | Own browser capability/origin; recent passkey proof for an existing target account | Create a purpose-bound ownership challenge; no general user login | +| `GET /portal-bindings/:id` | Originating browser capability | Challenge, verified state, expiry, and allowed next step; no public ownership lookup | +| `GET /portal-binding-requests/:handle` | Short-lived request-read handle | Return the Farm-signed request for Keyring approval; no browser session or completion authority | +| `POST /portal-bindings/:id/proof` | Fresh claimant signature | Verify the separate ownership ceremony; never return a browser login token to the wallet | +| `POST /portal-bindings/:id/passkeys/begin` | Verified limited onboarding context | Start new-account WebAuthn registration | +| `POST /portal-bindings/:id/complete` | Same browser; valid WebAuthn finish or fresh existing-account authentication | Atomically consume proof, finalize account if new, and attach the VTA; only then issue the browser session | + +GET must not provision resources or consume quota. Proposed HTTP errors are 400 for invalid fields, 401 for missing authentication, 403 for insufficient authority or CSRF/origin violations, 404 for unknown or non-owned resources, 409 for stale station generations, `event_exhausted`, `already_bound`, or target/email conflicts, 410 for expired/disabled events or expired attempts, 429 for rate limiting, and 503 for `pool_empty`/`at_capacity`/`dns_budget_exhausted`/`shared_infra_unavailable`. State GET may return 200 with temporary unavailability; allocation POST returns 503. Failed ownership proofs must not enumerate which email owns a VTA. + +Browser REST errors follow the repository's `error`/`reason` convention: + +```json +{"error":"No VTA is ready yet.","reason":"pool_empty","retry_after_seconds":10} +``` + +Include `Retry-After` for 429 and retryable 503 responses. Station views may include `reservation_id`, `generation`, `status`, `server_time`, `expires_at`, `confirm_by`, and `next_poll_after_seconds`. Only authorized station admins receive its trigger link for display. Use RFC 3339 timestamps in REST; preserve Unix seconds for VTI `_exp` and the existing Trust Task `confirmBy` field. + +Trust Tasks retain their signed response/error schemas, including HTTP 422 for redeem pending; do not apply browser REST errors to that endpoint. Each new redeem poll uses a new document ID, while retransmitting the same document uses replay handling. The original claimant must be able to redeem again after a lost success response; define a recovery-retention window before pruning handover/replay records. + +When implementing routes, update `internal/apidocs/openapi.yaml` and the mobile integration guide together. Do not publish these proposed endpoints in OpenAPI before implementation. Add required idempotency/CSRF headers to the existing CORS allowlist and restrict origins to the deployed frontend. + +## 8. Workers, Capacity, and Event Traffic + +- Persist build, activation, and cleanup work. Use cross-replica locks/leases and generation checks so a stale worker cannot modify a reassigned slot. External operations need stable resource identities and inspectable retry behavior. +- Pool readiness requires completed PVC/configuration work, verified DID publication/resolution, healthy shared dependencies, and no attendee administrator key. Recheck readiness freshness before claim. +- Retain the original provider URL/DID snapshot. Rebuilding the platform stack must not silently rewrite existing identities. Stop allocation when shared hosting or mediation is unavailable and report the reason. +- Configure target ready inventory, build/activation concurrency, storage/DNS budgets, and a total retained/running-VTA ceiling separately. Kept VTAs remain part of the capacity budget; refill cannot exceed the ceiling. +- Parked PVC count is not simultaneous activation capacity. Reserve enough runtime headroom for admitted claims. Existing cached, fail-open estimates are insufficient for atomic batch admission. +- Treat the prototype's concurrency of 8–10 as a candidate, not a measured safe value. Measure build/activation P50/P95, failure rate, and shared-service load. Prioritize attendee activation over background refill. +- Shared event Wi-Fi concentrates traffic behind one IP. Test existing Trust Task per-IP limits and long-poll HTTP/database connection use. Combine event, browser, handle, and global limits; neither IP-only nor freely generated DID-only limits are sufficient. +- Keep pool and DID-host domains stable after the event. For a different zone/subdomain, verify DNS authority and TLS coverage first: a single-level wildcard does not cover arbitrary deeper subdomains. Avoid per-slot certificate issuance. + +If FIFO is included initially, create expiring waiting tickets that atomically hold quota, cap queue depth, and define event fairness/cancellation. Order tickets in the database. Assign a slot and start the QR timer only when it is ready; a failed build must not strand a ticket. Expire abandoned tickets and refund once. ETA is an estimate and may be unknown. Do not admit unlimited waiting work without activation capacity. + +## 9. Retention and Cleanup + +The confirmed requirement is to batch-clean incomplete event claims and leave successfully claimed VTAs alone. A successful claim here means staff confirmation followed by completed provisioning (`released`/pool `handed_over`); the earlier wallet `claimed` state alone is incomplete. Portal account binding and first DIDComm connection are not cleanup prerequisites. + +| Event resource state | Batch cleanup decision | +| --- | --- | +| Event-allocated, unused queued/ready/building slot | Eligible after distribution closes; cancel builds and verify shutdown before removal | +| Open or wallet-claimed but unconfirmed | Terminate reservation atomically, invalidate handle, settle quota, then clean the untouched slot | +| Confirmed or activating | Protect automatically; operator review/retry because a key may already be installed | +| Handed over, with or without Portal account | Always excluded from event cleanup | +| Previously handed over but now stopped/unhealthy | Still excluded; current runtime status is not ownership state | +| Failed/uncertain activation | Quarantine; explicit separate disposition before destruction, never direct reuse | + +Record durable `handed_over_at`/entitlement state, not just `status == running`. A claimed VTA may later crash, stop, or upgrade and must not suddenly look like unused inventory. Similarly, `portal_user_id IS NULL` does not mean unclaimed. Do not use the old `connected_at IS NULL` condition. + +Disable new reservations before generating an event cleanup preview. Protect confirmed, handed-over, and uncertain slots, and exclude inventory reassigned to another event. Give batches/slots explicit event allocation when preparing event stock; otherwise unused slots that never had a reservation have no event association to clean by. Shared unallocated stock requires a separate pool-inventory cleanup selection. Re-lock and recheck each item during execution; a preview is not a permanent authorization snapshot. Never delete namespaces solely because they once belonged to an event. + +Unconfirmed, untouched reservations may return to shared inventory. Shrinking that inventory is a separate cleanup policy. A failed slot that may contain an attendee key must be explicitly disposed of and rebuilt, never reissued with a new handle. Track DNS, DID/ACL, Kubernetes, Vault, and owner cleanup independently. Existing teardown includes best-effort steps, so deleting a session row does not prove full cleanup. Retain tombstones, external resource identifiers, and retryable cleanup work until removal is verified. + +Refill runs only while enabled and within budget; ending an event should not automatically restore its peak inventory target. Closing event distribution, reclaiming unused inventory, and deleting a handed-over VTA are distinct operations in both the UI and API. + +### Ordered Retirement and Resource-Owner Removal + +The internal owner exists for the VTA's lifetime, not the attendee's login lifetime. Inactivity, a stopped Deployment, missing email, or an unbound Portal account are not orphan evidence. A handed-over VTA requires a separately authorized explicit deletion by its linked Portal user or a Farm admin. Deleting that VTA does not delete the person's account or other VTAs. Deleting a person's Portal account must not delete pooled infrastructure: use a restrictive `portal_user_id` FK until an explicit disposition is completed, and retain linked/revoked entitlement history so clearing a link cannot reopen initial binding. + +Implement retirement as durable work with these checkpoints: + +1. **Authorize and fence.** Re-lock the slot/session and recheck the cleanup plan, event membership, generation, and handover status. Mark `retiring`, block allocation/binding/activation/upgrades, and invalidate outstanding unconfirmed reservations. Serialize with build/provision workers; wait for active operations to stop. A lease timeout alone does not prove an external Job stopped. Preserve a cleanup manifest before deleting anything. +2. **Stop writers.** Cancel queued work and remove or stop all setup/import/maintenance Jobs and Deployments for this slot. Verify Pods have terminated and cannot recreate data. Record failure and retry if shutdown is uncertain; never clear the owner while stale work can still write. +3. **Remove external identities and routing.** Delete only this session's DNS records, shared DID-host document, and VTA ACL entry through the recorded provider. Do not delete the platform mediator/DID host, platform account, or shared credentials. Reconcile ambiguous provider responses rather than treating an HTTP timeout as success. +4. **Remove stored data and access.** Delete the session's Kubernetes resources/PVC and exact Vault secret subtree, including stored KV versions/metadata according to the deletion policy. Remove its exclusive Vault auth role/policy and verify the absence of effective workload access, including any issued token permissions. Do not equate role deletion alone with invalidating issued tokens. Check the storage reclaim policy: a deleted PVC may leave a retained backing volume or backup requiring explicit disposition. Never claim all data is erased while retained copies remain. +5. **Remove the namespace.** Require the expected namespace identity/labels, no unexpected workloads or other owners, and no remaining dependencies. Await actual namespace deletion; an accepted deletion request or `Terminating` status is not completion. Do not force-remove finalizers automatically. Quarantine mismatches for operator review. +6. **Finalize database retirement.** Only after all required external checkpoints are verified, mark the tombstone complete, invalidate binding entitlements/attempts, detach completed live FKs, and delete the setup-session and `pool_owner` rows in a transaction. Recheck owner kind, its exclusive slot association, absence of credentials/domains/other live references, and completion under lock. Preserve audit and original resource identifiers without deletion cascades. Keep the batch/event record and retired name reservation; do not reset any sequence. + +The current `setup_sessions.user_id ON DELETE CASCADE` is not a resource cleanup mechanism. Introduce a restrictive guard for pooled owners (for example, a live slot-to-owner `ON DELETE RESTRICT` FK plus guarded finalization), and route all pool deletions through this worker, including existing admin/user session-delete endpoints. Existing teardown must return inspectable step results rather than deleting the session after best-effort calls. A failure leaves the owner/session manifest discoverable as `cleanup_failed` progress on a retiring slot, with retry/backoff and operator diagnostics. Do not delete the original metadata just to make the dashboard appear empty. + +Recovery must handle the crash between an external success and saving its checkpoint: re-inspect the exact resource, treat verified absence as success, and retry remaining steps. Not-found is success only for a known target on a reachable, correct provider; authorization errors or missing provider configuration are not not-found. New creation and deletion share owner/slot locks so a zero-reference check cannot race with a new attachment. Owner rows with zero sessions but unfinished manifests remain protected; legacy orphan candidates require an explicit reviewed cleanup plan. + +For the 50-slot example, if 20 are handed over and 30 remain untouched, an event cleanup can retire those 30 and their internal owners. The 20 retained owners remain even if none of their holders enrolls in the Portal. IDs for the deleted 30 stay unused; future signups and new batches use newly generated IDs. Pausing a retained VTA preserves its owner/PVC/secrets; deletion is a different operation. + +## 10. Implementation Sequence and Acceptance + +1. Validate the Keyring retention/signing integration for later ownership proof; agree its concrete wire contract before implementing Portal binding. +2. Add account-kind classification, permanent owner/slot constraints, batch admission/idempotency, cleanup manifests, and pool/event/station/reservation models. Build directly in each committed owner's namespace/Vault path; address readiness/publication recovery and provide inventory views. +3. Implement station authorization and atomic slot/quota/replay transitions, provisioning recovery, quarantine, and per-attendee QR rotation. Preserve signed-in owner claim flows. +4. Migrate Portal-versus-infrastructure ownership authorization, then implement limited new-account onboarding and existing-account linking with passkey and wallet proof. +5. Add cleanup previews and checkpointed retirement with owner-last deletion, capacity ceilings, refill, and account-kind-aware operational visibility. Update OpenAPI/mobile documentation and rehearse station issuance, later Portal attachment, and cleanup in an isolated environment. Portal attachment is required for feature completion, even if station work is delivered first. + +Required acceptance cases: + +- A 50-slot admission creates exactly 50 distinct internal owners, sessions, and queued members before 202. Concurrent signup/batches remain unique even when IDs interleave. Rollbacks and deleted owners leave harmless gaps; no code derives ownership from a numeric interval. +- Retrying an accepted batch after a lost response returns the same members. A failed admission creates no committed partial batch or external resources; build retries reuse committed owner/session IDs without inserting new users. +- The namespace name/UID, PVC, ServiceAccount scope, Vault role/prefix, hostname, and VTA DID remain identical across ready, confirmed, running, and Portal-bound states. Claim/binding performs no seed copy, namespace move, or owner conversion. +- Pool owner A's workload cannot authenticate as owner B or the platform owner and cannot read/write their Vault paths. Shared mediator/DID-host integration remains functional. A conflicting existing namespace or mismatched role binding fails closed. +- Internal owners cannot log in, enroll passkeys, obtain recovery links, acquire email/fullstack access, or receive additional sessions/domains. Legacy personal accounts without email remain personal; UI filters/counts do not report pool owners as attendees. +- Cleanup protects claimed-but-unbound and stopped retained VTAs. An event with 20 handed-over and 30 unused slots retires only the 30 selected unused owners. Person accounts and unrelated resources are untouched. +- Failure injection at every retirement checkpoint preserves a resumable manifest and owner until completion. Test already-absent resources, namespace finalizers, retained storage, unavailable providers, DB finalization failure, and a crash after an external delete but before its checkpoint. +- Direct deletion of a live pool owner is rejected; cleanup never relies on cascading session deletion. Owner finalization races with claim, Portal binding, worker restart, and attempted new attachment cannot orphan resources or erase audit history. +- Backfill preserves ordinary-session access and quotas, isolates system owners, and rejects an unsafe rollback once pooled sessions depend on separate Portal ownership. Rehearse migration and rollback constraints on an isolated database before rollout. +- Multiple replicas compete for the last slot/event use: only one succeeds. Refreshes and lost responses do not consume additional quota. +- Event/station/reservation ID or trigger QR possession cannot authorize staff confirmation. Cross-origin mutations fail; stale tabs cannot confirm a newer display generation. +- Claim/reject/expiry and confirm/reaper/cleanup races cannot double-allocate or double-refund. Confirmed slots never return to ready. +- Recovery covers crashes between transaction commit and worker startup, partial DID publication, successful Jobs before DB updates, expired worker leases, and workers restarting during deletion. +- Invalid signatures, non-claimant redeem, replay, and expired handles retain existing protocol guarantees. The claimant can recover a lost successful response. +- Two phones scanning the station QR produce one claimant. A mismatch can be rejected safely; the next attendee gets a different handle and, after a successful handover, a different slot. Next attendee never grants another administrator on the previous VTA. +- Refreshing or reauthenticating the station resumes its current reservation. Keyring restarts and missed redeem responses preserve the original claimant and eventual entitlement. +- Provider, DNS, Vault, image-pull, PVC, and readiness failures produce accurate state and bounded retries; waiting is never reported as success. +- Disabling an event preserves confirmed handovers. Cleanup protects reassigned inventory, partial deletions, handed-over slots without Portal users, temporarily unhealthy handed-over VTAs, and ongoing activation. +- Portal binding rejects known DID strings without signatures, old claim proofs, wrong Farm/origin/purpose/target, expired challenges, and simultaneous attempts to bind different accounts. +- Runtime key rotation preserves the separate unbound ownership credential; a lost/revoked credential cannot be replaced by email, a new wallet DID, or a stale ACL snapshot. +- New-account email/passkey/binding finalization is atomic. Public signup, earlier signup cookies, duplicate-email races, and abandoned passkey registration cannot gain access to a claimed VTA. +- Existing-account binding requires that account's passkey and the claimant proof. It changes `portal_user_id` only; namespace, PVC, Vault paths, and VTA identity remain stable. +- Every user read/write/upgrade/export/log route uses Portal authorization, while teardown uses infrastructure ownership. A linked account sees its existing and claimed sessions; an unrelated account and the non-login resource owner cannot access them. +- Rehearsal covers shared Wi-Fi, API restarts, and multiple replicas, measuring request-to-reservation and confirm-to-redeem time and completing actual Keyring DIDComm use. + +For implementation, run `make test` (including `go vet`), test database races and migrations against isolated PostgreSQL, and format changed Go files with gofmt. Before rollout, verify the actual Farm DID, Keyring, and link host. External-readiness statements in older design documents are not current deployment evidence. + +## 11. Remaining Decisions and Integration Gates + +| Question | Proposed default | Implementation consequence | +| --- | --- | --- | +| Does mobile Keyring retain the original claim key and support a user-approved Portal-binding signature? | Require durable claim-key retention and an explicit binding ceremony | Must be verified with the actual mobile client; existing claim/redeem support alone is insufficient | +| Is email a profile/account hint with passkey login, or is email OTP login desired? | Keep current email-label/passkey authentication | Email OTP adds verification, delivery, recovery, and abuse handling beyond current behavior | +| First-event attendance, number of stations, acceptable activation wait, and simultaneous claims? | Pre-build enough inventory; staff retry on shortage | Determines resource ceilings, rate limits, and acceptance load | +| Domain/provider, image, resource profile, and operating ceiling? | Existing managed domain/platform stack and fixed profile | DID names are durable and kept VTAs consume ongoing resources | +| Who handles lost keys, mistaken confirmed claims, and activation failures? | Farm admin review with independent evidence and audit | Email or possession of a station QR cannot authorize takeover or reissue of a keyed VTA | + +## 12. Verification Boundary + +This change updates only this design document. It does not add APIs, migrations, pool resources, deployments, or version changes. The prototype, existing Farm signup/passkey and claim implementation, resource naming, user sequence/schema, teardown behavior, and related key-rotation code were inspected. Parked pre-provisioning, staffed confirmation, permanent per-slot resource ownership without migration, retention of successful claims, and optional later Portal management are confirmed requirements. Account-kind classification, batch admission, Portal authorization changes, and checkpointed cleanup are implementation requirements, not verified runtime behavior. Latency targets, actual mobile Keyring ownership-proof support, and current external-service readiness remain unverified. From cec50ba3ef0ad697a20fcb74aba5f02f8ac3b8c2 Mon Sep 17 00:00:00 2001 From: vthwang Date: Wed, 14 Oct 2026 23:49:20 -0700 Subject: [PATCH 2/4] feat: add pooled VTA inventory lifecycle Prebuild isolated VTA inventory without moving ownership or secrets at claim time. - Add minimal pool batch and session membership schema - Reuse mobile claims and park prepared deployments at zero replicas - Gate admission with the dashboard's live VTA-only capacity Signed-off-by: vthwang --- docs/pooled-vta-claim-design.md | 589 +++++++---------------- internal/connection/store.go | 22 +- internal/handler/dashboard.go | 141 ++++-- internal/handler/dashboard_usage_test.go | 20 + internal/handler/setup.go | 11 + internal/k8s/vta_resources.go | 11 +- internal/k8s/vta_resources_test.go | 18 + internal/model/pool.go | 31 ++ internal/pool/store.go | 285 +++++++++++ internal/pool/store_test.go | 132 +++++ internal/setup/orchestrator.go | 140 +++++- migrations/000043_vta_pool.down.sql | 7 + migrations/000043_vta_pool.up.sql | 17 + 13 files changed, 983 insertions(+), 441 deletions(-) create mode 100644 internal/model/pool.go create mode 100644 internal/pool/store.go create mode 100644 internal/pool/store_test.go create mode 100644 migrations/000043_vta_pool.down.sql create mode 100644 migrations/000043_vta_pool.up.sql diff --git a/docs/pooled-vta-claim-design.md b/docs/pooled-vta-claim-design.md index afadcb6..f2a7a96 100644 --- a/docs/pooled-vta-claim-design.md +++ b/docs/pooled-vta-claim-design.md @@ -1,430 +1,211 @@ -# Pooled VTA: Event Pre-provisioning and Claim Design +# Pooled VTA Creation and Claim Design -- Status: Draft for implementation. Event flow, retention, and permanent per-slot infrastructure ownership are confirmed; the Portal binding wire contract remains proposed below. -- Date: 2026-10-14. -- Goal: Prepare VTAs before an event so attendees can claim them through Keyring with minimal waiting and no mandatory registration. -- Reference: [Pooled VTA Claim Flow](https://claude.ai/artifact/5JGCqdwwWjT5Pc11ugp3rV), reviewed against this repository. The page references `~/Projects/one-step-vta.md`; that file was unavailable at the stated local path and has not been reviewed. -- Existing contracts: [Mobile Integration](mobile-connection-app-api.md) and [VTA Claim Design](vta-claim-implementation-design.md). New endpoints, tables, and policies below are proposals, not existing API behavior. +## 1. Product Model -## 1. Confirmed Product Requirements +An administrator creates a batch of pooled VTAs for an event. Each VTA is a +normal Farm resource from the beginning: -The prototype parks each `vta_only` slot at `vta_setup_complete`: its seed, PVC, configuration, DNS, and published DID are prepared, but no VTA Deployment exists. The attendee confirms the claim before the Farm imports the administrator DID and starts the VTA. +- it has its own real `users` row and database-generated user ID; +- it has its own `setup_sessions` row; +- its namespace and Vault paths are derived from that user and session; +- its DNS record, DID, PVC, configuration, Service, Ingress, and Deployment are + prepared before the event; +- its Deployment is parked at zero replicas, so it uses persistent storage but + no VTA pod CPU or memory; +- its setup-session status is `pool_waiting` while it is available. -The product owner confirmed this parked model. There is no idle VTA Deployment and therefore no per-slot running VTA CPU/memory request. PVCs and other persistent resources remain; build Jobs and shared infrastructure still consume compute. After confirmation, DID-host ACL setup, the import Job, Deployment startup, and readiness checks still take time. +At the station, an administrator selects an available session and displays the +ordinary VTA claim QR. Keyring scans it and shows the existing comparison code. +The administrator confirms the matching code. Confirmation writes the +claimant's admin DID through the existing mobile-connection flow, runs the +ordinary provision operation, scales the parked Deployment to one replica, and +finishes at `running`. -The primary event flow is a **staff-operated station**: an admin displays a claim QR on a computer, the attendee scans it with Keyring, and the admin compares the attendee's check with the station's check before confirming. A fresh QR is used for the next attendee. +There is no Portal activation link, account transfer, resource-owner transfer, +or new Keyring protocol in this feature. A claimed VTA already belongs to the +user row created for it. Future Portal enrollment is a separate feature. -The intended operating rule is one VTA per attendee, supervised by staff. A phone or wallet is not a unique-person credential: Keyring can create fresh keys, so this is not a cryptographic one-person limit. +## 2. Design Rule: Do Not Duplicate Existing State -After the event, admins can batch-clean unused inventory. Successfully claimed VTAs remain running and are excluded from that cleanup, even if their holders never enroll in the Portal. Holders can later prove ownership and attach their VTA to a Farm Portal account. Until then, Farm admins manage the deployment. +The pool schema records only batch membership. Existing tables remain the +authoritative source for everything else: -Each pre-built VTA gets its own internal resource owner immediately. Create its namespace, PVC, and Vault secrets under that owner from the first build; keep these locations unchanged through claim and later Portal attachment. Do not stage attendee resources under the platform user, migrate secrets at claim time, or turn the resource owner into a login account. The platform's mediator and DID-hosting services remain shared. - -## 2. Proposed Initial Scope - -- Support `vta_only` using a permanent Farm-managed domain and the existing platform stack's mediator and DID host. Give each slot its own owner, namespace, and seed. -- Let admins pre-build and inspect the pool, configure an event allocation and quota, open a staffed station, and disable further distribution. -- At the station, the admin selects Next attendee to reserve one slot. No email, Farm account, or passkey is required on the attendee's phone. -- Keyring claims the displayed QR. The station hides the QR after the first successful claim and displays the comparison check; the admin confirms or rejects it. -- Reuse existing claim/redeem Trust Tasks unchanged. Later Portal ownership verification is a separate, explicitly scoped ceremony requiring Keyring integration. -- Initially allocate ready slots only and show retryable unavailability when empty. A self-service event-link page and FIFO waiting queue are deferred; neither is necessary for the staffed-station flow. -- Retain handed-over VTAs after the event and provide both new-account enrollment and existing-account linking for Portal management. - -Full-stack pools, custom domains, post-claim DID-hostname changes, and unbounded automatic expansion are outside the proposed initial scope. This document does not implement the feature. - -## 3. Existing Implementation and Gaps - -| Source | Reusable behavior | Required pool work | -| --- | --- | --- | -| [`internal/setup/orchestrator.go`](../internal/setup/orchestrator.go) | VTA-only setup, DID publication, administrator import, Deployment readiness | Explicit pool readiness, build recovery, bounded batch concurrency | -| [`internal/connection/store.go`](../internal/connection/store.go) | Five-minute QR and confirmation windows, claim checks, session locks, claim/confirm/reject transitions | Atomic coordination with event quotas, slot ownership, and reclamation | -| [`internal/handler/mobile_connection.go`](../internal/handler/mobile_connection.go) | Trigger links and browser state views | Existing authorization requires the signed-in owner's `user_id`; station operations require admin authority instead | -| [`internal/handler/trust_task.go`](../internal/handler/trust_task.go), [`internal/trusttask`](../internal/trusttask) | Wallet proof validation, signed replies, replay handling, redeem long polling | Pool transaction integration, failure handling, and handover protection | -| [`internal/setup/initial_provision.go`](../internal/setup/initial_provision.go) | Persistent provisioning state, cross-replica advisory locks, cancellation during deletion | Recover confirmed claims before provisioning is queued; persist pool build/reaper work independently of process memory | -| [`internal/handler/capacity_service.go`](../internal/handler/capacity_service.go) | Resource estimates | Account for parked stock, concurrent activation, and retained VTAs; estimates do not reserve capacity atomically | -| [Provisioning Load Test](vta-provisioning-load-test.md) | Batch and cleanup patterns | Load tests share a system owner and use an administrator key that cannot be handed over; their sessions are not attendee inventory | -| [`internal/model/user.go`](../internal/model/user.go), [`internal/handler/user.go`](../internal/handler/user.go) | Auto-increment user IDs and an admin `system` flag | Add explicit account classification; current system detection only recognizes platform/load-test names; separate human counts from resource-owner counts | -| [`internal/handler/setup.go`](../internal/handler/setup.go), [`migrations/000003_create_setup_sessions.up.sql`](../migrations/000003_create_setup_sessions.up.sql) | Session teardown and user FK | External teardown is partly best-effort; deleting a user cascades session rows without cleaning external resources; owner removal must be guarded and last | -| [`internal/handler/signup.go`](../internal/handler/signup.go), [`internal/handler/passkey.go`](../internal/handler/passkey.go) | Passkey login/enrollment and user accounts | Email is unverified and signup resumes accounts without passkeys; pooled ownership must never be exposed through that email-only resume path | - -Two implementation gaps directly affect the prototype: - -1. **`vta_setup_complete` does not prove claim readiness.** The orchestrator writes this status before publishing the DID. Its comments identify a crash window that can leave the DID unpublished. Set `pool_slots.ready` only after publication and DID-resolution checks succeed, with recoverable publication or quarantine after interruption. On a publication conflict, verify the existing document and ownership; neither overwrite blindly nor treat every conflict as success. -2. **The current claim flow has no first-DIDComm-connection evidence.** An older migration contains `connected_at`, but the current `MobileConnection` model and claim/redeem flow do not maintain it. `released` means provisioning completed, not that the wallet received the response or connected over DIDComm. The prototype's cleanup rule cannot simply query `connected_at`. - -The prototype also includes `didLogUrl` in its success response. Current `redeemOutcome` returns the handle, VTA DID, operation, and optional mediator DID. Follow the generated protocol schema; do not add response fields without confirming protocol and Keyring support. - -## 4. QR Codes, Station Flow, and Competing Scans - -```text -Admin pre-builds -> slot ready -> admin opens event station - -> Next attendee -> reserve one slot -> display a fresh trigger QR - -> attendee's Keyring claims -> admin compares and confirms - -> administrator import and VTA startup -> redeem succeeds -> Keyring uses VTA - -> admin selects Next attendee -> different slot and fresh QR -``` - -### What the Prototype Says, and What Changes - -| QR or link | Reuse | Meaning | -| --- | --- | --- | -| Prototype event link | Shared across attendees | Opens a Farm browser page, which creates a separate reservation and trigger link for each attendee | -| VTA trigger QR | One reservation and one winning claimant | Contains a fresh `_id` handle and expiry; this is what Keyring opens | -| Proposed station page | Same computer/page across attendees | Admin-controlled UI; each Next attendee action produces a new reservation and trigger QR | - -The prototype does not give everyone the same reusable VTA-claim QR. The requested station replaces its attendee-browser confirmation with staff confirmation while retaining the per-reservation trigger link. A static event-link poster is not the station's claim QR, and the existing VTA-claim flow must never be reused to add successive attendees to an already-running VTA through `grant_acl`. - -Two phones can scan a visible QR, but the atomic claim transition admits only the first valid claimant. The other receives `alreadyClaimed`. The first scan has only locked an empty slot: it has not installed a key or received the VTA. Staff compare the check on the intended attendee's phone with the station. On mismatch or an unexpected claimant, staff reject and issue a new handle. Never confirm because someone merely scanned successfully. - -Once confirmed, that slot is permanently removed from allocation. A later attendee gets another slot and handle. Reject/expiry before confirmation may safely reuse the untouched slot, but never the old handle. QR photographs can cause a temporary reservation race, not a second successful handover; the comparison and staff confirmation are essential. - -### Station Authorization and Continuity - -All station mutations require the existing admin login, event/station scope, Origin validation, and CSRF protection. The event ID, reservation ID, QR handle, wallet proof, and attendee email cannot authorize station confirmation. The attendee-facing display contains only the QR, expiry, progress, and comparison code; it never exposes admin credentials or creates a Portal login on the shared computer. - -Persist the station's current reservation and display generation. Refreshing resumes it; concurrent Next attendee clicks are idempotent. Confirm/reject requests name the reservation and expected generation, preventing a stale tab from confirming the next attendee. Allow at most one unconfirmed reservation per station. After confirmation commits, staff can move to the next attendee while the previous activation continues; previous work remains visible in event history and Keyring continues redeem polling. Do not rotate a claimed QR automatically during comparison. - -Multiple stations allocate from the same pool transactionally and never share one active handle. An expired admin session requires staff reauthentication; it does not transfer confirmation authority to the attendee or discard a confirmed activation. Leaving the station must lock its controls. - -Keep trigger links, cookies, and claim payloads out of access logs, panic dumps, analytics, error tracking, and Referer headers. Set `Cache-Control: no-store` and `Referrer-Policy: no-referrer`, omit third-party tracking, and check CDN/Ingress and outer API loggers. Store new browser onboarding secrets as hashes; existing claim-handle storage remains governed by the current contract and restricted server access. - -Staff enforce the expected one-person-one-VTA practice. No device fingerprinting or phone identifier is proposed. A stricter guarantee would require individual admission tickets or another agreed eligibility mechanism. - -## 5. Data Model and State - -These are logical entities. Implementation should use additive migrations and existing relationships without reserving migration numbers in this draft. - -| Entity | Proposed fields and invariants | +| Concern | Source of truth | | --- | --- | -| Pool configuration | Target ready count, total inventory/retention ceilings, build and activation concurrency, pinned image, provider snapshot, pause flag | -| `users.account_kind` | Proposed non-null classification: `person`, `platform`, `load_test`, or `pool_owner`; internal owners cannot authenticate or acquire personal credentials | -| `pool_build_batches` | Requested count, creating admin, optional event allocation, frozen profile, request digest/idempotency key, admitted capacity, progress counts; each accepted member has a stable ordinal | -| `pool_slots` | Unique live `setup_session_id` and `resource_owner_user_id`, unique `(batch_id, member_ordinal)`, optional allocated event, state, generation, readiness/handover timestamps, failure reason, worker lease and cleanup progress | -| `pool_events` | ID, label, maximum uses, held/committed counters, expiry/disabled timestamps, creating admin; public event tokens are unnecessary for staffed distribution | -| `pool_stations` | Event, creating/operator admin, current reservation, display generation, enabled state; at most one unconfirmed reservation | -| `pool_reservations` | ID, event/slot/station/mobile-connection IDs, state, deadlines, confirming admin, quota-settlement marker, idempotency-key digest | -| `setup_sessions.portal_user_id` | Nullable FK to the real Portal user; immutable infrastructure ownership remains in `user_id` | -| Pool ownership entitlement | Immutable session/reservation/VTA DID/claimant DID relationship, handover timestamp, revoked/linked state; survives short-lived QR/replay cleanup | -| Portal binding attempts | Entitlement, target account or provisional enrollment ID, browser-secret hash, nonce, purpose, expiry, verified/consumed state | -| Audit records | Actor, event, slot/reservation/session IDs, timestamps, transitions; no bearer credentials or private keys | -| Cleanup manifest/tombstone | Immutable original owner/session IDs and exact external resource identities, per-step results, deletion actor/reason, retry state; survives removal of live rows without cascading away | - -### Infrastructure Owner Versus Portal Owner - -Create one internal non-login `users` row per slot, explicitly classified as a pool resource owner, with a unique generated ID, no email, and no passkey. Use its numeric ID in `setup_sessions.user_id`; retain it for that deployment's lifetime. Namespace, ServiceAccount, Vault policy, and seed paths continue to use this value. Never use the admin's ID, one shared platform user, the wallet DID string, or a guessed email as this owner. - -Store the attendee's real account in a separate nullable `setup_sessions.portal_user_id`. This permits an existing account to manage its newly claimed VTA without moving resources or changing namespace identity. For example: - -```text -Pre-build: session.user_id = 4102; portal_user_id = NULL -Claim + running: session.user_id = 4102; portal_user_id = NULL -Portal binding: session.user_id = 4102; portal_user_id = 87 +| Account and numeric owner | `users` and `setup_sessions.user_id` | +| VTA name, image, DID, DNS record ID, domain and error | `setup_sessions` | +| Namespace | Derived by the Kubernetes client from `setup_sessions.user_id` | +| Vault path | Derived from `user_id` and `setup_session.id` | +| Cloudflare zone and load-balancer IP | Current application configuration | +| QR, scan, claimant DID and confirmation time | `mobile_connections` | +| Creation, waiting, activation, failure and running state | `setup_sessions.status` | +| Pool grouping | `pool_batches` and `pool_sessions` | + +Consequently, the pool does not store a second user ID, VTA name, namespace, +Vault target, DNS zone, ingress IP, claimant DID, claim timestamps, cleanup +cursor, or lifecycle state. + +## 3. Schema + +```sql +CREATE TABLE pool_batches ( + id bigserial PRIMARY KEY, + created_by bigint NOT NULL REFERENCES admins(id) ON DELETE RESTRICT, + label text NOT NULL, + requested_count integer NOT NULL CHECK (requested_count > 0), + idempotency_key text NOT NULL, + request_digest text NOT NULL, + created_at timestamptz NOT NULL DEFAULT now(), + UNIQUE (created_by, idempotency_key) +); + +CREATE TABLE pool_sessions ( + session_id bigint PRIMARY KEY REFERENCES setup_sessions(id) ON DELETE RESTRICT, + batch_id bigint NOT NULL REFERENCES pool_batches(id) ON DELETE RESTRICT +); + +CREATE INDEX pool_sessions_batch_id ON pool_sessions(batch_id); ``` -Before binding, only Farm admins can manage this session through Portal APIs; the attendee already controls the VTA through Keyring. After binding, user 87 can manage it, while Farm admins retain their existing operator access. Admin UI should distinguish infrastructure owner, claimant DID, and Portal account. Portal enrollment is optional for keeping the VTA. - -This separation requires an authorization migration, not just a new column. Backfill `portal_user_id = user_id` for existing ordinary user sessions, leave platform/load-test/pool system sessions unbound, and set both IDs for newly created ordinary sessions. User-facing authorization must use `portal_user_id` exclusively after cutover; no fallback for NULL pooled owners. Update list/detail/delete, configuration/export/resources/logs, administrator grants, mobile connections, upgrade creation/status/cancellation, and user quota/profile queries. Keep namespace selection, Vault access, last-session cleanup, and resource-owner deletion keyed by the original `session.UserID`. Audit direct `user_id` queries as well as shared helpers. - -Pool resource owners are never login principals and must be excluded from signup, passkey enrollment, and account-recovery issuance. Portal-account deletion cannot cascade-delete or silently return a deployment to the pool. Retain bound deployments for explicit operator disposition. Initial binding is unique and atomic; already-bound sessions cannot be reassigned by repeating a wallet proof. - -Attaching an already-provisioned VTA must not be blocked by the normal two-VTA creation limit: binding allocates no new infrastructure. Count it toward the account's subsequent creation limit without granting `fullstack_access`; event quota continues to govern event issuance. - -### Permanent Resource Placement - -The following example uses resource owner `4102`, setup session `730`, and eventual Portal user `87`. The IDs are illustrative, not values to reserve or configure. - -| Resource or identity | Pre-build placement | After claim and Portal binding | -| --- | --- | --- | -| Infrastructure owner | `users.id = 4102`, `account_kind = pool_owner` | Unchanged; still a non-login owner | -| Session ownership | `user_id = 4102`, `portal_user_id = NULL` | `user_id = 4102`, `portal_user_id = 87` | -| Kubernetes namespace | `-4102`, normally `vtafarm-user-4102` | Unchanged | -| Workload identity | ServiceAccount `vta` in that namespace | Same namespace-scoped identity | -| Persistent data | PVC `vta-data-730` in that namespace | Same PVC and data | -| Vault policy/auth role | `vta-user-4102`, bound to that namespace and ServiceAccount | Unchanged | -| Vault KV prefix | `/vta/user-4102/session-730/` | Unchanged; never moved under `user-87` | -| Master seed | `master-seed` beneath that prefix | Same seed; no regeneration or copying during claim | -| VTA hostname and DID | Created once from the session name/provider snapshot | Unchanged | - -Vault paths above describe logical KV locations, not raw KV v2 HTTP API routes. Reuse `UserNamespace`, `VtaPVCName`, `UserName`, `VtaPrefix`, and `SeedPath`; do not introduce a second naming algorithm for pooled sessions. A dedicated owner means the existing per-user Vault policy cannot overlap another attendee's or the platform user's prefixes. Reusing the shared mediator/DID host does not grant access to their Vault keys. Namespace separation does not, by itself, assert network isolation or a separate Vault server per attendee. - -Build sequence after database admission: - -1. Load the committed slot, owner, session, and frozen provider/profile. Record expected external resource identities in the cleanup manifest before issuing side effects. -2. Ensure the owner's namespace, ServiceAccounts, and RBAC. On an existing resource, verify ownership before reuse; `AlreadyExists` alone is not ownership evidence. Record Kubernetes UIDs where available. -3. Ensure the owner's Vault policy and Kubernetes-auth role. Render setup configuration with that exact owner's seed path and role. -4. Provision session DNS and the PVC/configuration/setup Job in that namespace, then publish and verify the DID through the snapshotted shared provider. Never fall back to the platform namespace or seed path on failure. -5. Mark the slot ready only after all readiness checks complete and setup work no longer consumes running compute. Keep the PVC and required configuration; no VTA Deployment runs while parked. -6. On staff confirmation, run administrator import and start the Deployment against the same PVC and owner. Portal binding later updates management authorization only. - -Persist the cluster/environment identity, namespace name/UID, Vault KV/auth mounts and logical prefix/role, DNS record IDs, and DID-host provider identifiers needed for recovery. These are protected operational metadata, not user-visible secrets. A later configuration change must not redirect cleanup to a different cluster, mount, or namespace prefix; fail closed if the original target cannot be resolved. Do not store seed values or access credentials in the manifest. - -### User ID Allocation and a 50-Slot Batch - -`users.id` is currently a PostgreSQL `BIGSERIAL` primary key. Personal accounts, the platform account, and pool owners use the same database-generated sequence. Creating an event alone allocates no owners; accepting its explicit 50-slot build batch creates 50 real owner rows, not an application-side reservation of an ID range. - -For illustration, if the last allocated value is 20, no other inserts occur, and no values are skipped, the 50 owners receive 21 through 70; a later ordinary signup receives 71. Twenty existing users does not necessarily mean the sequence is at 20. With concurrent signup or another batch, IDs may interleave; transaction rollback or conflicts can leave gaps. Never promise consecutive IDs or derive batch membership from an ID interval. PostgreSQL allocates sequence values atomically across concurrent callers and does not generally reclaim aborted values. See [PostgreSQL sequence semantics](https://www.postgresql.org/docs/18/functions-sequence.html). - -Implementation rules: - -- Omit `id` on inserts and use the generated value returned by the database/ORM. Do not calculate `MAX(id) + 1`, manually insert a reserved range, reset the sequence, decrement it after cleanup, or use sequence values as an attendee count. -- Keep the numeric ID immutable. Deleted owner IDs are never intentionally reused for new inventory. An actual sequence misconfiguration after an import/restore requires a separate maintenance procedure, not a fallback in signup or pool creation. -- Generate `users.unique_id` independently within its existing 12-character column limit, enforce its unique index, and retry collisions. Account kind is stored explicitly, not encoded in a long prefix or inferred from an ID range. -- Persist exact owner/session/slot associations. Use a batch member ordinal for progress such as `17 of 50`; gaps in primary keys must not affect ordering, cleanup selection, or metrics. -- Allocate IDs only as part of committed ownership records before creating external resources. A rolled-back insert may consume a number, but it must not leave a namespace or Vault path based on an uncommitted owner. - -### Atomic Admission, Build Retries, and Account Classification - -For a bounded 50-slot request, resolve an existing idempotency outcome before attempting new admission. For a new request, validate the profile/event and reserve capacity under a shared database lock within the admission transaction. Persist the idempotency record, batch, 50 owner rows, 50 initial setup-session rows, 50 queued slots, and durable build-work records together. Commit before responding `202` and before workers make external calls. If existing setup creation requires DNS before its row insert, split database admission from external provisioning for this path; do not run the public signup or ordinary create handler 50 times. Existing startup recovery must not bypass the pool scheduler or activate queued/parked sessions; integrate both paths under the same work ownership and deletion fencing. - -The `202` guarantee is **50 durable members admitted**, not 50 ready VTAs. Cap the request count so admission stays bounded; larger inventories use multiple explicit batches. A database failure commits no partial batch, although unused sequence values may remain. The same actor/idempotency key and payload return the same batch; a changed payload conflicts. An HTTP timeout after commit must not create another 50 owners on retry. - -Workers update each existing member independently. A temporary build failure retries the same owner/session and inspects existing resources before continuing; it does not create another user. If the original identity cannot be safely recovered, quarantine and explicitly retire it before creating a separately tracked replacement with new owner/session/name/seed. Report failed and retired members separately from ready inventory. Periodically reconcile queued, failed, and retiring records; an empty namespace or old timestamp alone is not permission to delete an owner. - -Add `account_kind` using an additive, reviewed backfill: identify the platform owner and legacy load-test owners from existing provenance, classify existing ordinary accounts as `person`, and create future pooled owners as `pool_owner`. Do not classify by `email IS NULL`, because legitimate personal accounts can have no email. Preserve the existing admin `system` response as a derived compatibility field and add kind/filter support without silently changing the list's default semantics. Show human-account and internal-owner totals separately; derive personal quotas from Portal ownership, not total `users` rows. +### `pool_batches` -Enforce non-login status server-side across login/session issuance, passkey registration, invitation/account adoption, profile changes, recovery-link creation/consumption, and administrative personal-account controls. A pool owner always has no email, no personal passkeys, and no fullstack-access grant. Use database checks where possible and service-level checks for cross-table invariants. Do not upgrade it into a person at claim or Portal enrollment. The real Portal account gets a separate row or is an existing authenticated person account. +`id` identifies one batch creation request. It is not a user ID range and does +not reserve numeric user IDs. -Each pooled owner has exactly one lifetime slot association and at most one live setup session. Enforce unique live owner/slot and owner/session associations with constraints and a transactional cross-table guard; prevent ordinary creation paths from attaching another session or custom domain to that owner. Archive original numeric IDs in tombstones; live owner/session FKs may be cleared only in final retirement. Use restrictive FKs/guards so deleting an owner cannot cascade away a live slot or cleanup manifest. Check all references, not only a zero session count. +`created_by` records the administrator that created the batch and scopes the +idempotency key. The foreign key is restrictive so audit attribution cannot be +silently erased. -### Stable DNS Naming (Proposed Default) +`label` is the administrator-facing event or batch name. It has no effect on +resource identity. -Use the existing `[dev-]vta-.` shape with a generated lowercase random name, for example `vta-k7m2x9p4q8.`. Reserve the session name transactionally with collision retries and retain retired names in a registry checked by every name-allocation path, including ordinary setup. Keep the existing `-vta` DID path suffix. Do not include an email, wallet DID, numeric owner ID, event label, or temporary `pool` marker. A future display name may change without renaming DNS, the session route identifier, or the DID. The exact generated-name format remains an implementation default, not an already-shipped behavior. +`requested_count` records the original request. The live number of members is +always counted from `pool_sessions`; it can become smaller after cleanup. +The database constraint validates only that the request is positive. The +cluster's live VTA-only estimate is the actual upper bound. -### State Responsibilities - -```text -slot: queued -> building -> ready -> reserved -> claimed -> activating -> handed_over - -reserved/claimed -> ready - only after rejection, cancellation, or expiry, before confirmation/key import - -uncertain state or possible partial key import -> quarantined -> retiring -> retired -``` - -Preserve existing `mobile_connections` states: `open -> claimed -> confirmed -> provisioning -> released`, plus rejected/lapsed and existing terminal states. `handed_over` is a pool allocation state, not a new Trust Task state. `released` means available for handover, never available for reuse in the pool. - -- The five-minute QR clock starts when a ready slot is assigned and a handle is created. Claim starts a separate five-minute confirmation window, using existing constants and database time. -- Before confirmation, reject/expiry/cancel atomically terminates the reservation and settles quota. Return the slot only after verifying that no administrator key was installed. Every subsequent reservation gets a new handle; old handles never revive. -- Confirm must serialize with reject, expiry, and cleanup. Once confirmation commits, the slot cannot be reassigned, even before the import Job starts. -- QR and comparison deadlines no longer cancel confirmed work. An HTTP timeout is not provisioning failure and must not instruct the wallet to delete its key. -- Retry with the same reservation, slot generation, and claimant. Inspect Job/ACL/Deployment state after interruption or partial success. Quarantine or rebuild a slot if key-import state is uncertain; do not return it directly to ready. - -### Quota and Idempotency - -Proposed accounting: `held + committed <= max_uses`. Allocation increments held; confirmation transfers held to committed. An unconfirmed reject/expiry refunds once. Compensating a confirmed failure requires evidence that it cannot leave a usable or still-starting VTA, plus an auditable settlement operation. - -Allocate the slot, create the reservation/mobile connection, reserve event quota, and save the idempotency outcome in one database transaction. Define one lock order compatible with the existing session-first claim/confirm operations; adding event locks in inconsistent order can deadlock. Use unique constraints to prevent multiple active reservations for a slot. Commit before waking workers. - -Retries for the same station generation return the existing reservation. Scope `Idempotency-Key` to actor, event, station generation, and operation; return 409 when the same key carries a different payload. Lost responses must be recoverable. Repeated confirmation of the same already-confirmed claim returns current progress without restarting the Job or incorrectly reporting expiration. - -## 6. Claiming Portal Management After the Event - -### Recommended Product Flow - -Use **wallet ownership verification followed by account enrollment**, not email-only ownership lookup and not a mandatory new general-purpose wallet-login system. - -For a new Portal user: - -1. On their own phone, open Farm Portal from the claimed VTA in Keyring, carrying only its public VTA DID as a hint. On a computer, select Connect my claimed VTA and enter the VTA DID shown in Keyring. The DID is not a credential. The Farm creates a short-lived onboarding attempt bound to that browser, the selected VTA, and a provisional new-account ID. It displays a separate Portal-binding QR or app link, clearly labeled as Portal access, not another VTA claim. -2. Keyring selects the previously claimed VTA, verifies the Farm request, displays the operation and browser comparison check, and asks the holder to approve. It signs the fresh ownership challenge with the retained claimant key. The Farm matches that proof against the successful event entitlement. -3. Only that browser receives a limited onboarding session for the verified VTA. It can collect an email and register a passkey, but cannot list other users' resources, obtain a general user JWT, or modify the VTA yet. -4. The attendee enters an unused email and creates a passkey. Atomically save the new account's email/passkey, set `portal_user_id`, mark the entitlement linked, and consume the onboarding attempt. Only then issue the ordinary Farm user session and show the VTA in My VTAs. -5. Later login uses the passkey. Email may be an account hint/profile label, but typing it alone cannot authenticate or find claimable VTAs. - -For an existing Portal user, first authenticate with their passkey, then select Connect my claimed VTA and approve the wallet challenge. Bind the challenge to that exact account and show its identifying label in Keyring. Require a fresh passkey assertion for the binding, not merely a cookie issued by the existing public signup flow. After approval and browser confirmation, atomically attach the VTA to the authenticated account without changing the infrastructure owner. - -If the attendee never visits the Portal, keep `portal_user_id = NULL`. Their VTA keeps running, Keyring keeps working, and Farm admins retain management access. Portal enrollment is neither another allocation nor a condition for retention. - -### What the Proof Establishes - -At successful event handover, persist an entitlement containing the session ID, VTA DID, original claim reservation, claimant Ed25519 `did:key`, and confirming admin. Commit the entitlement, pool handover timestamp, and mobile release transition consistently in one transaction after readiness succeeds. A recovery worker must reconcile this even if both browsers disconnect; do not depend on a later status GET. Private keys remain in Keyring. This record is longer-lived than QR expiry and replay caches. - -The Portal challenge proves control of the private key recorded for that entitlement. Knowing a DID string, VTA DID, email, six-character check, or old trigger handle is insufficient. A saved claim/redeem signature cannot be replayed as a Portal-binding approval: purpose, audience, challenge, and browser binding are different. - -Only a completed handover entitlement may bind. Pending/failed/rejected claims cannot. Binding is single-assignment: the same account's retry returns its existing result; a different target receives a conflict. A later transfer or recovery is a separate audited action, not another claim. - -### Key Rotation and Long-Term Authority - -Do not assume the initial administrator DID remains the runtime administrator forever. Existing Farm code documents PNM rotation on first connection, and client onboarding supports replacing the ephemeral ACL identity. A cached ACL snapshot is not current ownership evidence, and refreshing that snapshot currently stops the VTA. - -For the initial Portal-binding design, require Keyring to retain the event claimant private key **as an unbound Farm ownership credential** until Portal attachment succeeds, even if it rotates its VTA runtime ACL key. This extends the existing mobile guide's key-retention requirement with an explicit purpose. Keyring must store the Farm DID, VTA DID, and original claimant relationship durably and sign only a Farm-verified, user-approved binding challenge. Verify this behavior in the actual mobile Keyring; browser-extension/PNM behavior is not evidence of mobile support. - -The Farm entitlement deliberately represents the right to attach the original event allocation, not current runtime ACL membership. Granting another VTA administrator does not transfer Farm hosting ownership. A runtime key swap does not silently revoke or transfer this entitlement. Provide an admin revocation/ownership-recovery procedure for a lost or compromised claim credential; key loss is never repaired by accepting an email assertion. - -Once Portal binding completes, that key cannot bind a second account, sign into the existing account, replace passkeys, or recover it. The Portal's passkey/recovery policy becomes authoritative for hosting management. Do not delete a key from Keyring merely because Portal binding succeeded if the key is still used for VTA administration or redeem recovery. - -If the actual Keyring discards the original key during onboarding, automated Portal binding is blocked until it either retains an appropriate Farm binding credential or implements a verified successor-key registration. A successor registration must authenticate the old entitlement key and prove possession of the new key, bind the Farm/VTA/purpose, and revoke the old binding credential atomically. A wallet-supplied new DID or an arbitrary current ACL administrator is not a substitute. This is a Keyring integration requirement, not a claim that it is already implemented. - -### Binding Ceremony Contract - -Keep `provision/vta/claim/0.1` and `provision/vta/redeem/0.1` unchanged. Define the Portal-binding ceremony separately before implementation. Its proposed semantic contract is: - -- Server-generated random nonce with at least 128 bits of entropy, five-minute expiry, single-use consumption, and server-side state. -- Explicit version, purpose (`enroll_new_portal_owner` or `bind_existing_portal_owner`), Farm DID/audience, exact Portal origin, attempt ID, browser-binding ID, VTA DID, claimant DID, target account/provisional ID, and deadline. Existing-account display identity is frozen for the attempt; changing the target requires a new challenge. -- A Farm-signed request verified by Keyring, and an Ed25519 proof over the complete canonical challenge under the recorded claimant key. Freeze the canonicalization, proof fields, and closed schema in the implementation contract; reuse existing cryptographic helpers only where their validation semantics match. -- Explicit wallet approval and a browser comparison check. The signing UI states that this grants Farm hosting management, including configuration/deletion rights, and does not grant staff access to the attendee's wallet. -- A random HttpOnly browser capability, hashed at rest and independent of the wallet-visible QR. Only its originating browser can poll private results and finish enrollment/linking. A wallet response never receives or forwards a Portal JWT to the station or another browser. -- A separate short-lived request-read handle in the QR/app link lets Keyring fetch the signed challenge without the browser cookie. It exposes only the request needed for informed approval, cannot finish enrollment, and is also redacted from logs. Polling/completion still require the private browser capability. -- Verification and final linking each recheck entitlement state, expiry, target account, and deletion/revocation. Linking and challenge consumption share one transaction. Replayed proofs, changed targets, and stale attempts cannot transfer ownership. - -This is a proposed Farm HTTPS ceremony, not a registered Trust Task or an assertion that Keyring already recognizes a new trigger-link type. Its app-link dispatch and signed schema are implementation dependencies. Do not put a Portal-binding request into a VTA-claim QR and expect existing handlers to interpret it. - -The existing [auth/oob/identify specification](https://github.com/trustoverip/dtgwg-trust-tasks-tf/blob/main/specs/auth/oob/identify/0.1/spec.md) verifies an already recognized member and participates in a larger approval flow; it is not, by itself, proof that the member owns this Farm allocation. Therefore generic VTI sign-in is not assumed to solve initial Portal attachment. It may be integrated later once its subject is securely mapped to Farm accounts and entitlements. - -### Email, Passkey, and Interrupted Enrollment - -The current Farm sends no verification email. Email is a self-declared label; passkeys authenticate accounts. Preserve that meaning in this feature. Email OTP/magic-link login would be a separate authentication and delivery-system change, not implied by this design. - -There is a concrete integration hazard: `SignupHandler.resumeOrReject` issues a full user session for an email whose account has no passkeys. Never write a pooled owner's email or attach their VTA to such an account before credential setup. Otherwise another person could enter the same email and resume an account that already controls the VTA. - -Keep new onboarding accounts isolated with no public email mapping or ordinary login session until successful WebAuthn completion. Use a purpose-limited registration context rather than the generic signup-issued JWT. Commit email uniqueness, passkey persistence, account activation, ownership binding, and attempt consumption together. If registration or the final transaction fails, leave the VTA unbound and recoverable through a fresh wallet challenge; it must not become reclaimable event inventory. - -If the requested email already exists, require authentication to that existing account and a fresh challenge targeting it, or choose another unused email. Never merge based on the email. An existing email-only account that cannot provide a passkey requires the ordinary recovery/onboarding resolution without attaching the VTA; do not trust its old signup cookies. Concurrent enrollment for the same email must have one database winner and no partial binding. Expired pending accounts can be cleaned independently of VTA resources. - -## 7. Proposed API - -The primary frontend is an admin event/station page. REST paths below are under `/api/v1`. The existing wallet endpoint remains `POST /trust-task/trust-tasks`. Public event-link/self-service allocation routes are deferred. - -| Method/path under `/api/v1` | Authority | Behavior | -| --- | --- | --- | -| `GET /admin/pool` | Admin | Inventory, active work, quarantine, retained count, budget, dependency state | -| `POST /admin/pool/slots` | Admin | `{count,event_id?}` with the configured pool profile and required idempotency key; 202 only after all requested owner/session/slot rows and work records commit; batch ID, admitted count, and status URL | -| `GET /admin/pool/batches/:id` | Admin | Requested/admitted/queued/building/ready/failed/retired counts and paginated member details; exact member identities, not an owner-ID range | -| `GET /admin/pool/slots` | Admin | Paginated inventory with state/batch/event filters; no bearer credentials | -| `POST /admin/pool/events` | Admin | `{label,max_uses,expires_at}`; 201; create a bounded distribution event | -| `GET /admin/pool/events` | Admin | Paginated events, quota, expiry, and disabled state | -| `PUT /admin/pool/events/:id/status` | Admin | `{enabled:false}` stops new reservations; does not delete resources or cancel confirmed delivery | -| `POST /admin/pool/events/:id/stations` | Admin | Create an event station; return station ID and page URL | -| `GET /admin/pool/stations/:id` | Admin with station scope | Current reservation/generation, QR while open, comparison check after claim, and progress | -| `POST /admin/pool/stations/:id/reservations` | Admin with station scope and CSRF protection | Next attendee; expected generation and idempotency key; 201 new or 200 resumed reservation | -| `POST /admin/pool/reservations/:id/confirm` | Admin with station scope and CSRF protection | Confirm the frozen reservation/generation; 202 activation; do not accept a replacement admin DID | -| `POST /admin/pool/reservations/:id/reject` | Admin with station scope and CSRF protection | 200 rejection before confirmation; next issuance gets a fresh handle | -| `DELETE /admin/pool/reservations/:id` | Admin with station scope and CSRF protection | 204 cancellation before claim; 409 for later states | -| `GET /admin/pool/events/:id/cleanup-preview` | Admin | Eligible/protected resources and a short-lived versioned plan ID | -| `POST /admin/pool/events/:id/cleanup` | Admin | Submit plan ID; recheck eligibility; 409 for stale plans, otherwise 202 with cleanup-job URL | -| `GET /admin/pool/cleanup-jobs/:id` | Admin | Progress, item failures, and retry status | -| `GET /admin/users` (extension) | Admin | Preserve the existing list contract; expose account kind and optional kind filtering, with internal owners marked `system: true` | -| `POST /portal-bindings` | Own browser capability/origin; recent passkey proof for an existing target account | Create a purpose-bound ownership challenge; no general user login | -| `GET /portal-bindings/:id` | Originating browser capability | Challenge, verified state, expiry, and allowed next step; no public ownership lookup | -| `GET /portal-binding-requests/:handle` | Short-lived request-read handle | Return the Farm-signed request for Keyring approval; no browser session or completion authority | -| `POST /portal-bindings/:id/proof` | Fresh claimant signature | Verify the separate ownership ceremony; never return a browser login token to the wallet | -| `POST /portal-bindings/:id/passkeys/begin` | Verified limited onboarding context | Start new-account WebAuthn registration | -| `POST /portal-bindings/:id/complete` | Same browser; valid WebAuthn finish or fresh existing-account authentication | Atomically consume proof, finalize account if new, and attach the VTA; only then issue the browser session | - -GET must not provision resources or consume quota. Proposed HTTP errors are 400 for invalid fields, 401 for missing authentication, 403 for insufficient authority or CSRF/origin violations, 404 for unknown or non-owned resources, 409 for stale station generations, `event_exhausted`, `already_bound`, or target/email conflicts, 410 for expired/disabled events or expired attempts, 429 for rate limiting, and 503 for `pool_empty`/`at_capacity`/`dns_budget_exhausted`/`shared_infra_unavailable`. State GET may return 200 with temporary unavailability; allocation POST returns 503. Failed ownership proofs must not enumerate which email owns a VTA. - -Browser REST errors follow the repository's `error`/`reason` convention: - -```json -{"error":"No VTA is ready yet.","reason":"pool_empty","retry_after_seconds":10} -``` +`idempotency_key` makes a retried HTTP request return the original batch instead +of creating another 50 users and VTAs. -Include `Retry-After` for 429 and retryable 503 responses. Station views may include `reservation_id`, `generation`, `status`, `server_time`, `expires_at`, `confirm_by`, and `next_poll_after_seconds`. Only authorized station admins receive its trigger link for display. Use RFC 3339 timestamps in REST; preserve Unix seconds for VTI `_exp` and the existing Trust Task `confirmBy` field. +`request_digest` binds the idempotency key to the original request body. Reusing +one key with a different count or configuration is rejected instead of silently +returning an unrelated batch. -Trust Tasks retain their signed response/error schemas, including HTTP 422 for redeem pending; do not apply browser REST errors to that endpoint. Each new redeem poll uses a new document ID, while retransmitting the same document uses replay handling. The original claimant must be able to redeem again after a lost success response; define a recovery-retention window before pruning handover/replay records. +`created_at` is the immutable creation timestamp. The batch has no mutable +state, so it needs no `updated_at`. -When implementing routes, update `internal/apidocs/openapi.yaml` and the mobile integration guide together. Do not publish these proposed endpoints in OpenAPI before implementation. Add required idempotency/CSRF headers to the existing CORS allowlist and restrict origins to the deployed frontend. +### `pool_sessions` -## 8. Workers, Capacity, and Event Traffic +`session_id` is both the primary key and a restrictive foreign key. A setup +session can belong to at most one pool batch. Restricting direct deletion forces +cleanup to remove external resources and pool membership deliberately before it +deletes the session and user. -- Persist build, activation, and cleanup work. Use cross-replica locks/leases and generation checks so a stale worker cannot modify a reassigned slot. External operations need stable resource identities and inspectable retry behavior. -- Pool readiness requires completed PVC/configuration work, verified DID publication/resolution, healthy shared dependencies, and no attendee administrator key. Recheck readiness freshness before claim. -- Retain the original provider URL/DID snapshot. Rebuilding the platform stack must not silently rewrite existing identities. Stop allocation when shared hosting or mediation is unavailable and report the reason. -- Configure target ready inventory, build/activation concurrency, storage/DNS budgets, and a total retained/running-VTA ceiling separately. Kept VTAs remain part of the capacity budget; refill cannot exceed the ceiling. -- Parked PVC count is not simultaneous activation capacity. Reserve enough runtime headroom for admitted claims. Existing cached, fail-open estimates are insufficient for atomic batch admission. -- Treat the prototype's concurrency of 8–10 as a candidate, not a measured safe value. Measure build/activation P50/P95, failure rate, and shared-service load. Prioritize attendee activation over background refill. -- Shared event Wi-Fi concentrates traffic behind one IP. Test existing Trust Task per-IP limits and long-poll HTTP/database connection use. Combine event, browser, handle, and global limits; neither IP-only nor freely generated DID-only limits are sufficient. -- Keep pool and DID-host domains stable after the event. For a different zone/subdomain, verify DNS authority and TLS coverage first: a single-level wildcard does not cover arbitrary deeper subdomains. Avoid per-slot certificate issuance. +`batch_id` groups sessions for listing, selecting an available VTA, retaining +unused inventory for another event, and bulk cleanup. Its index supports those +operations. -If FIFO is included initially, create expiring waiting tickets that atomically hold quota, cap queue depth, and define event fairness/cancellation. Order tickets in the database. Assign a slot and start the QR timer only when it is ready; a failed build must not strand a ticket. Expire abandoned tickets and refund once. ETA is an estimate and may be unknown. Do not admit unlimited waiting work without activation capacity. +The table intentionally has no synthetic row ID: the setup-session ID already +uniquely identifies the pooled VTA. -## 9. Retention and Cleanup +## 4. Lifecycle -The confirmed requirement is to batch-clean incomplete event claims and leave successfully claimed VTAs alone. A successful claim here means staff confirmation followed by completed provisioning (`released`/pool `handed_over`); the earlier wallet `claimed` state alone is incomplete. Portal account binding and first DIDComm connection are not cleanup prerequisites. +The setup session owns the lifecycle: -| Event resource state | Batch cleanup decision | +| Session status | Meaning | | --- | --- | -| Event-allocated, unused queued/ready/building slot | Eligible after distribution closes; cancel builds and verify shutdown before removal | -| Open or wallet-claimed but unconfirmed | Terminate reservation atomically, invalidate handle, settle quota, then clean the untouched slot | -| Confirmed or activating | Protect automatically; operator review/retry because a key may already be installed | -| Handed over, with or without Portal account | Always excluded from event cleanup | -| Previously handed over but now stopped/unhealthy | Still excluded; current runtime status is not ownership state | -| Failed/uncertain activation | Quarantine; explicit separate disposition before destruction, never direct reuse | - -Record durable `handed_over_at`/entitlement state, not just `status == running`. A claimed VTA may later crash, stop, or upgrade and must not suddenly look like unused inventory. Similarly, `portal_user_id IS NULL` does not mean unclaimed. Do not use the old `connected_at IS NULL` condition. - -Disable new reservations before generating an event cleanup preview. Protect confirmed, handed-over, and uncertain slots, and exclude inventory reassigned to another event. Give batches/slots explicit event allocation when preparing event stock; otherwise unused slots that never had a reservation have no event association to clean by. Shared unallocated stock requires a separate pool-inventory cleanup selection. Re-lock and recheck each item during execution; a preview is not a permanent authorization snapshot. Never delete namespaces solely because they once belonged to an event. - -Unconfirmed, untouched reservations may return to shared inventory. Shrinking that inventory is a separate cleanup policy. A failed slot that may contain an attendee key must be explicitly disposed of and rebuilt, never reissued with a new handle. Track DNS, DID/ACL, Kubernetes, Vault, and owner cleanup independently. Existing teardown includes best-effort steps, so deleting a session row does not prove full cleanup. Retain tombstones, external resource identifiers, and retryable cleanup work until removal is verified. - -Refill runs only while enabled and within budget; ending an event should not automatically restore its peak inventory target. Closing event distribution, reclaiming unused inventory, and deleting a handed-over VTA are distinct operations in both the UI and API. - -### Ordered Retirement and Resource-Owner Removal - -The internal owner exists for the VTA's lifetime, not the attendee's login lifetime. Inactivity, a stopped Deployment, missing email, or an unbound Portal account are not orphan evidence. A handed-over VTA requires a separately authorized explicit deletion by its linked Portal user or a Farm admin. Deleting that VTA does not delete the person's account or other VTAs. Deleting a person's Portal account must not delete pooled infrastructure: use a restrictive `portal_user_id` FK until an explicit disposition is completed, and retain linked/revoked entitlement history so clearing a link cannot reopen initial binding. - -Implement retirement as durable work with these checkpoints: - -1. **Authorize and fence.** Re-lock the slot/session and recheck the cleanup plan, event membership, generation, and handover status. Mark `retiring`, block allocation/binding/activation/upgrades, and invalidate outstanding unconfirmed reservations. Serialize with build/provision workers; wait for active operations to stop. A lease timeout alone does not prove an external Job stopped. Preserve a cleanup manifest before deleting anything. -2. **Stop writers.** Cancel queued work and remove or stop all setup/import/maintenance Jobs and Deployments for this slot. Verify Pods have terminated and cannot recreate data. Record failure and retry if shutdown is uncertain; never clear the owner while stale work can still write. -3. **Remove external identities and routing.** Delete only this session's DNS records, shared DID-host document, and VTA ACL entry through the recorded provider. Do not delete the platform mediator/DID host, platform account, or shared credentials. Reconcile ambiguous provider responses rather than treating an HTTP timeout as success. -4. **Remove stored data and access.** Delete the session's Kubernetes resources/PVC and exact Vault secret subtree, including stored KV versions/metadata according to the deletion policy. Remove its exclusive Vault auth role/policy and verify the absence of effective workload access, including any issued token permissions. Do not equate role deletion alone with invalidating issued tokens. Check the storage reclaim policy: a deleted PVC may leave a retained backing volume or backup requiring explicit disposition. Never claim all data is erased while retained copies remain. -5. **Remove the namespace.** Require the expected namespace identity/labels, no unexpected workloads or other owners, and no remaining dependencies. Await actual namespace deletion; an accepted deletion request or `Terminating` status is not completion. Do not force-remove finalizers automatically. Quarantine mismatches for operator review. -6. **Finalize database retirement.** Only after all required external checkpoints are verified, mark the tombstone complete, invalidate binding entitlements/attempts, detach completed live FKs, and delete the setup-session and `pool_owner` rows in a transaction. Recheck owner kind, its exclusive slot association, absence of credentials/domains/other live references, and completion under lock. Preserve audit and original resource identifiers without deletion cascades. Keep the batch/event record and retired name reservation; do not reset any sequence. - -The current `setup_sessions.user_id ON DELETE CASCADE` is not a resource cleanup mechanism. Introduce a restrictive guard for pooled owners (for example, a live slot-to-owner `ON DELETE RESTRICT` FK plus guarded finalization), and route all pool deletions through this worker, including existing admin/user session-delete endpoints. Existing teardown must return inspectable step results rather than deleting the session after best-effort calls. A failure leaves the owner/session manifest discoverable as `cleanup_failed` progress on a retiring slot, with retry/backoff and operator diagnostics. Do not delete the original metadata just to make the dashboard appear empty. - -Recovery must handle the crash between an external success and saving its checkpoint: re-inspect the exact resource, treat verified absence as success, and retry remaining steps. Not-found is success only for a known target on a reachable, correct provider; authorization errors or missing provider configuration are not not-found. New creation and deletion share owner/slot locks so a zero-reference check cannot race with a new attachment. Owner rows with zero sessions but unfinished manifests remain protected; legacy orphan candidates require an explicit reviewed cleanup plan. - -For the 50-slot example, if 20 are handed over and 30 remain untouched, an event cleanup can retire those 30 and their internal owners. The 20 retained owners remain even if none of their holders enrolls in the Portal. IDs for the deleted 30 stay unused; future signups and new batches use newly generated IDs. Pausing a retained VTA preserves its owner/PVC/secrets; deletion is a different operation. - -## 10. Implementation Sequence and Acceptance - -1. Validate the Keyring retention/signing integration for later ownership proof; agree its concrete wire contract before implementing Portal binding. -2. Add account-kind classification, permanent owner/slot constraints, batch admission/idempotency, cleanup manifests, and pool/event/station/reservation models. Build directly in each committed owner's namespace/Vault path; address readiness/publication recovery and provide inventory views. -3. Implement station authorization and atomic slot/quota/replay transitions, provisioning recovery, quarantine, and per-attendee QR rotation. Preserve signed-in owner claim flows. -4. Migrate Portal-versus-infrastructure ownership authorization, then implement limited new-account onboarding and existing-account linking with passkey and wallet proof. -5. Add cleanup previews and checkpointed retirement with owner-last deletion, capacity ceilings, refill, and account-kind-aware operational visibility. Update OpenAPI/mobile documentation and rehearse station issuance, later Portal attachment, and cleanup in an isolated environment. Portal attachment is required for feature completion, even if station work is delivered first. - -Required acceptance cases: - -- A 50-slot admission creates exactly 50 distinct internal owners, sessions, and queued members before 202. Concurrent signup/batches remain unique even when IDs interleave. Rollbacks and deleted owners leave harmless gaps; no code derives ownership from a numeric interval. -- Retrying an accepted batch after a lost response returns the same members. A failed admission creates no committed partial batch or external resources; build retries reuse committed owner/session IDs without inserting new users. -- The namespace name/UID, PVC, ServiceAccount scope, Vault role/prefix, hostname, and VTA DID remain identical across ready, confirmed, running, and Portal-bound states. Claim/binding performs no seed copy, namespace move, or owner conversion. -- Pool owner A's workload cannot authenticate as owner B or the platform owner and cannot read/write their Vault paths. Shared mediator/DID-host integration remains functional. A conflicting existing namespace or mismatched role binding fails closed. -- Internal owners cannot log in, enroll passkeys, obtain recovery links, acquire email/fullstack access, or receive additional sessions/domains. Legacy personal accounts without email remain personal; UI filters/counts do not report pool owners as attendees. -- Cleanup protects claimed-but-unbound and stopped retained VTAs. An event with 20 handed-over and 30 unused slots retires only the 30 selected unused owners. Person accounts and unrelated resources are untouched. -- Failure injection at every retirement checkpoint preserves a resumable manifest and owner until completion. Test already-absent resources, namespace finalizers, retained storage, unavailable providers, DB finalization failure, and a crash after an external delete but before its checkpoint. -- Direct deletion of a live pool owner is rejected; cleanup never relies on cascading session deletion. Owner finalization races with claim, Portal binding, worker restart, and attempted new attachment cannot orphan resources or erase audit history. -- Backfill preserves ordinary-session access and quotas, isolates system owners, and rejects an unsafe rollback once pooled sessions depend on separate Portal ownership. Rehearse migration and rollback constraints on an isolated database before rollout. -- Multiple replicas compete for the last slot/event use: only one succeeds. Refreshes and lost responses do not consume additional quota. -- Event/station/reservation ID or trigger QR possession cannot authorize staff confirmation. Cross-origin mutations fail; stale tabs cannot confirm a newer display generation. -- Claim/reject/expiry and confirm/reaper/cleanup races cannot double-allocate or double-refund. Confirmed slots never return to ready. -- Recovery covers crashes between transaction commit and worker startup, partial DID publication, successful Jobs before DB updates, expired worker leases, and workers restarting during deletion. -- Invalid signatures, non-claimant redeem, replay, and expired handles retain existing protocol guarantees. The claimant can recover a lost successful response. -- Two phones scanning the station QR produce one claimant. A mismatch can be rejected safely; the next attendee gets a different handle and, after a successful handover, a different slot. Next attendee never grants another administrator on the previous VTA. -- Refreshing or reauthenticating the station resumes its current reservation. Keyring restarts and missed redeem responses preserve the original claimant and eventual entitlement. -- Provider, DNS, Vault, image-pull, PVC, and readiness failures produce accurate state and bounded retries; waiting is never reported as success. -- Disabling an event preserves confirmed handovers. Cleanup protects reassigned inventory, partial deletions, handed-over slots without Portal users, temporarily unhealthy handed-over VTAs, and ongoing activation. -- Portal binding rejects known DID strings without signatures, old claim proofs, wrong Farm/origin/purpose/target, expired challenges, and simultaneous attempts to bind different accounts. -- Runtime key rotation preserves the separate unbound ownership credential; a lost/revoked credential cannot be replaced by email, a new wallet DID, or a stale ACL snapshot. -- New-account email/passkey/binding finalization is atomic. Public signup, earlier signup cookies, duplicate-email races, and abandoned passkey registration cannot gain access to a claimed VTA. -- Existing-account binding requires that account's passkey and the claimant proof. It changes `portal_user_id` only; namespace, PVC, Vault paths, and VTA identity remain stable. -- Every user read/write/upgrade/export/log route uses Portal authorization, while teardown uses infrastructure ownership. A linked account sees its existing and claimed sessions; an unrelated account and the non-login resource owner cannot access them. -- Rehearsal covers shared Wi-Fi, API restarts, and multiple replicas, measuring request-to-reservation and confirm-to-redeem time and completing actual Keyring DIDComm use. - -For implementation, run `make test` (including `go vet`), test database races and migrations against isolated PostgreSQL, and format changed Go files with gofmt. Before rollout, verify the actual Farm DID, Keyring, and link host. External-readiness statements in older design documents are not current deployment evidence. - -## 11. Remaining Decisions and Integration Gates - -| Question | Proposed default | Implementation consequence | -| --- | --- | --- | -| Does mobile Keyring retain the original claim key and support a user-approved Portal-binding signature? | Require durable claim-key retention and an explicit binding ceremony | Must be verified with the actual mobile client; existing claim/redeem support alone is insufficient | -| Is email a profile/account hint with passkey login, or is email OTP login desired? | Keep current email-label/passkey authentication | Email OTP adds verification, delivery, recovery, and abuse handling beyond current behavior | -| First-event attendance, number of stations, acceptable activation wait, and simultaneous claims? | Pre-build enough inventory; staff retry on shortage | Determines resource ceilings, rate limits, and acceptance load | -| Domain/provider, image, resource profile, and operating ceiling? | Existing managed domain/platform stack and fixed profile | DID names are durable and kept VTAs consume ongoing resources | -| Who handles lost keys, mistaken confirmed claims, and activation failures? | Farm admin review with independent evidence and audit | Email or possession of a station QR cannot authorize takeover or reissue of a keyed VTA | - -## 12. Verification Boundary - -This change updates only this design document. It does not add APIs, migrations, pool resources, deployments, or version changes. The prototype, existing Farm signup/passkey and claim implementation, resource naming, user sequence/schema, teardown behavior, and related key-rotation code were inspected. Parked pre-provisioning, staffed confirmation, permanent per-slot resource ownership without migration, retention of successful claims, and optional later Portal management are confirmed requirements. Account-kind classification, batch admission, Portal authorization changes, and checkpointed cleanup are implementation requirements, not verified runtime behavior. Latency targets, actual mobile Keyring ownership-proof support, and current external-service readiness remain unverified. +| `pool_creating` | User, session, and membership committed; preparation is pending | +| Existing setup stages | Namespace, Vault access, PVC, configuration and DID are being prepared | +| `pool_prepared` | DID publication completed; parked Kubernetes resources are being finalized | +| `pool_waiting` | Deployment exists at zero replicas and the VTA is available to claim | +| `provisioning` | A confirmed admin DID is being imported and the Deployment is starting | +| `running` | At least one Deployment replica passed readiness | +| `failed` | Preparation or activation failed; claim history determines whether cleanup is allowed | +| `deleting` | Explicit cleanup owns the session and no worker may restart it | + +The pool has no parallel state column. A list endpoint derives presentation +state by joining `pool_sessions`, `setup_sessions`, and the latest +`mobile_connections` row. + +## 5. Claim Flow + +1. Lock pool allocation and select the first session in `pool_waiting` with an + empty `admin_did`. +2. Create or return its ordinary active `mobile_connections` claim request. +3. Keyring scans the QR and submits its DID through the existing Trust Task. +4. The station displays the existing six-character comparison code. +5. Administrator confirmation sets `mobile_connections.confirmed_at`, copies + the DID into `setup_sessions.admin_did`, and advances the session to + `provisioning`. +6. The provision job imports the DID into the prepared VTA data. +7. The existing zero-replica Deployment is scaled to one. +8. Readiness advances the session to `running`, and redeem returns the VTA DID. + +The confirmed mobile connection is the durable handover record. A confirmed +activation that later fails remains claimed and cannot be allocated or cleaned +as unused inventory. + +## 6. Cleanup and Retention + +Unused inventory has no expiry. An administrator may keep it for another event +or explicitly select setup-session IDs for cleanup. + +Cleanup is allowed only when all of the following remain true while rows are +locked: + +- the session is a member of `pool_sessions`; +- `setup_sessions.admin_did` is empty; +- no mobile connection is open, claimed, confirmed, provisioning, or released; +- no mobile connection has `confirmed_at` set; +- the user has no email, passkey, or full-stack access. + +Cleanup removes the resources identified by the existing session data, then +deletes the `pool_sessions` membership, setup session, and dedicated user. It +never resets PostgreSQL sequences, so deleted numeric IDs are not reused and +later users continue from the database sequence. + +Claimed sessions are not affected by unused-inventory cleanup. They remain +ordinary running Farm sessions managed by an administrator until a separate +Portal enrollment design is implemented. + +## 7. Failure and Retry Rules + +- Batch creation is transactional for database rows. Either all requested users, + sessions and memberships are admitted, or none are. +- External preparation is asynchronous and idempotent per setup session. +- A failure before confirmation can be retried or explicitly cleaned. +- A failure after confirmation can be retried, but never returned to inventory. +- Two station requests cannot allocate the same session: allocation serializes + pool mutations and locks the chosen setup session. +- Deployment creation at zero replicas is idempotent. Claim activation scales + that existing Deployment instead of creating a second one. + +## 8. Capacity Admission + +Batch admission uses the same strict VTA-only estimate shown by the Admin +Dashboard. A request is rejected before any batch, user, or setup-session row +is inserted when `requested_count` exceeds that estimate. For example, when +the Dashboard reports 47 additional VTA-only sessions, a request for 48 is +rejected and a request for 47 is allowed. + +Capacity is checked after an idempotency lookup, so retrying a successful +request returns its original batch even if current capacity has fallen. New +admissions are serialized to prevent two administrators from consuming the +same capacity concurrently. Capacity lookup fails closed when the Dashboard +cannot produce its estimate. + +Non-running pool members are deducted from the Dashboard's VTA-only estimate. +This reserves their future CPU and memory while their Deployments are parked at +zero replicas; Kubernetes pod requests alone cannot represent that future load. +The reservation remains until the session is running, when the live Deployment +requests become part of cluster statistics. + +There is intentionally no fixed product limit such as 50. The positive-value +database check protects data shape; current cluster resources decide the +operational limit. + +## 9. Migration Safety + +Migration 0043 contains only the two pool tables described above. The earlier +ownership design and its `users.account_kind` and `setup_sessions.portal_user_id` +columns were rolled back and are not part of this schema. diff --git a/internal/connection/store.go b/internal/connection/store.go index c97ce52..59f6481 100644 --- a/internal/connection/store.go +++ b/internal/connection/store.go @@ -91,7 +91,7 @@ func Ready(s *model.SetupSession) bool { if s.IsFullStack() { return s.Status == "awaiting_admin_did" } - return s.Status == "vta_setup_complete" + return s.Status == "vta_setup_complete" || s.Status == model.StatusPoolWaiting } func lockSession(tx *gorm.DB, id uint) (*model.SetupSession, error) { @@ -143,6 +143,16 @@ func finalStatus(status string) bool { // Change creates, refreshes, or cancels an owner-visible reservation. It always // locks the setup session first, matching claim/confirm/provision transitions. func Change(ctx context.Context, db *gorm.DB, sessionID uint, action, expectedID string) (*model.MobileConnection, error) { + return change(ctx, db, sessionID, action, expectedID, false) +} + +// ChangePool is called inside the pool allocation transaction, after the slot +// is reserved. Public owner routes cannot allocate parked pool inventory. +func ChangePool(ctx context.Context, db *gorm.DB, sessionID uint) (*model.MobileConnection, error) { + return change(ctx, db, sessionID, "create", "", true) +} + +func change(ctx context.Context, db *gorm.DB, sessionID uint, action, expectedID string, pooled bool) (*model.MobileConnection, error) { var result *model.MobileConnection err := db.WithContext(ctx).Transaction(func(tx *gorm.DB) error { s, err := lockSession(tx, sessionID) @@ -177,6 +187,10 @@ func Change(ctx context.Context, db *gorm.DB, sessionID uint, action, expectedID if err != nil { return err } + isPool := s.Status == model.StatusPoolWaiting + if isPool != pooled || isPool && operation != OperationProvisionVTA { + return ErrConflict + } if current != nil { switch current.Status { case "claimed", "confirmed", "provisioning": @@ -403,6 +417,9 @@ func StartProvisioning(ctx context.Context, db *gorm.DB, requestID string) (*mod } switch result.Operation { case OperationProvisionVTA: + if s.Status == model.StatusPoolWaiting && result.ConfirmedAt == nil { + return ErrConflict + } if err := beginProvision(tx, s, result.AdminDid, now); err != nil { return err } @@ -491,6 +508,9 @@ func AcceptManual(ctx context.Context, db *gorm.DB, sessionID uint, value string if err != nil { return err } + if s.Status == model.StatusPoolWaiting { + return ErrConflict + } current, err := Current(tx, sessionID) if err != nil { return err diff --git a/internal/handler/dashboard.go b/internal/handler/dashboard.go index f0cf70f..4243f11 100644 --- a/internal/handler/dashboard.go +++ b/internal/handler/dashboard.go @@ -2,6 +2,7 @@ package handler import ( "context" + "errors" "net/http" "time" @@ -9,6 +10,7 @@ import ( "github.com/ic3software/vtafarm-api/internal/capacity" "github.com/ic3software/vtafarm-api/internal/k8s" + "github.com/ic3software/vtafarm-api/internal/model" "github.com/ic3software/vtafarm-api/internal/resourceprofile" "gorm.io/gorm" ) @@ -76,10 +78,13 @@ func (h *DashboardHandler) Get(c *gin.Context) { c.JSON(http.StatusBadGateway, gin.H{"error": "cluster stats: " + err.Error()}) return } - planningUsage := h.usage.Observe(time.Now(), stats.Nodes, stats.MetricsAvailable) + modes, calculated, err := h.capacityEstimates(ctx, stats) + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": "could not calculate capacity"}) + return + } nodes := make([]dashboardNode, len(stats.Nodes)) - var freeNodes []capacity.NodeFree var totalCPU, totalCPUReq, totalCPUUsed, totalMem, totalMemReq, totalMemUsed int64 for i, n := range stats.Nodes { nodes[i] = dashboardNode{ @@ -98,32 +103,9 @@ func (h *DashboardHandler) Get(c *gin.Context) { totalMem += n.MemAllocatableBytes totalMemReq += n.MemRequestedBytes totalMemUsed += n.MemUsedBytes - - // Estimates draw only on nodes new pods can land on. - if n.Schedulable { - usage, usageAvailable := planningUsage[n.Name] - var cpuFree, memFree int64 - if usageAvailable { - cpuFree = capacity.PlanningHeadroom( - n.CPUAllocatableMillis, - n.CPURequestedMillis, - usage.CPUMillis, - ) - memFree = capacity.PlanningHeadroom( - n.MemAllocatableBytes, - n.MemRequestedBytes, - usage.MemBytes, - ) - } - freeNodes = append(freeNodes, capacity.NodeFree{ - CPUMillis: cpuFree, - MemBytes: memFree, - }) - } } storageNodes := make([]dashboardStorageNode, len(stats.StorageNodes)) - var freeDisks []capacity.DiskFree var totalStorage, totalStorageReserved, totalStorageScheduled, totalStorageAvail, totalStorageSched int64 for i, s := range stats.StorageNodes { storageNodes[i] = dashboardStorageNode{ @@ -140,18 +122,12 @@ func (h *DashboardHandler) Get(c *gin.Context) { totalStorageScheduled += s.ScheduledBytes totalStorageAvail += s.AvailableBytes totalStorageSched += s.SchedulableBytes - freeDisks = append(freeDisks, capacity.DiskFree{Bytes: s.SchedulableBytes}) } replicas := stats.StorageReplicaCount estimates := gin.H{} - profiles, err := resourceprofile.Load(ctx, h.db) - if err != nil { - c.JSON(http.StatusInternalServerError, gin.H{"error": "could not load resource defaults"}) - return - } - for _, mode := range capacity.Modes(profiles) { - est := capacity.EstimateMode(mode, freeNodes, freeDisks, replicas, stats.StorageAvailable) + for _, mode := range modes { + est := calculated[mode.Name] var cpuCost, memCost, storageCost int64 for _, comp := range mode.Components { cpuCost += comp.CPUMillis @@ -199,3 +175,102 @@ func (h *DashboardHandler) Get(c *gin.Context) { "estimates": estimates, }) } + +// AvailableVTAOnly returns the same strict capacity shown by the Admin +// Dashboard. Pool admission calls this while holding its database advisory +// lock and before inserting any batch, user, or setup-session row. +func (h *DashboardHandler) AvailableVTAOnly(ctx context.Context) (int, error) { + if h.k8s == nil { + return 0, errors.New("kubernetes is not available") + } + stats, err := h.k8s.ClusterResourceStats(ctx) + if err != nil { + return 0, err + } + _, estimates, err := h.capacityEstimates(ctx, stats) + if err != nil { + return 0, err + } + return estimates[capacity.VtaOnly.Name].Count, nil +} + +func (h *DashboardHandler) capacityEstimates(ctx context.Context, stats *k8s.ClusterStats) ([]capacity.Mode, map[string]capacity.Estimate, error) { + planningUsage := h.usage.Observe(time.Now(), stats.Nodes, stats.MetricsAvailable) + var freeNodes []capacity.NodeFree + for _, node := range stats.Nodes { + if !node.Schedulable { + continue + } + usage, ok := planningUsage[node.Name] + var cpuFree, memoryFree int64 + if ok { + cpuFree = capacity.PlanningHeadroom(node.CPUAllocatableMillis, node.CPURequestedMillis, usage.CPUMillis) + memoryFree = capacity.PlanningHeadroom(node.MemAllocatableBytes, node.MemRequestedBytes, usage.MemBytes) + } + freeNodes = append(freeNodes, capacity.NodeFree{CPUMillis: cpuFree, MemBytes: memoryFree}) + } + freeDisks := make([]capacity.DiskFree, 0, len(stats.StorageNodes)) + for _, disk := range stats.StorageNodes { + freeDisks = append(freeDisks, capacity.DiskFree{Bytes: disk.SchedulableBytes}) + } + profiles, err := resourceprofile.Load(ctx, h.db) + if err != nil { + return nil, nil, err + } + modes := capacity.Modes(profiles) + estimates := make(map[string]capacity.Estimate, len(modes)) + for _, mode := range modes { + estimates[mode.Name] = capacity.EstimateMode( + mode, freeNodes, freeDisks, stats.StorageReplicaCount, stats.StorageAvailable, + ) + } + + reserved, err := h.nonRunningPoolSessions(ctx) + if err != nil { + return nil, nil, err + } + if estimate, ok := estimates[capacity.VtaOnly.Name]; ok { + estimates[capacity.VtaOnly.Name] = subtractReservedSessions(estimate, reserved) + } + return modes, estimates, nil +} + +// A parked Deployment has zero replicas and therefore does not appear in pod +// CPU or memory requests. Reserve one future VTA footprint until the pooled +// session is running; otherwise repeated pool admissions could all consume the +// same capacity displayed by the Dashboard. +func (h *DashboardHandler) nonRunningPoolSessions(ctx context.Context) (int, error) { + if !h.db.Migrator().HasTable(&model.PoolSession{}) { + return 0, nil + } + var count int64 + err := h.db.WithContext(ctx).Table("pool_sessions ps"). + Joins("JOIN setup_sessions s ON s.id = ps.session_id"). + Where("s.status <> ?", "running").Count(&count).Error + return int(count), err +} + +func subtractReservedSessions(estimate capacity.Estimate, reserved int) capacity.Estimate { + if reserved <= 0 { + return estimate + } + subtract := func(value int) int { + if value < 0 { + return value + } + return max(value-reserved, 0) + } + estimate.Count = subtract(estimate.Count) + estimate.ByCPU = subtract(estimate.ByCPU) + estimate.ByMemory = subtract(estimate.ByMemory) + estimate.ByStorage = subtract(estimate.ByStorage) + estimate.LimitingResource = "cpu" + minimum := estimate.ByCPU + if estimate.ByMemory < minimum { + estimate.LimitingResource, minimum = "memory", estimate.ByMemory + } + if estimate.ByStorage >= 0 && estimate.ByStorage < minimum { + estimate.LimitingResource = "storage" + } + return estimate +} diff --git a/internal/handler/dashboard_usage_test.go b/internal/handler/dashboard_usage_test.go index 589f52c..43cfab7 100644 --- a/internal/handler/dashboard_usage_test.go +++ b/internal/handler/dashboard_usage_test.go @@ -4,6 +4,7 @@ import ( "testing" "time" + "github.com/ic3software/vtafarm-api/internal/capacity" "github.com/ic3software/vtafarm-api/internal/k8s" ) @@ -42,6 +43,25 @@ func TestUsageHighWaterWithoutMetrics(t *testing.T) { } } +func TestSubtractReservedSessions(t *testing.T) { + got := subtractReservedSessions(capacity.Estimate{ + Count: 47, ByCPU: 112, ByMemory: 47, ByStorage: 173, LimitingResource: "memory", + }, 7) + if got.Count != 40 || got.ByCPU != 105 || got.ByMemory != 40 || got.ByStorage != 166 { + t.Fatalf("subtractReservedSessions() = %+v", got) + } + if got.LimitingResource != "memory" { + t.Fatalf("limiting resource = %q, want memory", got.LimitingResource) + } + + unknownStorage := subtractReservedSessions(capacity.Estimate{ + Count: 2, ByCPU: 2, ByMemory: 2, ByStorage: -1, + }, 5) + if unknownStorage.Count != 0 || unknownStorage.ByStorage != -1 { + t.Fatalf("unknown storage estimate = %+v", unknownStorage) + } +} + func assertNodeUsage(t *testing.T, got map[string]nodeUsage, node string, cpu, memory int64) { t.Helper() usage, ok := got[node] diff --git a/internal/handler/setup.go b/internal/handler/setup.go index 2338b94..565b0de 100644 --- a/internal/handler/setup.go +++ b/internal/handler/setup.go @@ -820,6 +820,17 @@ func (h *SetupHandler) Delete(c *gin.Context) { // user, AdminDeleteSession accepts any session. Everything after that point is // identical, so both funnel through here. func (h *SetupHandler) teardownSession(c *gin.Context, session *model.SetupSession) { + var poolMembership int64 + if h.db.Migrator().HasTable(&model.PoolSession{}) { + if err := h.db.Model(&model.PoolSession{}).Where("session_id = ?", session.ID).Count(&poolMembership).Error; err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": "Failed to check pool membership"}) + return + } + } + if poolMembership != 0 && session.AdminDid == "" { + c.JSON(http.StatusConflict, gin.H{"error": "Use pool inventory cleanup for unclaimed VTAs. Confirmed claims are protected.", "reason": "pooled_session"}) + return + } if h.orch != nil { if err := h.orch.Cancel(session.ID); err != nil { c.JSON(http.StatusServiceUnavailable, gin.H{"error": "Unable to stop provisioning before deletion. Please retry."}) diff --git a/internal/k8s/vta_resources.go b/internal/k8s/vta_resources.go index 1c77aad..647451f 100644 --- a/internal/k8s/vta_resources.go +++ b/internal/k8s/vta_resources.go @@ -44,9 +44,18 @@ func VtaTuningEnv() []corev1.EnvVar { // /work/vta to match, so VTA resolves its relative config.toml and data_dir // onto it. Port 8100. Idempotent — AlreadyExists is ignored. func (c *Client) CreateVtaDeployment(ctx context.Context, ns string, sessionID uint, image string, resources corev1.ResourceRequirements) error { + return c.createVtaDeployment(ctx, ns, sessionID, image, resources, 1) +} + +// CreateParkedVtaDeployment creates the final Deployment without starting a +// pod. Claim confirmation later uses the ordinary provisioning path to scale it. +func (c *Client) CreateParkedVtaDeployment(ctx context.Context, ns string, sessionID uint, image string, resources corev1.ResourceRequirements) error { + return c.createVtaDeployment(ctx, ns, sessionID, image, resources, 0) +} + +func (c *Client) createVtaDeployment(ctx context.Context, ns string, sessionID uint, image string, resources corev1.ResourceRequirements, replicas int32) error { name := vtaDeploymentName(sessionID) pvcName := VtaPVCName(sessionID) - replicas := int32(1) port := int32(8100) labels := map[string]string{ "app": "vta", diff --git a/internal/k8s/vta_resources_test.go b/internal/k8s/vta_resources_test.go index 03e6740..385c435 100644 --- a/internal/k8s/vta_resources_test.go +++ b/internal/k8s/vta_resources_test.go @@ -44,3 +44,21 @@ func TestCreateVtaDeploymentUsesHealthReadinessProbe(t *testing.T) { t.Fatalf("unexpected readiness timing: %+v", probe) } } + +func TestCreateParkedVtaDeploymentHasNoReplicas(t *testing.T) { + client := &Client{kube: fake.NewSimpleClientset()} + + if err := client.CreateParkedVtaDeployment(context.Background(), "test", 43, "example/vta:test", ComponentResources("10m", "32Mi", "128Mi")); err != nil { + t.Fatalf("CreateParkedVtaDeployment() error = %v", err) + } + + deployment, err := client.kube.AppsV1().Deployments("test").Get( + context.Background(), VtaDeploymentName(43), metav1.GetOptions{}, + ) + if err != nil { + t.Fatalf("get Deployment: %v", err) + } + if deployment.Spec.Replicas == nil || *deployment.Spec.Replicas != 0 { + t.Fatalf("replicas = %v, want 0", deployment.Spec.Replicas) + } +} diff --git a/internal/model/pool.go b/internal/model/pool.go new file mode 100644 index 0000000..159e999 --- /dev/null +++ b/internal/model/pool.go @@ -0,0 +1,31 @@ +package model + +import "time" + +// Pool setup statuses extend the ordinary setup-session lifecycle while a VTA +// is being prepared as inactive inventory. +const ( + StatusPoolCreating = "pool_creating" + StatusPoolPublishing = "pool_publishing" + StatusPoolPrepared = "pool_prepared" + StatusPoolWaiting = "pool_waiting" +) + +// PoolBatch records one administrative request to create pooled VTAs. The +// actual users, sessions, images, infrastructure and lifecycle state remain in +// their existing tables. +type PoolBatch struct { + ID uint `gorm:"primaryKey" json:"id"` + CreatedBy uint `json:"created_by"` + Label string `json:"label"` + RequestedCount int `json:"requested_count"` + IdempotencyKey string `json:"-"` + RequestDigest string `json:"-"` + CreatedAt time.Time `json:"created_at"` +} + +// PoolSession is membership, not another copy of setup-session state. +type PoolSession struct { + SessionID uint `gorm:"primaryKey" json:"session_id"` + BatchID uint `gorm:"not null;index" json:"batch_id"` +} diff --git a/internal/pool/store.go b/internal/pool/store.go new file mode 100644 index 0000000..9647a16 --- /dev/null +++ b/internal/pool/store.go @@ -0,0 +1,285 @@ +package pool + +import ( + "context" + "crypto/rand" + "encoding/hex" + "errors" + "sort" + + "github.com/ic3software/vtafarm-api/internal/connection" + "github.com/ic3software/vtafarm-api/internal/model" + "gorm.io/gorm" + "gorm.io/gorm/clause" +) + +var ( + ErrConflict = errors.New("pool state changed or session is protected") + ErrEmpty = errors.New("no waiting VTA is available in this batch") + ErrCapacity = errors.New("pool inventory limit reached") +) + +// All pool mutations use one transaction lock so two operators cannot select +// the same waiting session. +func lock(tx *gorm.DB) error { + return tx.Exec("SELECT pg_advisory_xact_lock(565443, 1)").Error +} + +type Admission struct { + AdminID uint + Key, Digest, Label string + Count int + Available func(context.Context) (int, error) + Session model.SetupSession + MemoryRequest string + MemoryLimit string +} + +func LookupBatch(ctx context.Context, db *gorm.DB, adminID uint, key, digest string) (*model.PoolBatch, error) { + var batch model.PoolBatch + err := db.WithContext(ctx).Where("created_by = ? AND idempotency_key = ?", adminID, key).First(&batch).Error + if err == nil && batch.RequestDigest != digest { + return nil, ErrConflict + } + return &batch, err +} + +// Admit creates real users and setup sessions. pool_sessions only groups those +// sessions; it does not duplicate their ownership, infrastructure, or status. +func Admit(ctx context.Context, db *gorm.DB, input Admission) (*model.PoolBatch, bool, error) { + if input.Count < 1 || input.Key == "" || len(input.Key) > 128 || input.Session.VtaImage == "" || input.Available == nil { + return nil, false, errors.New("invalid pool admission") + } + var batch *model.PoolBatch + created := false + err := db.WithContext(ctx).Transaction(func(tx *gorm.DB) error { + if err := lock(tx); err != nil { + return err + } + previous, err := LookupBatch(ctx, tx, input.AdminID, input.Key, input.Digest) + if err == nil { + batch = previous + return nil + } + if !errors.Is(err, gorm.ErrRecordNotFound) { + return err + } + available, err := input.Available(ctx) + if err != nil { + return err + } + if input.Count > available { + return ErrCapacity + } + batch = &model.PoolBatch{ + CreatedBy: input.AdminID, Label: input.Label, RequestedCount: input.Count, + IdempotencyKey: input.Key, RequestDigest: input.Digest, + } + if err := tx.Create(batch).Error; err != nil { + return err + } + for range input.Count { + user, err := createUser(tx) + if err != nil { + return err + } + name, err := randomName() + if err != nil { + return err + } + session := input.Session + session.ID, session.UserID, session.Status = 0, user.ID, model.StatusPoolCreating + session.VtaName, session.AdminDid = name, "" + session.Subdomain = input.Session.Subdomain + name + session.VtaDidUrl = session.DidHostingServerURL + "/" + name + "-vta" + if err := tx.Create(&session).Error; err != nil { + return err + } + if err := tx.Create(&model.PoolSession{BatchID: batch.ID, SessionID: session.ID}).Error; err != nil { + return err + } + resource := model.WorkloadResource{ + SetupSessionID: session.ID, Component: "vta", + MemoryRequest: input.MemoryRequest, MemoryLimit: input.MemoryLimit, + } + if err := tx.Create(&resource).Error; err != nil { + return err + } + } + created = true + return nil + }) + return batch, created, err +} + +func createUser(tx *gorm.DB) (*model.User, error) { + for range 5 { + name, err := randomName() + if err != nil { + return nil, err + } + user := &model.User{UniqueId: name} + result := tx.Clauses(clause.OnConflict{Columns: []clause.Column{{Name: "unique_id"}}, DoNothing: true}).Create(user) + if result.Error != nil { + return nil, result.Error + } + if result.RowsAffected == 1 { + return user, nil + } + } + return nil, errors.New("could not allocate user identifier") +} + +func randomName() (string, error) { + var value [6]byte + if _, err := rand.Read(value[:]); err != nil { + return "", err + } + return hex.EncodeToString(value[:]), nil +} + +// Reserve chooses one waiting session and creates its ordinary mobile claim. +// The QR lifecycle remains entirely in mobile_connections. +func Reserve(ctx context.Context, db *gorm.DB, batchID uint) (*model.MobileConnection, error) { + var result *model.MobileConnection + err := db.WithContext(ctx).Transaction(func(tx *gorm.DB) error { + if err := lock(tx); err != nil { + return err + } + var candidates []model.SetupSession + if err := tx.Table("setup_sessions s"). + Select("s.*"). + Joins("JOIN pool_sessions ps ON ps.session_id = s.id"). + Where("ps.batch_id = ? AND s.status = ? AND s.admin_did = ''", batchID, model.StatusPoolWaiting). + Order("s.id").Find(&candidates).Error; err != nil { + return err + } + for i := range candidates { + var session model.SetupSession + query := tx.Clauses(clause.Locking{Strength: "UPDATE", Options: "SKIP LOCKED"}).First(&session, candidates[i].ID) + if query.Error != nil { + if errors.Is(query.Error, gorm.ErrRecordNotFound) { + continue + } + return query.Error + } + if session.Status != model.StatusPoolWaiting || session.AdminDid != "" { + continue + } + mobile, err := connection.ChangePool(ctx, tx, session.ID) + if err != nil { + if errors.Is(err, connection.ErrConflict) { + continue + } + return err + } + result = mobile + return nil + } + return ErrEmpty + }) + return result, err +} + +func Confirm(ctx context.Context, db *gorm.DB, requestID string) (*model.MobileConnection, error) { + var result *model.MobileConnection + err := db.WithContext(ctx).Transaction(func(tx *gorm.DB) error { + if err := lock(tx); err != nil { + return err + } + var candidate model.MobileConnection + if err := tx.Select("session_id").First(&candidate, "id = ?", requestID).Error; err != nil { + return err + } + if !IsSession(tx, candidate.SessionID) { + return ErrConflict + } + confirmed, err := connection.Confirm(ctx, tx, candidate.SessionID, requestID) + if err != nil { + var current model.MobileConnection + if loadErr := tx.First(¤t, "id = ?", requestID).Error; loadErr == nil && + (current.Status == "confirmed" || current.Status == "provisioning" || current.Status == "released") { + result = ¤t + return nil + } + return err + } + result, err = connection.StartProvisioning(ctx, tx, confirmed.ID) + return err + }) + return result, err +} + +func Reject(ctx context.Context, db *gorm.DB, requestID string) error { + return db.WithContext(ctx).Transaction(func(tx *gorm.DB) error { + if err := lock(tx); err != nil { + return err + } + var candidate model.MobileConnection + if err := tx.Select("session_id").First(&candidate, "id = ?", requestID).Error; err != nil { + return err + } + if !IsSession(tx, candidate.SessionID) { + return ErrConflict + } + _, err := connection.Reject(ctx, tx, candidate.SessionID, requestID) + return err + }) +} + +func IsSession(db *gorm.DB, sessionID uint) bool { + var count int64 + return db.Model(&model.PoolSession{}).Where("session_id = ?", sessionID).Count(&count).Error == nil && count == 1 +} + +// QueueCleanup accepts setup-session IDs. Confirmed claims remain protected +// even if activation failed and the setup session is no longer running. +func QueueCleanup(ctx context.Context, db *gorm.DB, sessionIDs []uint) error { + if len(sessionIDs) < 1 { + return ErrConflict + } + sessionIDs = append([]uint(nil), sessionIDs...) + sort.Slice(sessionIDs, func(i, j int) bool { return sessionIDs[i] < sessionIDs[j] }) + return db.WithContext(ctx).Transaction(func(tx *gorm.DB) error { + if err := lock(tx); err != nil { + return err + } + for _, sessionID := range sessionIDs { + var membership model.PoolSession + if err := tx.First(&membership, "session_id = ?", sessionID).Error; err != nil { + return err + } + var session model.SetupSession + if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).First(&session, sessionID).Error; err != nil { + return err + } + if session.AdminDid != "" { + return ErrConflict + } + var claims int64 + if err := tx.Model(&model.MobileConnection{}). + Where("session_id = ? AND (confirmed_at IS NOT NULL OR status IN ?)", sessionID, []string{"open", "claimed", "confirmed", "provisioning", "released"}). + Count(&claims).Error; err != nil { + return err + } + if claims != 0 { + return ErrConflict + } + var user model.User + if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).First(&user, session.UserID).Error; err != nil { + return err + } + var passkeys int64 + if err := tx.Model(&model.UserPasskey{}).Where("user_id = ?", user.ID).Count(&passkeys).Error; err != nil { + return err + } + if user.Email != nil || passkeys != 0 || user.FullstackAccess { + return ErrConflict + } + if err := connection.Stop(ctx, tx, session.ID); err != nil { + return err + } + } + return nil + }) +} diff --git a/internal/pool/store_test.go b/internal/pool/store_test.go new file mode 100644 index 0000000..645d6d0 --- /dev/null +++ b/internal/pool/store_test.go @@ -0,0 +1,132 @@ +package pool + +import ( + "context" + "errors" + "testing" + + "github.com/ic3software/vtafarm-api/internal/connection" + "github.com/ic3software/vtafarm-api/internal/didkey" + "github.com/ic3software/vtafarm-api/internal/model" + "github.com/ic3software/vtafarm-api/internal/testutil" + "gorm.io/gorm" +) + +func poolDatabase(t *testing.T) *gorm.DB { + t.Helper() + return testutil.Postgres(t) +} + +func admission(t *testing.T, db *gorm.DB, count, available int) (*model.PoolBatch, bool, error) { + t.Helper() + admin := model.Admin{UniqueId: "pool-admin"} + if err := db.FirstOrCreate(&admin, "unique_id = ?", admin.UniqueId).Error; err != nil { + t.Fatal(err) + } + return Admit(context.Background(), db, Admission{ + AdminID: admin.ID, + Key: "create-event-one", + Digest: "digest-one", + Label: "Event One", + Count: count, + Available: func(context.Context) (int, error) { + return available, nil + }, + Session: model.SetupSession{ + Mode: model.ModeVtaOnly, + Domain: "example.com", + Subdomain: "vta-", + VtaImage: "example/vta:test", + DidHostingServerURL: "https://dids.example.com", + DidHostingControlURL: "https://dids.example.com", + }, + MemoryRequest: "32Mi", + MemoryLimit: "64Mi", + }) +} + +func TestAdmitStoresOnlyBatchMembership(t *testing.T) { + db := poolDatabase(t) + batch, created, err := admission(t, db, 2, 2) + if err != nil || !created { + t.Fatalf("Admit() = %#v, %v, %v", batch, created, err) + } + + var memberships []model.PoolSession + if err := db.Where("batch_id = ?", batch.ID).Order("session_id").Find(&memberships).Error; err != nil { + t.Fatal(err) + } + if len(memberships) != 2 { + t.Fatalf("memberships = %d, want 2", len(memberships)) + } + for _, membership := range memberships { + var session model.SetupSession + if err := db.First(&session, membership.SessionID).Error; err != nil { + t.Fatal(err) + } + if session.Status != model.StatusPoolCreating || session.UserID == 0 { + t.Fatalf("session not admitted as a real owned VTA: %+v", session) + } + } + + retried, created, err := admission(t, db, 2, 0) + if err != nil || created || retried.ID != batch.ID { + t.Fatalf("idempotent Admit() = %#v, %v, %v", retried, created, err) + } +} + +func TestAdmitUsesDashboardCapacity(t *testing.T) { + db := poolDatabase(t) + if _, _, err := admission(t, db, 48, 47); !errors.Is(err, ErrCapacity) { + t.Fatalf("Admit(48 with 47 available) error = %v, want ErrCapacity", err) + } + var batches int64 + if err := db.Model(&model.PoolBatch{}).Count(&batches).Error; err != nil { + t.Fatal(err) + } + if batches != 0 { + t.Fatalf("capacity rejection created %d batches", batches) + } + if _, created, err := admission(t, db, 47, 47); err != nil || !created { + t.Fatalf("Admit(47 with 47 available) created=%v error=%v", created, err) + } +} + +func TestConfirmedFailureCannotReturnToInventory(t *testing.T) { + db := poolDatabase(t) + batch, _, err := admission(t, db, 1, 1) + if err != nil { + t.Fatal(err) + } + var membership model.PoolSession + if err := db.First(&membership, "batch_id = ?", batch.ID).Error; err != nil { + t.Fatal(err) + } + if err := db.Model(&model.SetupSession{}).Where("id = ?", membership.SessionID).Updates(map[string]any{ + "status": model.StatusPoolWaiting, + "vta_did": "did:webvh:example.com:test", + }).Error; err != nil { + t.Fatal(err) + } + + reservation, err := Reserve(context.Background(), db, batch.ID) + if err != nil { + t.Fatal(err) + } + did, err := didkey.Generate() + if err != nil { + t.Fatal(err) + } + if _, err := connection.Claim(context.Background(), db, *reservation.ClaimHandle, did); err != nil { + t.Fatal(err) + } + if _, err := Confirm(context.Background(), db, reservation.ID); err != nil { + t.Fatal(err) + } + if err := db.Model(&model.SetupSession{}).Where("id = ?", membership.SessionID).Update("status", "failed").Error; err != nil { + t.Fatal(err) + } + if err := QueueCleanup(context.Background(), db, []uint{membership.SessionID}); !errors.Is(err, ErrConflict) { + t.Fatalf("QueueCleanup() error = %v, want ErrConflict", err) + } +} diff --git a/internal/setup/orchestrator.go b/internal/setup/orchestrator.go index e5b43e6..500233a 100644 --- a/internal/setup/orchestrator.go +++ b/internal/setup/orchestrator.go @@ -18,6 +18,7 @@ import ( "github.com/ic3software/vtafarm-api/internal/model" "github.com/ic3software/vtafarm-api/internal/resourceprofile" "github.com/ic3software/vtafarm-api/internal/vault" + "github.com/ic3software/vtafarm-api/internal/webvh" ) // Orchestrator drives a setup session through its full lifecycle. @@ -169,7 +170,11 @@ func (o *Orchestrator) Cancel(sessionID uint) error { // Resume re-attaches goroutines for sessions that were interrupted mid-run at startup. func (o *Orchestrator) Resume(ctx context.Context) { var running []model.SetupSession - if err := o.db.Where("status = ?", "vta_setup_running").Find(&running).Error; err != nil { + query := o.db.Where("status = ?", "vta_setup_running") + if o.db.Migrator().HasTable(&model.PoolSession{}) { + query = query.Where("NOT EXISTS (SELECT 1 FROM pool_sessions WHERE session_id = setup_sessions.id)") + } + if err := query.Find(&running).Error; err != nil { log.Printf("[orchestrator] resume: query failed: %v", err) } for _, s := range running { @@ -197,6 +202,11 @@ func (o *Orchestrator) runSetup(ctx context.Context, sessionID uint) { log.Printf("[orchestrator] session %d: load failed: %v", sessionID, err) return } + pooled, err := o.isPoolSession(ctx, sessionID) + if err != nil { + log.Printf("[orchestrator] session %d: pool membership lookup failed: %v", sessionID, err) + return + } ns := o.k8s.UserNamespace(fmt.Sprintf("%d", session.UserID)) @@ -280,7 +290,7 @@ func (o *Orchestrator) runSetup(ctx context.Context, sessionID uint) { // Extract did.jsonl content appended to the job logs after the marker. didLog := ParseVtaDidLog(logs) - if session.ConnectionSource == model.ConnectionExternal { + if session.ConnectionSource == model.ConnectionExternal || pooled { if didLog == "" || len(didLog) >= MaxVtaDIDLogBytes { o.markFailed(sessionID, "VTA setup produced no usable DID log for external publication") return @@ -292,6 +302,9 @@ func (o *Orchestrator) runSetup(ctx context.Context, sessionID uint) { readyStatus = "awaiting_did_publication" storedDidLog = didLog } + if pooled { + readyStatus, storedDidLog = model.StatusPoolPublishing, didLog + } if err := o.db.Model(&model.SetupSession{}).Where("id = ?", sessionID).Updates(map[string]any{ "status": readyStatus, "vta_did": vtaDID, @@ -302,6 +315,12 @@ func (o *Orchestrator) runSetup(ctx context.Context, sessionID uint) { return } log.Printf("[orchestrator] session %d: setup complete, VTA DID=%s", sessionID, vtaDID) + if pooled { + if err := o.publishPoolDID(ctx, sessionID); err != nil && ctx.Err() == nil { + o.markFailed(sessionID, err.Error()) + } + return + } if session.ConnectionSource == model.ConnectionExternal { return } @@ -457,6 +476,21 @@ func (o *Orchestrator) runProvision(ctx context.Context, sessionID uint, adminDi o.markFailed(sessionID, "failed to load resource defaults: "+err.Error()) return } + pooled, err := o.isPoolSession(ctx, session.ID) + if err != nil { + o.markFailed(sessionID, "failed to identify pool session") + return + } + if pooled { + var saved model.WorkloadResource + if err := o.db.Where("setup_session_id = ? AND component = 'vta'", session.ID).First(&saved).Error; err != nil { + o.markFailed(sessionID, "pool resource profile is missing") + return + } + p := profiles[k8s.ComponentVTA] + p.MemoryRequest, p.MemoryLimit = saved.MemoryRequest, saved.MemoryLimit + profiles[k8s.ComponentVTA] = p + } if err := o.k8s.CreateVtaDeployment(ctx, ns, sessionID, session.VtaImage, profiles.Requirements(k8s.ComponentVTA)); err != nil { if ctx.Err() != nil { return @@ -464,6 +498,12 @@ func (o *Orchestrator) runProvision(ctx context.Context, sessionID uint, adminDi o.markFailed(sessionID, "failed to create VTA deployment: "+err.Error()) return } + if pooled { + if err := o.k8s.ScaleComponentDeployment(ctx, ns, k8s.VtaDeploymentName(sessionID), 1); err != nil { + o.markFailed(sessionID, "failed to start parked VTA deployment: "+err.Error()) + return + } + } if err := o.k8s.CreateVtaService(ctx, ns, sessionID); err != nil { if ctx.Err() != nil { @@ -499,6 +539,102 @@ func (o *Orchestrator) runProvision(ctx context.Context, sessionID uint, adminDi "updated_at": time.Now(), }) log.Printf("[orchestrator] session %d: VTA running at %s", sessionID, session.FQDN()) + if pooled { + if current, err := connection.Current(o.db, session.ID); err == nil && current != nil { + _ = connection.MarkReleased(ctx, o.db, current.ID) + } + } +} + +// BuildPool runs synchronously under the pool worker's cross-replica lock. +func (o *Orchestrator) BuildPool(ctx context.Context, sessionID uint) error { + var session model.SetupSession + if err := o.db.WithContext(ctx).First(&session, sessionID).Error; err != nil { + return err + } + pooled, err := o.isPoolSession(ctx, sessionID) + if err != nil { + return err + } + if !pooled || session.AdminDid != "" || session.Status == "deleting" { + return connection.ErrConflict + } + if session.Status == model.StatusPoolWaiting { + return nil + } + if session.VtaDid == "" { + o.runSetup(ctx, sessionID) + } else if err := o.publishPoolDID(ctx, sessionID); err != nil { + return err + } + if err := ctx.Err(); err != nil { + return err + } + if err := o.db.First(&session, sessionID).Error; err != nil { + return err + } + if session.Status != model.StatusPoolPrepared { + return fmt.Errorf("pool setup did not complete: %s", session.ErrorMsg) + } + profiles, err := resourceprofile.Load(ctx, o.db) + if err != nil { + return err + } + var saved model.WorkloadResource + if err := o.db.Where("setup_session_id = ? AND component = 'vta'", session.ID).First(&saved).Error; err != nil { + return err + } + profile := profiles[k8s.ComponentVTA] + profile.MemoryRequest, profile.MemoryLimit = saved.MemoryRequest, saved.MemoryLimit + profiles[k8s.ComponentVTA] = profile + ns := o.k8s.UserNamespace(fmt.Sprintf("%d", session.UserID)) + if err := o.k8s.CreateParkedVtaDeployment(ctx, ns, session.ID, session.VtaImage, profiles.Requirements(k8s.ComponentVTA)); err != nil { + return err + } + if err := o.k8s.CreateVtaService(ctx, ns, session.ID); err != nil { + return err + } + if err := o.k8s.CreateVtaIngress(ctx, ns, session.ID, session.FQDN()); err != nil { + return err + } + return o.db.WithContext(ctx).Model(&session).Updates(map[string]any{ + "status": model.StatusPoolWaiting, "error_msg": "", "failed_stage": "", "updated_at": time.Now(), + }).Error +} + +func (o *Orchestrator) publishPoolDID(ctx context.Context, sessionID uint) error { + var s model.SetupSession + if err := o.db.WithContext(ctx).First(&s, sessionID).Error; err != nil { + return err + } + if o.didHosting == nil || s.VtaDid == "" || s.VtaDidLog == "" { + return fmt.Errorf("pool DID publication is not configured or its log is missing") + } + dh, err := o.didHosting.For(s.DidHostingControlURL, s.DIDHostingDid) + if err != nil { + return err + } + resolver := webvh.NewResolver(5 * time.Second) + // Resolving the exact self-certifying DID validates a prior successful + // publication. Never overwrite a different document after a conflict. + if err := resolver.ResolveDID(ctx, s.VtaDid); err != nil { + if err := dh.RegisterDid(ctx, VtaDidPath(s.VtaName), s.VtaDidLog); err != nil { + return err + } + if err := resolver.ResolveDID(ctx, s.VtaDid); err != nil { + return err + } + } + return o.db.WithContext(ctx).Model(&s).Where("status <> 'deleting'").Updates(map[string]any{"status": model.StatusPoolPrepared, "error_msg": "", "failed_stage": ""}).Error +} + +func (o *Orchestrator) isPoolSession(ctx context.Context, sessionID uint) (bool, error) { + if !o.db.Migrator().HasTable(&model.PoolSession{}) { + return false, nil + } + var count int64 + err := o.db.WithContext(ctx).Model(&model.PoolSession{}).Where("session_id = ?", sessionID).Count(&count).Error + return count == 1, err } func (o *Orchestrator) markFailed(sessionID uint, msg string) { diff --git a/migrations/000043_vta_pool.down.sql b/migrations/000043_vta_pool.down.sql new file mode 100644 index 0000000..3b91200 --- /dev/null +++ b/migrations/000043_vta_pool.down.sql @@ -0,0 +1,7 @@ +DO $$ BEGIN + IF EXISTS (SELECT 1 FROM pool_sessions) OR EXISTS (SELECT 1 FROM pool_batches) THEN + RAISE EXCEPTION 'cannot roll back while pool records exist'; + END IF; +END $$; +DROP TABLE pool_sessions; +DROP TABLE pool_batches; diff --git a/migrations/000043_vta_pool.up.sql b/migrations/000043_vta_pool.up.sql new file mode 100644 index 0000000..bda8f96 --- /dev/null +++ b/migrations/000043_vta_pool.up.sql @@ -0,0 +1,17 @@ +CREATE TABLE pool_batches ( + id bigserial PRIMARY KEY, + created_by bigint NOT NULL REFERENCES admins(id) ON DELETE RESTRICT, + label text NOT NULL, + requested_count integer NOT NULL CHECK (requested_count > 0), + idempotency_key text NOT NULL, + request_digest text NOT NULL, + created_at timestamptz NOT NULL DEFAULT now(), + UNIQUE (created_by, idempotency_key) +); + +CREATE TABLE pool_sessions ( + session_id bigint PRIMARY KEY REFERENCES setup_sessions(id) ON DELETE RESTRICT, + batch_id bigint NOT NULL REFERENCES pool_batches(id) ON DELETE RESTRICT +); + +CREATE INDEX pool_sessions_batch_id ON pool_sessions(batch_id); From 6985688aec6b9b3606a6664a9b102052d8565ad8 Mon Sep 17 00:00:00 2001 From: vthwang Date: Thu, 15 Oct 2026 00:10:56 -0700 Subject: [PATCH 3/4] feat: complete pooled VTA claim workflow Make prebuilt inventory durable and safe to distribute without transferring infrastructure ownership. - add admin pool APIs, reconciliation workers, cleanup, and retry handling - reserve dashboard capacity and activate VTAs through the existing Keyring claim protocol - document and test the contract, crash recovery, and protected cleanup behavior Signed-off-by: vthwang --- internal/apidocs/openapi.yaml | 318 ++++++++++++++++- internal/cloudflare/client.go | 42 +++ internal/handler/admin_pool.go | 515 ++++++++++++++++++++++++++++ internal/handler/admin_pool_test.go | 56 +++ internal/handler/setup.go | 85 ++++- internal/pool/runner.go | 160 +++++++++ internal/pool/runner_test.go | 49 +++ internal/pool/store.go | 78 ++++- internal/pool/store_test.go | 101 ++++++ internal/router/router.go | 19 +- 10 files changed, 1413 insertions(+), 10 deletions(-) create mode 100644 internal/handler/admin_pool.go create mode 100644 internal/handler/admin_pool_test.go create mode 100644 internal/pool/runner.go create mode 100644 internal/pool/runner_test.go diff --git a/internal/apidocs/openapi.yaml b/internal/apidocs/openapi.yaml index 64bf2b0..f23857a 100644 --- a/internal/apidocs/openapi.yaml +++ b/internal/apidocs/openapi.yaml @@ -231,6 +231,60 @@ components: [id, status, requested_count, created_count, running_count, failed_count, vta_image, created_at, updated_at, sessions] + PoolCounts: + type: object + properties: + total: { type: integer } + creating: { type: integer } + waiting: { type: integer } + claiming: { type: integer } + activating: { type: integer } + running: { type: integer } + failed: { type: integer } + deleting: { type: integer } + required: [total, creating, waiting, claiming, activating, running, failed, deleting] + + PoolSession: + type: object + properties: + session_id: { type: integer } + user_id: { type: integer } + vta_name: { type: string } + status: { type: string } + error_msg: { type: string } + url: { type: string, format: uri } + claimed: { type: boolean } + cleanable: { type: boolean } + claim_state: + type: string + enum: [open, claimed, confirmed, provisioning, released, rejected, lapsed] + required: [session_id, user_id, vta_name, status, url, claimed, cleanable] + + PoolBatch: + type: object + properties: + id: { type: integer } + label: { type: string } + requested_count: { type: integer } + vta_image: { type: string } + created_at: { type: string, format: date-time } + counts: { $ref: "#/components/schemas/PoolCounts" } + sessions: + type: array + items: { $ref: "#/components/schemas/PoolSession" } + required: [id, label, requested_count, created_at, counts] + + PoolClaim: + type: object + properties: + batch_id: { type: integer } + session_id: { type: integer } + vta_name: { type: string } + enabled: { type: boolean } + server_time: { type: string, format: date-time } + connection: { $ref: "#/components/schemas/MobileConnection" } + required: [batch_id, session_id, vta_name, enabled, server_time, connection] + RateLimited: type: object properties: @@ -1627,6 +1681,264 @@ paths: "503": description: Kubernetes is not configured + /api/v1/admin/pool-batches: + get: + summary: List pooled VTA batches + description: Returns batch summaries with counts derived from their setup sessions and latest claim records. + tags: [Admin] + security: + - CookieAuthAdmin: [] + responses: + "200": + description: Pool batch summaries + content: + application/json: + schema: + type: array + items: { $ref: "#/components/schemas/PoolBatch" } + post: + summary: Create a pooled VTA batch + description: | + Atomically creates one dedicated user and setup session per requested + VTA, then queues durable background preparation. The request is rejected + before any rows are inserted when `count` exceeds the same strict + VTA-only capacity shown by the Admin Dashboard. + + Repeating an accepted request with the same `Idempotency-Key` and body + returns the original batch. Reusing the key with a different body is a + conflict. + tags: [Admin] + security: + - CookieAuthAdmin: [] + parameters: + - in: header + name: Idempotency-Key + required: true + schema: { type: string, minLength: 1, maxLength: 128 } + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + label: { type: string, minLength: 1, maxLength: 120 } + count: { type: integer, minimum: 1 } + vta_image: { type: string, minLength: 1 } + required: [label, count, vta_image] + responses: + "201": + description: Batch admitted and queued + content: + application/json: + schema: { $ref: "#/components/schemas/PoolBatch" } + "200": + description: Idempotent replay of an existing batch + content: + application/json: + schema: { $ref: "#/components/schemas/PoolBatch" } + "409": + description: Idempotency key conflicts with another request + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + "422": + description: Requested count exceeds live VTA-only capacity + content: + application/json: + schema: + allOf: + - { $ref: "#/components/schemas/Error" } + - type: object + properties: + reason: { type: string, const: insufficient_capacity } + available: { type: integer, minimum: 0 } + + /api/v1/admin/pool-batches/{id}: + get: + summary: Get one pooled VTA batch + tags: [Admin] + security: + - CookieAuthAdmin: [] + parameters: + - in: path + name: id + required: true + schema: { type: integer, minimum: 1 } + responses: + "200": + description: Batch details and every remaining member + content: + application/json: + schema: { $ref: "#/components/schemas/PoolBatch" } + "404": + description: Batch not found + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + + /api/v1/admin/pool-batches/{id}/claims: + post: + summary: Reserve the next waiting VTA and create its Keyring QR + description: Concurrent calls never receive the same active setup session. The returned trigger link is rendered as the QR payload. + tags: [Admin] + security: + - CookieAuthAdmin: [] + parameters: + - in: path + name: id + required: true + schema: { type: integer, minimum: 1 } + responses: + "200": + description: Claim QR reservation + content: + application/json: + schema: { $ref: "#/components/schemas/PoolClaim" } + "409": + description: No waiting VTA is available + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + + /api/v1/admin/pool-claims/{request_id}: + get: + summary: Poll a pooled VTA claim + description: Reconciles QR and confirmation expiry before returning the latest state. + tags: [Admin] + security: + - CookieAuthAdmin: [] + parameters: + - in: path + name: request_id + required: true + schema: { type: string, format: uuid } + responses: + "200": + description: Current claim state + content: + application/json: + schema: { $ref: "#/components/schemas/PoolClaim" } + "404": + description: Claim not found + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + + /api/v1/admin/pool-claims/{request_id}/confirm: + post: + summary: Confirm matching Keyring comparison codes and activate the VTA + description: Confirmation permanently protects this VTA from unused-inventory cleanup and starts its parked Deployment. + tags: [Admin] + security: + - CookieAuthAdmin: [] + parameters: + - in: path + name: request_id + required: true + schema: { type: string, format: uuid } + responses: + "200": + description: Confirmed or provisioning claim + content: + application/json: + schema: { $ref: "#/components/schemas/PoolClaim" } + "409": + description: Claim is not confirmable + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + "410": + description: Confirmation window expired + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + + /api/v1/admin/pool-claims/{request_id}/reject: + post: + summary: Reject a mismatched pooled VTA claim + tags: [Admin] + security: + - CookieAuthAdmin: [] + parameters: + - in: path + name: request_id + required: true + schema: { type: string, format: uuid } + responses: + "200": + description: Rejected claim; its untouched VTA may be offered again + content: + application/json: + schema: { $ref: "#/components/schemas/PoolClaim" } + + /api/v1/admin/pool-sessions/cleanup: + post: + summary: Queue cleanup for selected unused pooled VTAs + description: The whole selection is rejected atomically if any member is claimed, actively being claimed, or has acquired login credentials. + tags: [Admin] + security: + - CookieAuthAdmin: [] + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + session_ids: + type: array + minItems: 1 + uniqueItems: true + items: { type: integer, minimum: 1 } + required: [session_ids] + responses: + "202": + description: Cleanup queued; background workers will retry after restarts + content: + application/json: + schema: + type: object + properties: + session_ids: + type: array + items: { type: integer } + status: { type: string, const: deleting } + required: [session_ids, status] + "409": + description: At least one selected session is protected + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + + /api/v1/admin/pool-sessions/{id}/retry: + post: + summary: Retry preparation of an unclaimed failed pooled VTA + tags: [Admin] + security: + - CookieAuthAdmin: [] + parameters: + - in: path + name: id + required: true + schema: { type: integer, minimum: 1 } + responses: + "202": + description: Preparation requeued + content: + application/json: + schema: + type: object + properties: + session_id: { type: integer } + status: { type: string, const: pool_creating } + required: [session_id, status] + "409": + description: Session is claimed or is not a retryable failed pool member + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + /api/v1/admin/load-tests: post: summary: Start a VTA provisioning load test @@ -1943,7 +2255,7 @@ paths: "400": { description: Invalid JSON body } "401": { description: Admin authentication required } "403": { description: Insufficient role } - "422": { description: Missing, duplicate, or unknown component, or invalid memory values } + "422": { description: "Missing, duplicate, or unknown component, or invalid memory values" } "500": { description: Could not save defaults } /api/v1/admin/setup-sessions/resources: @@ -2009,7 +2321,7 @@ paths: error: { type: string } "404": { description: One or more sessions were not found } "409": { description: One or more sessions are not running } - "422": { description: Validation, capacity, or Kubernetes dry-run failed } + "422": { description: "Validation, capacity, or Kubernetes dry-run failed" } "503": { description: Kubernetes is not configured } /api/v1/admin/setup-sessions/{id}/resources: @@ -5759,7 +6071,7 @@ paths: '400': {description: Malformed document or payload} '409': {description: Document ID reused with different content} '415': {description: Content-Type is not application/json} - '422': {description: Task, proof, recipient, expiry, or business-policy rejection} + '422': {description: "Task, proof, recipient, expiry, or business-policy rejection"} '429': {description: Rate limited} '503': {description: Trust Task service or replay record unavailable} diff --git a/internal/cloudflare/client.go b/internal/cloudflare/client.go index bad6f47..71f3144 100644 --- a/internal/cloudflare/client.go +++ b/internal/cloudflare/client.go @@ -8,6 +8,7 @@ import ( "fmt" "net" "net/http" + "net/url" ) const baseURL = "https://api.cloudflare.com/client/v4" @@ -121,6 +122,47 @@ func (c *Client) CreateARecord(ctx context.Context, name, ip string) (string, er return result.Result.ID, nil } +// EnsureARecord returns an existing matching record or creates it. Pool builds +// use this instead of a blind create so a process crash after Cloudflare +// accepts the record but before its ID reaches PostgreSQL is recoverable. +func (c *Client) EnsureARecord(ctx context.Context, name, ip string) (string, error) { + endpoint, err := url.Parse(fmt.Sprintf("%s/zones/%s/dns_records", baseURL, c.zoneID)) + if err != nil { + return "", err + } + query := endpoint.Query() + query.Set("type", "A") + query.Set("name", name) + endpoint.RawQuery = query.Encode() + req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint.String(), nil) + if err != nil { + return "", err + } + resp, err := c.do(req) + if err != nil { + return "", err + } + defer resp.Body.Close() + var result apiResponse[[]DNSRecord] + if err := json.NewDecoder(resp.Body).Decode(&result); err != nil { + return "", err + } + if err := cfError(result.Success, result.Errors); err != nil { + return "", err + } + if len(result.Result) > 1 { + return "", fmt.Errorf("cloudflare: multiple A records exist for %s", name) + } + if len(result.Result) == 1 { + record := result.Result[0] + if record.Content != ip || !record.Proxied { + return "", fmt.Errorf("cloudflare: A record for %s has unexpected target or proxy mode", name) + } + return record.ID, nil + } + return c.CreateARecord(ctx, name, ip) +} + // DeleteRecord removes the DNS record with the given Cloudflare record ID. // Returns nil if the record does not exist (idempotent). func (c *Client) DeleteRecord(ctx context.Context, recordID string) error { diff --git a/internal/handler/admin_pool.go b/internal/handler/admin_pool.go new file mode 100644 index 0000000..8af8cb4 --- /dev/null +++ b/internal/handler/admin_pool.go @@ -0,0 +1,515 @@ +package handler + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "net/http" + "strconv" + "strings" + "time" + + "github.com/gin-gonic/gin" + "gorm.io/gorm" + + "github.com/ic3software/vtafarm-api/internal/connection" + "github.com/ic3software/vtafarm-api/internal/k8s" + "github.com/ic3software/vtafarm-api/internal/middleware" + "github.com/ic3software/vtafarm-api/internal/model" + poolstore "github.com/ic3software/vtafarm-api/internal/pool" + "github.com/ic3software/vtafarm-api/internal/resourceprofile" + "github.com/ic3software/vtafarm-api/internal/setup" +) + +type PoolHandler struct { + db *gorm.DB + setup *SetupHandler + dashboard *DashboardHandler + mobile *MobileConnectionHandler + runner *poolstore.Runner +} + +func NewPoolHandler( + db *gorm.DB, + setupHandler *SetupHandler, + dashboard *DashboardHandler, + mobile *MobileConnectionHandler, +) *PoolHandler { + h := &PoolHandler{db: db, setup: setupHandler, dashboard: dashboard, mobile: mobile} + var build func(context.Context, uint) error + if setupHandler != nil && setupHandler.orch != nil { + build = setupHandler.orch.BuildPool + } + h.runner = poolstore.NewRunner(db, h.prepareSession, build, h.cleanupSession) + return h +} + +func (h *PoolHandler) Start(ctx context.Context) { + h.runner.Start(ctx) +} + +type createPoolBatchRequest struct { + Label string `json:"label"` + Count int `json:"count"` + VtaImage string `json:"vta_image"` +} + +type poolCounts struct { + Total int `json:"total"` + Creating int `json:"creating"` + Waiting int `json:"waiting"` + Claiming int `json:"claiming"` + Activating int `json:"activating"` + Running int `json:"running"` + Failed int `json:"failed"` + Deleting int `json:"deleting"` +} + +type poolSessionItem struct { + SessionID uint `json:"session_id"` + UserID uint `json:"user_id"` + VtaName string `json:"vta_name"` + Status string `json:"status"` + ErrorMsg string `json:"error_msg,omitempty"` + URL string `json:"url"` + Claimed bool `json:"claimed"` + Cleanable bool `json:"cleanable"` + ClaimState string `json:"claim_state,omitempty"` +} + +type poolBatchView struct { + ID uint `json:"id"` + Label string `json:"label"` + RequestedCount int `json:"requested_count"` + VtaImage string `json:"vta_image,omitempty"` + CreatedAt time.Time `json:"created_at"` + Counts poolCounts `json:"counts"` + Sessions []poolSessionItem `json:"sessions,omitempty"` +} + +type poolClaimView struct { + BatchID uint `json:"batch_id"` + SessionID uint `json:"session_id"` + VtaName string `json:"vta_name"` + Enabled bool `json:"enabled"` + ServerTime time.Time `json:"server_time"` + Connection *mobileView `json:"connection"` +} + +// CreateBatch — POST /api/v1/admin/pool-batches. +func (h *PoolHandler) CreateBatch(c *gin.Context) { + if h.setup == nil || h.setup.cf == nil || h.setup.k8s == nil || h.setup.orch == nil || + h.setup.didHosting == nil || h.dashboard == nil || h.mobile == nil || !h.mobile.enabled { + c.JSON(http.StatusServiceUnavailable, gin.H{"error": "pooled VTA creation is not fully configured"}) + return + } + if h.setup.ingressIP == "" || h.setup.clusterDomain == "" { + c.JSON(http.StatusUnprocessableEntity, gin.H{"error": "cluster not configured: CLUSTER_INGRESS_IP and CLUSTER_DOMAIN must be set"}) + return + } + var request createPoolBatchRequest + if err := c.ShouldBindJSON(&request); err != nil { + c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + request.Label = strings.TrimSpace(request.Label) + request.VtaImage = strings.TrimSpace(request.VtaImage) + key := strings.TrimSpace(c.GetHeader("Idempotency-Key")) + if request.Label == "" || len(request.Label) > 120 { + c.JSON(http.StatusBadRequest, gin.H{"error": "label must be between 1 and 120 characters"}) + return + } + if request.Count < 1 { + c.JSON(http.StatusBadRequest, gin.H{"error": "count must be at least 1"}) + return + } + if request.VtaImage == "" { + c.JSON(http.StatusBadRequest, gin.H{"error": "vta_image is required"}) + return + } + if key == "" || len(key) > 128 { + c.JSON(http.StatusBadRequest, gin.H{"error": "Idempotency-Key must be between 1 and 128 characters"}) + return + } + + infra, _, reason, detail := h.setup.resolvePlatformInfra() + if reason != "" { + c.JSON(http.StatusServiceUnavailable, gin.H{"error": detail, "reason": reason}) + return + } + profiles, err := resourceprofile.Load(c.Request.Context(), h.db) + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": "could not load resource defaults"}) + return + } + profile := profiles[k8s.ComponentVTA] + digestBody, _ := json.Marshal(request) + digestBytes := sha256.Sum256(digestBody) + adminID := c.MustGet(middleware.ContextUserID).(uint) + batch, created, err := poolstore.Admit(c.Request.Context(), h.db, poolstore.Admission{ + AdminID: adminID, + Key: key, + Digest: hex.EncodeToString(digestBytes[:]), + Label: request.Label, + Count: request.Count, + Available: h.dashboard.AvailableVTAOnly, + Session: model.SetupSession{ + Mode: model.ModeVtaOnly, + DomainType: model.DomainManaged, + Domain: h.setup.clusterDomain, + Subdomain: setup.EnvPrefix(h.setup.appEnv) + "vta-", + MediatorDid: infra.MediatorDid, + DidHostingServerURL: strings.TrimSuffix(infra.ServerURL, "/"), + DidHostingControlURL: infra.ControlURL, + DIDHostingDid: infra.DaemonDid, + ConnectionSource: model.ConnectionPlatform, + VtaImage: request.VtaImage, + Portable: true, + PreRotationCount: 1, + }, + MemoryRequest: profile.MemoryRequest, + MemoryLimit: profile.MemoryLimit, + }) + if err != nil { + var capacityErr *poolstore.CapacityError + switch { + case errors.As(err, &capacityErr): + c.JSON(http.StatusUnprocessableEntity, gin.H{ + "error": fmt.Sprintf("the cluster currently has capacity for %d additional VTA-only sessions", capacityErr.Available), + "reason": "insufficient_capacity", "available": capacityErr.Available, + }) + case errors.Is(err, poolstore.ErrConflict): + c.JSON(http.StatusConflict, gin.H{"error": "Idempotency-Key was already used with a different request"}) + default: + c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to create pool batch"}) + } + return + } + h.kickBatch(batch.ID) + view, err := h.batchView(c.Request.Context(), batch.ID, true) + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": "batch was created but could not be loaded"}) + return + } + status := http.StatusOK + if created { + status = http.StatusCreated + } + c.JSON(status, view) +} + +func (h *PoolHandler) ListBatches(c *gin.Context) { + var batches []model.PoolBatch + if err := h.db.WithContext(c.Request.Context()).Order("id DESC").Find(&batches).Error; err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to list pool batches"}) + return + } + views := make([]poolBatchView, 0, len(batches)) + for _, batch := range batches { + view, err := h.batchView(c.Request.Context(), batch.ID, false) + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to load pool batches"}) + return + } + views = append(views, *view) + } + c.JSON(http.StatusOK, views) +} + +func (h *PoolHandler) GetBatch(c *gin.Context) { + batchID, ok := positiveID(c.Param("id")) + if !ok { + c.JSON(http.StatusBadRequest, gin.H{"error": "invalid pool batch ID"}) + return + } + view, err := h.batchView(c.Request.Context(), batchID, true) + if errors.Is(err, gorm.ErrRecordNotFound) { + c.JSON(http.StatusNotFound, gin.H{"error": "pool batch not found"}) + return + } + if err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to load pool batch"}) + return + } + c.JSON(http.StatusOK, view) +} + +func (h *PoolHandler) Reserve(c *gin.Context) { + if h.mobile == nil || !h.mobile.enabled { + c.JSON(http.StatusServiceUnavailable, gin.H{"error": "VTA claim is not available", "reason": "vta_claim_disabled"}) + return + } + batchID, ok := positiveID(c.Param("id")) + if !ok { + c.JSON(http.StatusBadRequest, gin.H{"error": "invalid pool batch ID"}) + return + } + var exists int64 + if err := h.db.Model(&model.PoolBatch{}).Where("id = ?", batchID).Count(&exists).Error; err != nil { + c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to load pool batch"}) + return + } + if exists == 0 { + c.JSON(http.StatusNotFound, gin.H{"error": "pool batch not found"}) + return + } + claim, err := poolstore.Reserve(c.Request.Context(), h.db, batchID) + if err != nil { + if errors.Is(err, poolstore.ErrEmpty) { + c.JSON(http.StatusConflict, gin.H{"error": "no waiting VTA is available in this batch", "reason": "pool_empty"}) + return + } + writeConnectionError(c, err) + return + } + h.writeClaim(c, claim.ID) +} + +func (h *PoolHandler) GetClaim(c *gin.Context) { + h.writeClaim(c, c.Param("request_id")) +} + +func (h *PoolHandler) ConfirmClaim(c *gin.Context) { + claim, err := poolstore.Confirm(c.Request.Context(), h.db, c.Param("request_id")) + if err != nil { + writeConnectionError(c, err) + return + } + if h.setup != nil && h.setup.orch != nil { + h.setup.orch.KickProvision(claim.SessionID) + } + h.writeClaim(c, claim.ID) +} + +func (h *PoolHandler) RejectClaim(c *gin.Context) { + requestID := c.Param("request_id") + if err := poolstore.Reject(c.Request.Context(), h.db, requestID); err != nil { + writeConnectionError(c, err) + return + } + h.writeClaim(c, requestID) +} + +func (h *PoolHandler) Cleanup(c *gin.Context) { + var request struct { + SessionIDs []uint `json:"session_ids"` + } + if err := c.ShouldBindJSON(&request); err != nil { + c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()}) + return + } + if err := poolstore.QueueCleanup(c.Request.Context(), h.db, request.SessionIDs); err != nil { + if errors.Is(err, poolstore.ErrConflict) || errors.Is(err, connection.ErrConflict) { + c.JSON(http.StatusConflict, gin.H{"error": "one or more sessions are claimed, active, or no longer cleanable"}) + return + } + writeConnectionError(c, err) + return + } + for _, sessionID := range request.SessionIDs { + h.runner.Kick(sessionID) + } + c.JSON(http.StatusAccepted, gin.H{"session_ids": request.SessionIDs, "status": "deleting"}) +} + +func (h *PoolHandler) Retry(c *gin.Context) { + sessionID, ok := positiveID(c.Param("id")) + if !ok { + c.JSON(http.StatusBadRequest, gin.H{"error": "invalid pool session ID"}) + return + } + if err := poolstore.RetryBuild(c.Request.Context(), h.db, sessionID); err != nil { + if errors.Is(err, poolstore.ErrConflict) { + c.JSON(http.StatusConflict, gin.H{"error": "only an unclaimed failed pool session can be retried"}) + return + } + writeConnectionError(c, err) + return + } + h.runner.Kick(sessionID) + c.JSON(http.StatusAccepted, gin.H{"session_id": sessionID, "status": model.StatusPoolCreating}) +} + +func (h *PoolHandler) batchView(ctx context.Context, batchID uint, includeSessions bool) (*poolBatchView, error) { + var batch model.PoolBatch + if err := h.db.WithContext(ctx).First(&batch, batchID).Error; err != nil { + return nil, err + } + var sessions []model.SetupSession + if err := h.db.WithContext(ctx).Table("setup_sessions s").Select("s.*"). + Joins("JOIN pool_sessions ps ON ps.session_id = s.id"). + Where("ps.batch_id = ?", batchID).Order("s.id").Find(&sessions).Error; err != nil { + return nil, err + } + latest := make(map[uint]model.MobileConnection, len(sessions)) + if len(sessions) > 0 { + ids := make([]uint, len(sessions)) + for i := range sessions { + ids[i] = sessions[i].ID + } + now, err := connection.Now(h.db.WithContext(ctx)) + if err != nil { + return nil, err + } + if err := h.db.WithContext(ctx).Model(&model.MobileConnection{}). + Where("session_id IN ? AND status = 'open' AND expires_at <= ?", ids, now). + Update("status", "lapsed").Error; err != nil { + return nil, err + } + if err := h.db.WithContext(ctx).Model(&model.MobileConnection{}). + Where("session_id IN ? AND status = 'claimed' AND confirm_by <= ?", ids, now). + Updates(map[string]any{"status": "lapsed", "admin_did": ""}).Error; err != nil { + return nil, err + } + var connections []model.MobileConnection + if err := h.db.WithContext(ctx).Raw(` + SELECT DISTINCT ON (session_id) * + FROM mobile_connections + WHERE session_id IN ? + ORDER BY session_id, created_at DESC, id DESC`, ids).Scan(&connections).Error; err != nil { + return nil, err + } + for _, item := range connections { + latest[item.SessionID] = item + } + } + view := &poolBatchView{ + ID: batch.ID, Label: batch.Label, RequestedCount: batch.RequestedCount, + CreatedAt: batch.CreatedAt, Sessions: []poolSessionItem{}, + } + if len(sessions) > 0 { + view.VtaImage = sessions[0].VtaImage + } + for i := range sessions { + session := &sessions[i] + current, hasClaim := latest[session.ID] + item := poolSessionItem{ + SessionID: session.ID, UserID: session.UserID, VtaName: session.VtaName, + Status: session.Status, ErrorMsg: session.ErrorMsg, URL: session.PublicURL(), + } + if hasClaim { + item.ClaimState = current.Status + } + item.Claimed = session.AdminDid != "" || hasClaim && current.ConfirmedAt != nil + activeClaim := hasClaim && (current.Status == "open" || current.Status == "claimed" || current.Status == "confirmed" || current.Status == "provisioning") + item.Cleanable = !item.Claimed && !activeClaim && session.Status != "deleting" + view.Counts.Total++ + switch { + case session.Status == "deleting": + view.Counts.Deleting++ + case session.Status == "running": + view.Counts.Running++ + case session.Status == "failed": + view.Counts.Failed++ + case item.Claimed || session.Status == "provisioning": + view.Counts.Activating++ + case activeClaim: + view.Counts.Claiming++ + case session.Status == model.StatusPoolWaiting: + view.Counts.Waiting++ + default: + view.Counts.Creating++ + } + if includeSessions { + view.Sessions = append(view.Sessions, item) + } + } + return view, nil +} + +func (h *PoolHandler) writeClaim(c *gin.Context, requestID string) { + var claim model.MobileConnection + if err := h.db.WithContext(c.Request.Context()).First(&claim, "id = ?", requestID).Error; err != nil { + writeConnectionError(c, err) + return + } + var membership model.PoolSession + if err := h.db.WithContext(c.Request.Context()).First(&membership, "session_id = ?", claim.SessionID).Error; err != nil { + writeConnectionError(c, err) + return + } + var session model.SetupSession + if err := h.db.WithContext(c.Request.Context()).First(&session, claim.SessionID).Error; err != nil { + writeConnectionError(c, err) + return + } + if err := connection.Reconcile(c.Request.Context(), h.db, &claim, &session); err != nil { + writeConnectionError(c, err) + return + } + now, err := connection.Now(h.db.WithContext(c.Request.Context())) + if err != nil { + writeConnectionError(c, err) + return + } + c.JSON(http.StatusOK, poolClaimView{ + BatchID: membership.BatchID, SessionID: session.ID, VtaName: session.VtaName, + Enabled: h.mobile != nil && h.mobile.enabled, ServerTime: now, + Connection: h.mobile.view(&claim), + }) +} + +func (h *PoolHandler) kickBatch(batchID uint) { + var sessionIDs []uint + if err := h.db.Model(&model.PoolSession{}).Where("batch_id = ?", batchID). + Pluck("session_id", &sessionIDs).Error; err != nil { + return + } + for _, sessionID := range sessionIDs { + h.runner.Kick(sessionID) + } +} + +func (h *PoolHandler) prepareSession(ctx context.Context, sessionID uint) error { + if h.setup == nil || h.setup.cf == nil { + return errors.New("cloudflare is not configured") + } + var session model.SetupSession + if err := h.db.WithContext(ctx).First(&session, sessionID).Error; err != nil { + return err + } + if !poolstore.IsSession(h.db.WithContext(ctx), sessionID) || session.Status == "deleting" { + return poolstore.ErrConflict + } + if session.CFRecordID != "" { + return nil + } + recordID, err := h.setup.cf.EnsureARecord(ctx, session.FQDN(), h.setup.ingressIP) + if err != nil { + return fmt.Errorf("failed to create DNS record: %w", err) + } + updated := h.db.WithContext(ctx).Model(&model.SetupSession{}). + Where("id = ? AND cf_record_id = '' AND status <> 'deleting'", sessionID). + Update("cf_record_id", recordID) + if updated.Error != nil { + return updated.Error + } + if updated.RowsAffected == 0 { + var current model.SetupSession + if err := h.db.WithContext(ctx).Select("cf_record_id", "status").First(¤t, sessionID).Error; err != nil { + return err + } + if current.Status == "deleting" { + return poolstore.ErrConflict + } + if current.CFRecordID == "" { + return errors.New("DNS record ownership changed during pool preparation") + } + } + return nil +} + +func (h *PoolHandler) cleanupSession(ctx context.Context, sessionID uint) error { + if h.setup == nil { + return errors.New("setup cleanup is not configured") + } + return h.setup.cleanupPoolSession(ctx, sessionID) +} + +func positiveID(value string) (uint, bool) { + id, err := strconv.ParseUint(value, 10, 64) + return uint(id), err == nil && id > 0 +} diff --git a/internal/handler/admin_pool_test.go b/internal/handler/admin_pool_test.go new file mode 100644 index 0000000..1d5764d --- /dev/null +++ b/internal/handler/admin_pool_test.go @@ -0,0 +1,56 @@ +package handler + +import ( + "context" + "errors" + "testing" + + "github.com/ic3software/vtafarm-api/internal/model" + poolstore "github.com/ic3software/vtafarm-api/internal/pool" + "github.com/ic3software/vtafarm-api/internal/testutil" + "gorm.io/gorm" +) + +func TestCleanupPoolSessionDeletesDedicatedOwner(t *testing.T) { + db := testutil.Postgres(t) + admin := model.Admin{UniqueId: "pool-cleanup"} + if err := db.Create(&admin).Error; err != nil { + t.Fatal(err) + } + batch, _, err := poolstore.Admit(context.Background(), db, poolstore.Admission{ + AdminID: admin.ID, Key: "cleanup", Digest: "cleanup", Label: "Cleanup", Count: 1, + Available: func(context.Context) (int, error) { return 1, nil }, + Session: model.SetupSession{ + Mode: model.ModeVtaOnly, Domain: "example.com", Subdomain: "vta-", + VtaImage: "example/vta:test", DidHostingServerURL: "https://dids.example.com", + }, + MemoryRequest: "32Mi", MemoryLimit: "64Mi", + }) + if err != nil { + t.Fatal(err) + } + var membership model.PoolSession + if err := db.First(&membership, "batch_id = ?", batch.ID).Error; err != nil { + t.Fatal(err) + } + var session model.SetupSession + if err := db.First(&session, membership.SessionID).Error; err != nil { + t.Fatal(err) + } + if err := poolstore.QueueCleanup(context.Background(), db, []uint{session.ID}); err != nil { + t.Fatal(err) + } + handler := &SetupHandler{db: db} + if err := handler.cleanupPoolSession(context.Background(), session.ID); err != nil { + t.Fatal(err) + } + if err := db.First(&model.SetupSession{}, session.ID).Error; !errors.Is(err, gorm.ErrRecordNotFound) { + t.Fatalf("setup session lookup error = %v, want not found", err) + } + if err := db.First(&model.User{}, session.UserID).Error; !errors.Is(err, gorm.ErrRecordNotFound) { + t.Fatalf("dedicated user lookup error = %v, want not found", err) + } + if err := db.First(&model.PoolBatch{}, batch.ID).Error; err != nil { + t.Fatalf("batch audit row was removed: %v", err) + } +} diff --git a/internal/handler/setup.go b/internal/handler/setup.go index 565b0de..7c026e3 100644 --- a/internal/handler/setup.go +++ b/internal/handler/setup.go @@ -859,6 +859,14 @@ func (h *SetupHandler) teardownSession(c *gin.Context, session *model.SetupSessi // response. The user/admin delete handlers and load-test batch cleanup share it // so all three remove the same DNS, DID-hosting, Kubernetes and Vault state. func (h *SetupHandler) teardownVtaOnlySession(ctx context.Context, session *model.SetupSession) (int, error) { + var poolMembership int64 + if h.db.Migrator().HasTable(&model.PoolSession{}) { + if err := h.db.Model(&model.PoolSession{}).Where("session_id = ?", session.ID).Count(&poolMembership).Error; err != nil { + return http.StatusInternalServerError, errors.New("failed to check pool membership") + } + } + isPoolSession := poolMembership == 1 + if h.cf != nil && session.CFRecordID != "" { if err := h.cf.DeleteRecord(ctx, session.CFRecordID); err != nil { return http.StatusBadGateway, fmt.Errorf("failed to delete DNS record: %w", err) @@ -903,7 +911,38 @@ func (h *SetupHandler) teardownVtaOnlySession(ctx context.Context, session *mode h.orch.TeardownVaultSecrets(ctx, session.UserID, session.ID) } - if err := h.db.Delete(session).Error; err != nil { + if err := h.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error { + if isPoolSession { + if err := tx.Delete(&model.PoolSession{}, "session_id = ?", session.ID).Error; err != nil { + return err + } + } + if err := tx.Delete(session).Error; err != nil { + return err + } + if !isPoolSession { + return nil + } + var remaining int64 + if err := tx.Model(&model.SetupSession{}).Where("user_id = ?", session.UserID).Count(&remaining).Error; err != nil { + return err + } + if remaining != 0 { + return nil + } + var user model.User + if err := tx.First(&user, session.UserID).Error; err != nil { + return err + } + var passkeys int64 + if err := tx.Model(&model.UserPasskey{}).Where("user_id = ?", user.ID).Count(&passkeys).Error; err != nil { + return err + } + if user.Email == nil && passkeys == 0 && !user.FullstackAccess { + return tx.Delete(&user).Error + } + return nil + }); err != nil { return http.StatusInternalServerError, errors.New("failed to delete session") } @@ -921,6 +960,50 @@ func (h *SetupHandler) teardownVtaOnlySession(ctx context.Context, session *mode return http.StatusNoContent, nil } +// cleanupPoolSession finalizes a session that QueueCleanup already protected +// and moved to deleting. It is called by the durable pool runner and can be +// retried after a process restart. +func (h *SetupHandler) cleanupPoolSession(ctx context.Context, sessionID uint) error { + var session model.SetupSession + if err := h.db.WithContext(ctx).First(&session, sessionID).Error; err != nil { + if errors.Is(err, gorm.ErrRecordNotFound) { + return nil + } + return err + } + var membership int64 + if err := h.db.WithContext(ctx).Model(&model.PoolSession{}). + Where("session_id = ?", sessionID).Count(&membership).Error; err != nil { + return err + } + if membership == 0 { + return nil + } + if session.Status != "deleting" || session.AdminDid != "" { + return errors.New("pool session is no longer eligible for cleanup") + } + var confirmed int64 + if err := h.db.WithContext(ctx).Model(&model.MobileConnection{}). + Where("session_id = ? AND confirmed_at IS NOT NULL", sessionID).Count(&confirmed).Error; err != nil { + return err + } + if confirmed != 0 { + return errors.New("confirmed pool claim cannot be cleaned as unused inventory") + } + if h.orch != nil { + if err := h.orch.Cancel(sessionID); err != nil { + return err + } + } + release, err := h.acquireSessionMaintenance(sessionID) + if err != nil { + return err + } + defer release() + _, err = h.teardownVtaOnlySession(ctx, &session) + return err +} + // GET /api/v1/setup/:id/logs func (h *SetupHandler) Logs(c *gin.Context) { publicID := c.Param("id") diff --git a/internal/pool/runner.go b/internal/pool/runner.go new file mode 100644 index 0000000..3b79ceb --- /dev/null +++ b/internal/pool/runner.go @@ -0,0 +1,160 @@ +package pool + +import ( + "context" + "database/sql/driver" + "errors" + "log" + "time" + + "github.com/ic3software/vtafarm-api/internal/model" + "gorm.io/gorm" +) + +const ( + workerCount = 4 + reconcileInterval = 10 * time.Second + workerLockClass = 565445 +) + +type Runner struct { + db *gorm.DB + prepare func(context.Context, uint) error + build func(context.Context, uint) error + cleanup func(context.Context, uint) error + queue chan uint +} + +func NewRunner( + db *gorm.DB, + prepare func(context.Context, uint) error, + build func(context.Context, uint) error, + cleanup func(context.Context, uint) error, +) *Runner { + return &Runner{db: db, prepare: prepare, build: build, cleanup: cleanup, queue: make(chan uint, 256)} +} + +func (r *Runner) Start(ctx context.Context) { + for range workerCount { + go r.runWorker(ctx) + } + go func() { + ticker := time.NewTicker(reconcileInterval) + defer ticker.Stop() + for { + r.reconcile(ctx) + select { + case <-ctx.Done(): + return + case <-ticker.C: + } + } + }() +} + +func (r *Runner) Kick(sessionID uint) { + select { + case r.queue <- sessionID: + default: + // The periodic reconciliation loop is the durable fallback. + } +} + +func (r *Runner) reconcile(ctx context.Context) { + var sessionIDs []uint + err := r.db.WithContext(ctx).Table("pool_sessions ps"). + Select("ps.session_id"). + Joins("JOIN setup_sessions s ON s.id = ps.session_id"). + Where("s.status IN ?", []string{ + model.StatusPoolCreating, "vta_setup_running", model.StatusPoolPublishing, + model.StatusPoolPrepared, "deleting", + }).Order("ps.session_id").Limit(500).Scan(&sessionIDs).Error + if err != nil { + if ctx.Err() == nil { + log.Printf("[pool] reconcile failed: %v", err) + } + return + } + for _, sessionID := range sessionIDs { + r.Kick(sessionID) + } +} + +func (r *Runner) runWorker(ctx context.Context) { + for { + select { + case <-ctx.Done(): + return + case sessionID := <-r.queue: + r.runOne(ctx, sessionID) + } + } +} + +func (r *Runner) runOne(ctx context.Context, sessionID uint) { + pool, err := r.db.DB() + if err != nil { + return + } + conn, err := pool.Conn(ctx) + if err != nil { + return + } + defer conn.Close() + var locked bool + if err := conn.QueryRowContext(ctx, "SELECT pg_try_advisory_lock($1, $2)", workerLockClass, int64(sessionID)).Scan(&locked); err != nil || !locked { + return + } + defer func() { + releaseCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if _, err := conn.ExecContext(releaseCtx, "SELECT pg_advisory_unlock($1, $2)", workerLockClass, int64(sessionID)); err != nil { + _ = conn.Raw(func(any) error { return driver.ErrBadConn }) + } + }() + + var session model.SetupSession + if err := r.db.WithContext(ctx).First(&session, sessionID).Error; err != nil { + return + } + if session.Status == "deleting" { + if r.cleanup != nil { + if err := r.cleanup(ctx, sessionID); err != nil && ctx.Err() == nil { + r.recordCleanupError(sessionID, err) + } + } + return + } + if session.Status == model.StatusPoolWaiting || session.Status == "running" || session.Status == "failed" { + return + } + if r.prepare == nil || r.build == nil { + r.recordBuildError(sessionID, errors.New("pool worker is not configured")) + return + } + if err := r.prepare(ctx, sessionID); err != nil { + if ctx.Err() == nil { + r.recordBuildError(sessionID, err) + } + return + } + if err := r.build(ctx, sessionID); err != nil && ctx.Err() == nil { + r.recordBuildError(sessionID, err) + } +} + +func (r *Runner) recordBuildError(sessionID uint, err error) { + if updateErr := r.db.Model(&model.SetupSession{}). + Where("id = ? AND status <> 'deleting'", sessionID). + Updates(map[string]any{"status": "failed", "error_msg": err.Error()}).Error; updateErr != nil { + log.Printf("[pool] session %d: failed to record build error: %v", sessionID, updateErr) + } +} + +func (r *Runner) recordCleanupError(sessionID uint, err error) { + if updateErr := r.db.Model(&model.SetupSession{}). + Where("id = ? AND status = 'deleting'", sessionID). + Update("error_msg", err.Error()).Error; updateErr != nil { + log.Printf("[pool] session %d: failed to record cleanup error: %v", sessionID, updateErr) + } +} diff --git a/internal/pool/runner_test.go b/internal/pool/runner_test.go new file mode 100644 index 0000000..16b1f24 --- /dev/null +++ b/internal/pool/runner_test.go @@ -0,0 +1,49 @@ +package pool + +import ( + "context" + "testing" + + "github.com/ic3software/vtafarm-api/internal/model" +) + +func TestRunnerDispatchesDurablePoolWork(t *testing.T) { + db := poolDatabase(t) + batch, _, err := admission(t, db, 1, 1) + if err != nil { + t.Fatal(err) + } + var membership model.PoolSession + if err := db.First(&membership, "batch_id = ?", batch.ID).Error; err != nil { + t.Fatal(err) + } + prepared, built, cleaned := 0, 0, 0 + runner := NewRunner(db, + func(context.Context, uint) error { + prepared++ + return nil + }, + func(_ context.Context, sessionID uint) error { + built++ + return db.Model(&model.SetupSession{}).Where("id = ?", sessionID). + Update("status", model.StatusPoolWaiting).Error + }, + func(context.Context, uint) error { + cleaned++ + return nil + }, + ) + + runner.runOne(context.Background(), membership.SessionID) + if prepared != 1 || built != 1 || cleaned != 0 { + t.Fatalf("build dispatch counts = prepare:%d build:%d cleanup:%d", prepared, built, cleaned) + } + if err := db.Model(&model.SetupSession{}).Where("id = ?", membership.SessionID). + Update("status", "deleting").Error; err != nil { + t.Fatal(err) + } + runner.runOne(context.Background(), membership.SessionID) + if cleaned != 1 { + t.Fatalf("cleanup calls = %d, want 1", cleaned) + } +} diff --git a/internal/pool/store.go b/internal/pool/store.go index 9647a16..4d576a5 100644 --- a/internal/pool/store.go +++ b/internal/pool/store.go @@ -19,6 +19,18 @@ var ( ErrCapacity = errors.New("pool inventory limit reached") ) +type CapacityError struct { + Available int +} + +func (e *CapacityError) Error() string { + return ErrCapacity.Error() +} + +func (e *CapacityError) Unwrap() error { + return ErrCapacity +} + // All pool mutations use one transaction lock so two operators cannot select // the same waiting session. func lock(tx *gorm.DB) error { @@ -69,7 +81,7 @@ func Admit(ctx context.Context, db *gorm.DB, input Admission) (*model.PoolBatch, return err } if input.Count > available { - return ErrCapacity + return &CapacityError{Available: max(available, 0)} } batch = &model.PoolBatch{ CreatedBy: input.AdminID, Label: input.Label, RequestedCount: input.Count, @@ -151,6 +163,15 @@ func Reserve(ctx context.Context, db *gorm.DB, batchID uint) (*model.MobileConne Select("s.*"). Joins("JOIN pool_sessions ps ON ps.session_id = s.id"). Where("ps.batch_id = ? AND s.status = ? AND s.admin_did = ''", batchID, model.StatusPoolWaiting). + Where(`NOT EXISTS ( + SELECT 1 FROM mobile_connections mc + WHERE mc.session_id = s.id + AND ( + (mc.status = 'open' AND mc.expires_at > clock_timestamp()) + OR (mc.status = 'claimed' AND mc.confirm_by > clock_timestamp()) + OR mc.status IN ('confirmed', 'provisioning') + ) + )`). Order("s.id").Find(&candidates).Error; err != nil { return err } @@ -168,7 +189,7 @@ func Reserve(ctx context.Context, db *gorm.DB, batchID uint) (*model.MobileConne } mobile, err := connection.ChangePool(ctx, tx, session.ID) if err != nil { - if errors.Is(err, connection.ErrConflict) { + if errors.Is(err, connection.ErrConflict) || errors.Is(err, connection.ErrCooldown) { continue } return err @@ -232,6 +253,40 @@ func IsSession(db *gorm.DB, sessionID uint) bool { return db.Model(&model.PoolSession{}).Where("session_id = ?", sessionID).Count(&count).Error == nil && count == 1 } +// RetryBuild returns an unclaimed failed member to the durable build queue. +func RetryBuild(ctx context.Context, db *gorm.DB, sessionID uint) error { + return db.WithContext(ctx).Transaction(func(tx *gorm.DB) error { + if err := lock(tx); err != nil { + return err + } + if err := tx.Exec("SELECT pg_advisory_xact_lock(?, ?)", workerLockClass, int64(sessionID)).Error; err != nil { + return err + } + if !IsSession(tx, sessionID) { + return ErrConflict + } + var session model.SetupSession + if err := tx.Clauses(clause.Locking{Strength: "UPDATE"}).First(&session, sessionID).Error; err != nil { + return err + } + if session.Status != "failed" || session.AdminDid != "" { + return ErrConflict + } + var protected int64 + if err := tx.Model(&model.MobileConnection{}). + Where("session_id = ? AND confirmed_at IS NOT NULL", sessionID). + Count(&protected).Error; err != nil { + return err + } + if protected != 0 { + return ErrConflict + } + return tx.Model(&session).Updates(map[string]any{ + "status": model.StatusPoolCreating, "error_msg": "", "failed_stage": "", + }).Error + }) +} + // QueueCleanup accepts setup-session IDs. Confirmed claims remain protected // even if activation failed and the setup session is no longer running. func QueueCleanup(ctx context.Context, db *gorm.DB, sessionIDs []uint) error { @@ -245,6 +300,11 @@ func QueueCleanup(ctx context.Context, db *gorm.DB, sessionIDs []uint) error { return err } for _, sessionID := range sessionIDs { + // Wait for preparation/build to finish before changing the session to + // deleting, otherwise a late build status update could revive it. + if err := tx.Exec("SELECT pg_advisory_xact_lock(?, ?)", workerLockClass, int64(sessionID)).Error; err != nil { + return err + } var membership model.PoolSession if err := tx.First(&membership, "session_id = ?", sessionID).Error; err != nil { return err @@ -256,6 +316,20 @@ func QueueCleanup(ctx context.Context, db *gorm.DB, sessionIDs []uint) error { if session.AdminDid != "" { return ErrConflict } + now, err := connection.Now(tx) + if err != nil { + return err + } + if err := tx.Model(&model.MobileConnection{}). + Where("session_id = ? AND status = 'open' AND expires_at <= ?", sessionID, now). + Update("status", "lapsed").Error; err != nil { + return err + } + if err := tx.Model(&model.MobileConnection{}). + Where("session_id = ? AND status = 'claimed' AND confirm_by <= ?", sessionID, now). + Updates(map[string]any{"status": "lapsed", "admin_did": ""}).Error; err != nil { + return err + } var claims int64 if err := tx.Model(&model.MobileConnection{}). Where("session_id = ? AND (confirmed_at IS NOT NULL OR status IN ?)", sessionID, []string{"open", "claimed", "confirmed", "provisioning", "released"}). diff --git a/internal/pool/store_test.go b/internal/pool/store_test.go index 645d6d0..ce37e93 100644 --- a/internal/pool/store_test.go +++ b/internal/pool/store_test.go @@ -4,6 +4,7 @@ import ( "context" "errors" "testing" + "time" "github.com/ic3software/vtafarm-api/internal/connection" "github.com/ic3software/vtafarm-api/internal/didkey" @@ -130,3 +131,103 @@ func TestConfirmedFailureCannotReturnToInventory(t *testing.T) { t.Fatalf("QueueCleanup() error = %v, want ErrConflict", err) } } + +func TestReserveDoesNotReuseActiveClaim(t *testing.T) { + db := poolDatabase(t) + batch, _, err := admission(t, db, 2, 2) + if err != nil { + t.Fatal(err) + } + if err := db.Model(&model.SetupSession{}). + Where("id IN (SELECT session_id FROM pool_sessions WHERE batch_id = ?)", batch.ID). + Updates(map[string]any{"status": model.StatusPoolWaiting, "vta_did": "did:webvh:example.com:test"}).Error; err != nil { + t.Fatal(err) + } + first, err := Reserve(context.Background(), db, batch.ID) + if err != nil { + t.Fatal(err) + } + second, err := Reserve(context.Background(), db, batch.ID) + if err != nil { + t.Fatal(err) + } + if first.SessionID == second.SessionID { + t.Fatalf("two active claims used session %d", first.SessionID) + } +} + +func TestExpiredClaimCanBeReplacedAndCleaned(t *testing.T) { + db := poolDatabase(t) + batch, _, err := admission(t, db, 1, 1) + if err != nil { + t.Fatal(err) + } + var membership model.PoolSession + if err := db.First(&membership, "batch_id = ?", batch.ID).Error; err != nil { + t.Fatal(err) + } + if err := db.Model(&model.SetupSession{}).Where("id = ?", membership.SessionID). + Updates(map[string]any{"status": model.StatusPoolWaiting, "vta_did": "did:webvh:example.com:test"}).Error; err != nil { + t.Fatal(err) + } + first, err := Reserve(context.Background(), db, batch.ID) + if err != nil { + t.Fatal(err) + } + past := time.Now().Add(-10 * time.Minute) + if err := db.Model(first).Updates(map[string]any{"created_at": past, "expires_at": past}).Error; err != nil { + t.Fatal(err) + } + second, err := Reserve(context.Background(), db, batch.ID) + if err != nil { + t.Fatal(err) + } + if second.ID == first.ID || second.SessionID != first.SessionID { + t.Fatalf("replacement claim = %+v, first = %+v", second, first) + } + if err := db.Model(second).Updates(map[string]any{"created_at": past, "expires_at": past}).Error; err != nil { + t.Fatal(err) + } + if err := QueueCleanup(context.Background(), db, []uint{membership.SessionID}); err != nil { + t.Fatalf("QueueCleanup(expired claim) error = %v", err) + } + var session model.SetupSession + if err := db.First(&session, membership.SessionID).Error; err != nil { + t.Fatal(err) + } + if session.Status != "deleting" { + t.Fatalf("status = %q, want deleting", session.Status) + } +} + +func TestRetryBuildProtectsClaimedSession(t *testing.T) { + db := poolDatabase(t) + batch, _, err := admission(t, db, 1, 1) + if err != nil { + t.Fatal(err) + } + var membership model.PoolSession + if err := db.First(&membership, "batch_id = ?", batch.ID).Error; err != nil { + t.Fatal(err) + } + if err := db.Model(&model.SetupSession{}).Where("id = ?", membership.SessionID). + Updates(map[string]any{"status": "failed", "error_msg": "temporary"}).Error; err != nil { + t.Fatal(err) + } + if err := RetryBuild(context.Background(), db, membership.SessionID); err != nil { + t.Fatal(err) + } + var session model.SetupSession + if err := db.First(&session, membership.SessionID).Error; err != nil { + t.Fatal(err) + } + if session.Status != model.StatusPoolCreating || session.ErrorMsg != "" { + t.Fatalf("retried session = %+v", session) + } + if err := db.Model(&session).Updates(map[string]any{"status": "failed", "admin_did": "did:key:z6Mktest"}).Error; err != nil { + t.Fatal(err) + } + if err := RetryBuild(context.Background(), db, membership.SessionID); !errors.Is(err, ErrConflict) { + t.Fatalf("RetryBuild(claimed) error = %v, want ErrConflict", err) + } +} diff --git a/internal/router/router.go b/internal/router/router.go index 3ea4589..e85095e 100644 --- a/internal/router/router.go +++ b/internal/router/router.go @@ -53,7 +53,7 @@ func Setup( r.Use(cors.New(cors.Config{ AllowOrigins: allowOrigins, AllowMethods: []string{"GET", "POST", "PUT", "DELETE", "OPTIONS"}, - AllowHeaders: []string{"Authorization", "Content-Type"}, + AllowHeaders: []string{"Authorization", "Content-Type", "Idempotency-Key"}, ExposeHeaders: []string{"Content-Length", "Content-Disposition"}, AllowCredentials: true, MaxAge: 12 * time.Hour, @@ -133,6 +133,10 @@ func Setup( // Image upgrades — shared by the admin batch routes and the user // self-service routes below; background runner, DB-backed queue. uph := handler.NewUpgradeHandler(db, upgradeRunner, ghcrClient, mediatorGhcrClient, didsGhcrClient, vtcGhcrClient) + mobile := handler.NewMobileConnectionHandler(db, orch, sh.KickAdditionalMobile, cfg.VTAClaim, trustTask.Enabled()) + dashH := handler.NewDashboardHandler(db, k8sClient) + poolH := handler.NewPoolHandler(db, sh, dashH, mobile) + poolH.Start(workerCtx) // Admin routes — cookie: vtafarm_admin adminAuth := v1.Group("", @@ -167,6 +171,16 @@ func Setup( adminAuth.GET("/admin/load-tests/:id", sh.AdminGetLoadTest) adminAuth.POST("/admin/load-tests/:id/check", sh.AdminCheckLoadTest) adminAuth.DELETE("/admin/load-tests/:id", sh.AdminDeleteLoadTest) + adminAuth.POST("/admin/pool-batches", poolH.CreateBatch) + adminAuth.GET("/admin/pool-batches", poolH.ListBatches) + adminAuth.GET("/admin/pool-batches/:id", poolH.GetBatch) + adminAuth.POST("/admin/pool-batches/:id/claims", poolH.Reserve) + adminAuth.POST("/admin/pool-sessions/cleanup", poolH.Cleanup) + adminAuth.POST("/admin/pool-sessions/:id/retry", poolH.Retry) + poolClaims := adminAuth.Group("/admin/pool-claims", middleware.NoStore(), middleware.RateLimit(120, time.Minute)) + poolClaims.GET("/:request_id", poolH.GetClaim) + poolClaims.POST("/:request_id/confirm", poolH.ConfirmClaim) + poolClaims.POST("/:request_id/reject", poolH.RejectClaim) // Same teardown as the user-facing DELETE /setup/:id, but reaches any // user's session rather than only the caller's. Deleting the platform // stack additionally requires {"confirm": "