release: v0.19.9 with bounded owner recovery and retained container stop

docs/architecture.md

@@ -62,6 +62,11 @@ that has no serialization or compatibility meaning. An exception at a public
 boundary must be explicit and documented; convenience alone is not an
 exception.
 
+The local operator container-stop receipt has a separate, narrowly scoped
+[Docker evidence exception](container-stop.md#contract-exception-and-lifecycle),
+including its fixed shape, fresh-comparison lifecycle and refusal on incompatible
+evidence. It does not extend the process-group registry exception to other state.
+
 Per-consumer module update evidence and verdicts use `codefly.update.v0`.
 See [module updates](module-updates.md) for the offline comparison, authenticated
 release artifacts, usage completeness requirements, and delivery ownership.

docs/container-stop.md

@@ -0,0 +1,127 @@
+# Explicit container stop with retained data
+
+`runners/dockerrun` supplies a bounded operator-selected stop operation, separate
+from workspace adoption, process-group recovery, ledger reconciliation and
+container destruction. CLI owns target selection and operator authorization.
+Core owns inspection, expectation comparison, delivery and observed disposition.
+Scope/name labels are comparison evidence, not permission to infer a parent
+service. A receipt grants no authority by itself.
+
+## API and receipt
+
+`PreviewContainerStop(ctx, fullID)` accepts only a complete lowercase 64-hex
+Docker ID. It never resolves a name or a shortened ID. The inspection must name
+that exact ID, include Created, Config, HostConfig and State, have
+`codefly.owner=true`, and agree between the actual container name and
+`codefly.name`. **AutoRemove is refused before stopping**: Docker could otherwise
+delete the object on exit. Missing HostConfig also refuses.
+
+The returned `ContainerStopExpectation` contains ID, Created, Name, every label,
+and every field of Docker's `container.MountPoint` records. The producer sorts
+mounts by their complete JSON representation and emits an explicit empty `[]`
+for an unmounted container. These are custody facts, not a hash of stored data
+or an invented ledger ownership claim. No environment variables or secret
+configuration bodies are included. Existing labels and mount paths are included
+verbatim, so receipts remain operator evidence and should be retained privately.
+
+Persist the receipt with the default `json.Encoder` or `json.Marshal`, and decode
+with `ParseContainerStopExpectation`. It refuses unknown fields, duplicate JSON
+keys, nulls, trailing documents and missing identity/owner/name/mount evidence.
+Receipt fields use their exact lowercase schema spelling; nested MountPoint
+fields use Docker's exact spelling (for example `RW`). Case aliases refuse,
+including when placed before or after a correctly spelled field. Arbitrary
+label keys remain case-sensitive data. Legal JSON escapes and whitespace remain
+accepted; byte-canonical JSON is not required.
+Missing or null mounts are not an empty mount list. The owner constant
+`MaxContainerStopExpectationBytes` bounds the whole receipt to 64 KiB, including
+the default Encoder newline. Preview, parser and direct execution validate the
+same canonical-output budget; oversized inspections refuse instead of emitting
+unusable receipts. CLI uses that constant and parser, not its own schema.
+
+## Contract exception and lifecycle
+
+Core explicitly chooses a narrow exception to the default protobuf contract rule
+in [architecture](architecture.md#contract-source-of-truth) for
+`ContainerStopExpectation` and its `ContainerStopResult` disposition. This is a
+local operator inspection/compare/stop receipt, including when the CLI saves it
+to a file. It is not covered implicitly by the process-group registry exception.
+
+The evidence owner is Docker: Core reuses the existing Docker inspection identity,
+label map and complete native `container.MountPoint` values, then compares them
+against a new inspection of the same immutable ID. Keeping those native mount
+values avoids a second translated mount model that could omit custody fields
+while still claiming complete comparison. The exception is bounded to this
+operation and these types; it does not authorize JSON agent contracts, remote
+recovery APIs, a new registry, or a general evidence format. Core owns the one
+strict parser; consumers import it and never reproduce the schema.
+
+The current unversioned receipt shape is fixed, not an extensible protocol.
+Preview and execution use the same reviewed Core implementation, and the saved
+receipt has only the lifetime of that operator-selected attempt and unchanged
+object. It is untrusted comparison input, never a reusable authorization grant.
+Execution always obtains current evidence independently; an unchanged stopped
+object may be rechecked, but changed or missing evidence requires a fresh preview
+and operator selection. An archived result records only the observed disposition,
+not continuing stopped state or permission for later effects.
+
+There is no cross-release migration or compatibility fallback. Unknown fields,
+case aliases, duplicates, missing required fields and nulls refuse rather than
+being discarded or filled in. After changing the executing implementation, take
+a fresh preview rather than relying on an older saved receipt. Any future change
+to this shape or its evidence meaning must use a separately versioned contract
+and entrypoint under the protobuf default, unless a new explicit owner exception
+is reviewed. Do not add optional fields or silently reinterpret this shape.
+Docker dependency changes affecting `MountPoint` must requalify complete field
+comparison and producer/parser roundtrip; they cannot inherit this exception as
+permission to accept an incomplete projection. This lifecycle policy is a caller
+requirement: the receipt carries no binary identity or version discriminator,
+and Core does not claim to detect which release wrote identical receipt bytes.
+
+## Conditional stop and disposition
+
+`StopContainer(ctx, expected)` independently inspects the same full ID, refuses
+AutoRemove or missing evidence, and compares every expected field before
+calling the existing Docker `ContainerStop` on that ID. Mount order is not
+significant; every mount field and label is. The stop has a three-second Docker
+grace period and an overall fifteen-second context bound. It never resolves the
+name again, claims a session ledger resource, changes labels, removes an object
+or volume, starts a container, or falls back to another operation.
+
+Readback must find the same ID and Created, unchanged labels/name/mounts, and a
+non-running, non-restarting, non-paused state. `ContainerStopResult` reports
+`Stopped`, `Retained` and `MountsUnchanged`; success requires a nil error and all
+three true. A repeat for the same unchanged stopped object succeeds without
+another stop. A missing object is an error, not proof it was retained. A
+same-name replacement is never selected by the old ID.
+
+Cancellation, timeout or a lost reply may follow a successful Docker effect.
+Such errors do not claim that nothing happened. Readback failure returns an
+error and only postconditions actually observed. Docker supplies no atomic
+compare-and-stop transaction: the full immutable ID prevents a replacement from
+being stopped, but an external actor can rename or restart the selected object
+between observations. Changed custody in readback refuses success. This API
+does not promise the object stays stopped after the observation.
+
+The intended CLI surface is `recover container FULL_ID --preview` and
+`recover container FULL_ID --expect FILE`. Execution checks that the positional
+ID equals the receipt ID and delegates to Core. It reports the owner result and
+requires all postconditions for exit zero. Ordinary `stop`/`clear` remain
+unchanged. Operational use requires a reviewed binary and fresh exact receipt;
+the API does not authorize a target through its name or scope.
+
+## Qualification and limits
+
+Maintained Docker tests use only their own created IDs. They prove refusal of
+changed ID/creation/name/scope/namespace/mount evidence, AutoRemove refusal while
+the container stays running, preserved writable-layer and named-volume content,
+same stopped-generation retry, survival of an authenticated sibling and an
+unattributed control, and refusal to reach a same-name replacement. Transport
+tests cover explicit empty mounts, bounds including Encoder newline, malformed
+receipts and unavailable inspection fields.
+
+Stopping an old container does not add a parent-service label. The producer's
+later exact-name reuse is a separate operation: its configuration fingerprint
+may cause recreation, and identical reuse leaves legacy labels unchanged.
+Qualify the corrected producer's retained-resource/data behavior independently
+before claiming migration or workspace adoption. A matching mount receipt alone
+does not prove application data contents or consistency.

docs/session-ledger.md

@@ -316,8 +316,8 @@ protobuf contract rule in [architecture](architecture.md#contract-source-of-trut
 Core retains the existing strict JSON storage contract so it can recover records
 from independently released owners without rewriting their birth evidence.
 The versioned namespace isolates incompatible readers. Consumers use the Core
-ownership API instead of implementing a JSON decoder; this exception does not
-introduce another public serialization surface or relax the protobuf default.
+ownership API instead of implementing a JSON decoder; the explicit recovery receipt described below is its only selection transport,
+and the exception does not relax the protobuf default elsewhere.
 
 The v1 wire format is unchanged. Older strict readers ignore the v2 directory;
 they cannot recover its groups and must upgrade before claiming scoped cleanup
@@ -334,8 +334,70 @@ identity nor signals the unverified process. Inspection, stale recovery and
 explicit signals share this refusal; an actual different start identity retains
 the existing reused-process handling.
 
+## Explicit orphan-generation recovery
+
+`runners/base.PreviewProcessGroupRecovery(ctx, pgid)` is a separate, read-only
+admission path for an explicitly operator-selected orphan registration. It is
+not workspace-scoped stop, startup reaping, or permission to include unknown
+workspaces. CLI owns selection, authorization and reporting; Core owns decoding,
+registration identity, birth authentication, delivery and conditional removal.
+The operation never assigns a workspace or rewrites stored boot evidence.
+
+The preview returns `ProcessGroupRecoveryExpectation`: namespace, SHA-256 of the
+exact registration bytes, PGID, recorded leader and owner identities, and the
+currently authenticated member identities. Recorded identities retain their
+original boot representation; the owner's PGID is zero because it is not stored.
+Members carry current boot/start identities for kernel-bound signal delivery.
+The receipt is untrusted input, not a credential. Preserve it with `json.Marshal`
+and read it with Core's `ParseProcessGroupRecoveryExpectation`, which refuses
+unknown fields, duplicate JSON keys, nulls, trailing values and missing birth
+proof. `MaxProcessGroupRecoveryExpectationBytes` limits a whole receipt to
+1 MiB, independently of the unchanged 16 KiB registration cap. Preview reserves
+one byte beyond compact `json.Marshal` output for the newline emitted by the
+CLI's default `json.Encoder.Encode`; unsupported previews fail instead of emitting
+an unusable receipt. Parsing enforces both this canonical-output budget and the
+inclusive input-byte limit, so extra formatting also counts toward the limit.
+Consumers use the exported owner limit when bounding file reads. This bounded
+local JSON receipt extends the registry exception above;
+it is not a new agent protocol or a parallel registry. Core reuses its existing
+`internal/wire` document checks and JSON decoding, not a consumer parser.
+
+`RecoverProcessGroup(ctx, expected)` locks both supported namespaces in a fixed
+order and independently authenticates the current registration. Missing or
+changed registration bytes, namespace ambiguity, changed expected births,
+incomplete proof, a live original owner, or uncertain inspection refuse before
+signals. The full initial member set must still match at admission; take a fresh
+preview if it changed. Each subsequent signal sweep rechecks the registration,
+owner absence and group authentication. Existing escalation can reach surviving
+same-generation descendants; this is not a frozen list of PIDs or authority over
+a replacement group. PID, boot and start evidence remains required.
+
+Only this explicit path accepts Darwin's historical `sec:usec` boot format, and
+only by exact equality to the kernel's current `kern.boottime`. A changed clock
+value refuses; no UUID conversion, approximate match, producer-version roster,
+or weakening of ordinary incompatible-boot refusal is permitted. Current
+kernel-bound delivery is reused unchanged: Darwin acquires PIDVersion before
+checking birth on every attempt, preserving safety across PID reuse and exec.
+The old reader's signal implementation is not used.
+
+Both namespace locks remain held through termination and conditional removal.
+The private file snapshot and expected bytes are rechecked before effects and
+removal; a replacement record is retained. Cooperating writers use these registry
+locks; this does not claim atomicity against a same-user actor deliberately
+bypassing the filesystem protocol. The result reports proven `GroupDead` and
+`RecordRemoved` postconditions. Success requires a nil error and both true.
+An error may follow partial termination; false means that postcondition was not
+established, not proof that the process is alive. A replacement or uncertain
+record is never reported removed. A nil `RemoveIfDeadContext` alone is not used
+as evidence of successful recovery. This operation deletes no container or data.
+
 ## Startup container recovery scope
 
+Explicit operator-selected stop without ledger adoption is documented in
+[container-stop.md](container-stop.md). It compares fresh custody evidence and
+stops a full immutable Docker ID while retaining the object and mounts. It does
+not fabricate an invocation label or modify ledger ownership.
+
 The legacy PID-based startup sweep requires a `codefly.recovery-scope`
 label before checking owner liveness. The label hashes canonical Codefly home,
 workspace path and resolved naming scope. A different home, workspace or scope

runners/base/owned_group_test.go

@@ -466,6 +466,11 @@ func TestAuthenticateOwnedProcessGroupLeaderlessDecision(t *testing.T) {
 // executable the one under test: until the exec chain settles the member is
 // still sh or env, whose environment darwin withholds, and every case would
 // take the unobservable branch no matter what the descendant carries.
+// The platform binary has no ready-file protocol. Wait for its executable and,
+// on Linux, its inherited credential before testing the leaderless decision.
+// /proc can expose sleep's executable while environ is still empty during exec.
+// This is fixture readiness, not a retry of the authentication under test; a
+// missing credential fails setup. Darwin deliberately withholds that evidence.
 func startLeaderlessOwnedGroup(t *testing.T, script string, usesHelper bool) *OwnedProcessGroup {
 	t.Helper()
 	cmd := exec.Command("sh", "-c", script)
@@ -495,10 +500,19 @@ func startLeaderlessOwnedGroup(t *testing.T, script string, usesHelper bool) *Ow
 			if member.pid == group.PGID() {
 				return false
 			}
+			if !usesHelper && filepath.Base(member.executable) != "sleep" {
+				return false
+			}
+			if !usesHelper && runtime.GOOS == "linux" {
+				value, observed, err := readProcessGroupAuthentication(member.pid)
+				if err != nil || !observed || value != group.record.Authentication {
+					return false
+				}
+			}
 		}
 		return true
 	}) {
-		t.Skip("could not observe a leaderless group holding the pgid")
+		t.Fatal("could not observe a ready leaderless group holding the pgid")
 	}
 	return group
 }

runners/base/pgid.go

@@ -4,6 +4,7 @@ import (
 	"bufio"
 	"context"
 	"crypto/rand"
+	"crypto/sha256"
 	"encoding/hex"
 	"encoding/json"
 	"errors"
@@ -156,7 +157,8 @@ type pgidRecordV2 struct {
 }
 
 type recordSnapshot struct {
-	info os.FileInfo
+	info   os.FileInfo
+	digest string
 }
 
 // TrackedProcessGroup is the identity-bearing handle returned for a process
@@ -764,7 +766,7 @@ func readPgidRecord(path string) (pgidRecord, recordSnapshot, error) {
 	if !sameRecordFile(before, after) || !sameRecordFile(after, current) {
 		return pgidRecord{}, recordSnapshot{}, errors.New("record changed while being read")
 	}
-	snapshot := recordSnapshot{info: current}
+	snapshot := recordSnapshot{info: current, digest: fmt.Sprintf("%x", sha256.Sum256(data))}
 	if len(data) > maxRecordSize {
 		return pgidRecord{}, snapshot, errors.New("record is too large")
 	}

runners/base/pgid_decision_test.go

@@ -39,11 +39,17 @@ func TestProcessGroupRegistryHelper(t *testing.T) {
 		defer signal.Stop(stopping)
 		writeTestFile(t, os.Getenv(processGroupReadyFileEnv), "ready")
 		<-stopping
+	case "recovery-replace-on-term":
+		stopping := armTerminationSignal()
+		defer signal.Stop(stopping)
+		writeTestFile(t, os.Getenv(processGroupReadyFileEnv), "ready")
+		<-stopping
+		rewriteRecord(t, recordPath(t, os.Getpid()), func(rec *pgidRecord) { rec.Authentication = strings.Repeat("0", groupAuthBytes*2) })
 	case "ignores-term":
 		signal.Ignore(syscall.SIGTERM)
 		writeTestFile(t, os.Getenv(processGroupReadyFileEnv), "ready")
 		select {}
-	case "owner", "stubborn-owner", "live-owner", "workspace-owner":
+	case "owner", "stubborn-owner", "live-owner", "workspace-owner", "recovery-replacement-owner":
 		var stopping chan os.Signal
 		if os.Getenv(processGroupRoleEnv) == "live-owner" {
 			stopping = armTerminationSignal()
@@ -53,6 +59,9 @@ func TestProcessGroupRegistryHelper(t *testing.T) {
 		if os.Getenv(processGroupRoleEnv) == "stubborn-owner" {
 			leaderRole = "ignores-term"
 		}
+		if os.Getenv(processGroupRoleEnv) == "recovery-replacement-owner" {
+			leaderRole = "recovery-replace-on-term"
+		}
 		leader := registryHelperCommand(leaderRole)
 		leader.Env = append(leader.Env, processGroupReadyFileEnv+"="+os.Getenv(processGroupReadyFileEnv))
 		if os.Getenv(processGroupRoleEnv) == "workspace-owner" {