[Bug]: Interrupted stop during update leaves orphaned pending-teardown receipts, causing subsequent ocx update runs to permanently fail-closed

[Vocllum]

Client or integration

CLI / Updater / Process Control

Area

Installation or packaging

Summary

When ocx update attempts to stop the running proxy before replacing packages, it records a shared teardown claim via claimPendingTeardown (~/.opencodex/pending-teardown-<nonce>.json). If the stop or drain sequence is interrupted (e.g. timeout, service kill, watchdog kickstart, or external SIGTERM/SIGHUP), the process exits before reaching clearPendingTeardown(nonce).

Because bin/ocx.mjs checks hasPendingTeardownIn(readdirSync, configDir()) without validating process liveness (isProcessAlive(ownerPid)), any subsequent run of ocx update immediately aborts with:

opencodex: a shared teardown from an earlier stop is still outstanding and needs manual review; aborting the update.
opencodex: confirm no proxy is running, run 'ocx restore', then remove the pending-teardown file in the opencodex home.

If multiple attempts are made while the service is unstable, each attempt claims a new receipt and fails to clean it up. In our environment, this caused 6 orphaned pending-teardown-*.json receipts to accumulate, wedging the updater until files were manually deleted.

Reproduction

1. Run opencodex as a managed service on macOS (gui/<uid>/com.opencodex.proxy).
2. Trigger an update via ocx update --tag latest while the proxy is under load or handling long requests.
3. Interrupt or kill the update worker during the stop phase before the final clearPendingTeardown() call.
4. The worker process (ownerPid) exits. The receipt pending-teardown-<nonce>.json remains in ~/.opencodex/.
5. Run ocx update again.
6. Observe:

opencodex v2.56.0 (installed via npm, tag latest)
Verified @bitkyc08/opencodex@2.57.0 integrity metadata sha512-lKEPsKq7DYSg2AhA6…
opencodex: a shared teardown from an earlier stop is still outstanding and needs manual review; aborting the update.
opencodex: confirm no proxy is running, run 'ocx restore', then remove the pending-teardown file in the opencodex home.

7. The updater refuses to proceed due to teardown-outstanding, even though the recorded ownerPid is dead and no proxy is running.

Version

2.56.0 -> 2.57.0

Operating system

macOS 27.0 (Darwin 27.0.0 arm64)

Provider and model

Not provider-specific

Logs or error output

opencodex: a shared teardown from an earlier stop is still outstanding and needs manual review; aborting the update.
opencodex: confirm no proxy is running, run 'ocx restore', then remove the pending-teardown file in the opencodex home.

In update-job.json:

{
  "currentVersion": "2.56.0",
  "latestVersion": "2.57.0",
  "exitCode": 1,
  "error": "update command failed (1)",
  "log": [
    "exit 1 · 505 bytes of output withheld (no recognized diagnostic fields)"
  ]
}

Orphaned receipt files in ~/.opencodex/:

pending-teardown-75280d7e4be191581141ca7863fbc7aa.json
pending-teardown-756b6d207bdfc07862a7969abb7b2c39.json
pending-teardown-a8d6f0e0c1d042ff281c49b7962265bb.json
pending-teardown-b13be77f97d4b87e0f0b6160279dc2b4.json
pending-teardown-b3bee9495dd4b13b3414c3c2013e1281.json
pending-teardown-f9f3728779188403ef2e9179778a0d04.json

Receipt payload example:

{
  "ownerPid": 90028,
  "nonce": "b3bee9495dd4b13b3414c3c2013e1281",
  "createdAt": "2026-09-17T08:52:45.393Z",
  "endpoint": { "hostname": "127.0.0.1", "port": 10100 },
  "endpointSource": "exact"
}

Root cause analysis

1. Lack of exit/signal rollback: In src/cli/index.ts, claimPendingTeardown drops a durable receipt on disk before stopping the proxy. If stopProxy or eviction fails or is aborted, there is no signal handler or try...finally block to prune the uncommitted claim.
2. Stat-only check in update launcher: hasPendingTeardownIn in bin/ocx.mjs only matches the filename pattern pending-teardown-*.json. It does not inspect the receipt payload or verify whether ownerPid is still alive via isProcessAlive(receipt.ownerPid).
3. Deadlock accumulation: When bin/ocx.mjs sees an existing receipt, it invokes ocx stop to settle it, which claims yet another receipt before verifying the inherited ones. An interrupted chain easily snowballs into multiple orphaned files.

Suggested fix

1. In bin/ocx.mjs / hasPendingTeardownIn: Parse existing receipts and check if ownerPid is still active. If isProcessAlive(ownerPid) is false and the target port is dead, the abandoned receipt should either be auto-recovered/cleared or safely ignored for the update gate.
2. In src/cli/index.ts: Wrap stopWithDeferral with cleanup hooks so that if the process terminates unexpectedly before shared teardown completes, the uncommitted receipt is retracted.

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[github-actions]

Issue reopened

The report now contains the information required by the automated check. Thanks for updating it.

[Vocllum]

Added the required Version and Operating System sections to the issue body.

[lidge-jun]

리뷰 · 우선순위 70 / 80

이 이슈는 지금 dev HEAD fa26404d7(패키지 2.58.0)에서 ocx update 중 stop/teardown이 끊기면 ~/.opencodex/pending-teardown-*.json 영수증이 남고, 이후 업데이트가 영구 fail-closed 되는 문제를 보고한다. 작성 환경에서는 ownerPid가 이미 죽은 영수증이 6개까지 쌓여 수동 삭제 전까지 teardown-outstanding로 막혔다. 버전 기록은 2.56.0→2.57.0, macOS launchd 서비스 경로다.

코드 경로를 현재 체크아웃에서 확인했다. bin/ocx.mjs는 hasPendingTeardownIn(readdirSync, configDir())로 파일 존재만 보고 stop을 강제한다. src/config/pending-teardown-names.mjs의 hasPendingTeardownIn은 이름 패턴(pending-teardown- + 32hex + .json 또는 .unreadable.json)만 검사하고, 페이로드의 ownerPid나 isProcessAlive는 보지 않는다. 주석도 「Presence is the whole test here — the launcher cannot parse it」라고 #3008 의도를 명시한다. 즉 런처가 죽은 PID를 자동으로 무시하지 않는 것은 우연 빠진 게 아니라, 예전에 singleton 파일명 불일치로 의무를 놓쳤던 사고를 막으려는 fail-closed 설계다.

src/cli/index.ts 쪽은 claimPendingTeardown으로 nonce를 남긴 뒤 stopWithDeferral을 타고, 성공 경로에서 clearPendingTeardown(nonce)를 부른다. 중단·시그널·타임아웃으로 clear 전에 죽으면 영수증이 남는다. 그 다음 ocx update가 다시 stop을 호출하면 또 claim이 생길 수 있어, 작성자가 말한 눈덩이 축적과 맞다. 수동 복구 문구(ocx restore 후 pending-teardown 삭제)도 현재 에러 메시지와 일치한다.

그래서 “죽은 ownerPid면 런처가 그냥 무시”는 #3008 안전 모델과 충돌할 수 있다. 더 안전한 방향은 (1) stop/recovery 경로에서 영수증을 파싱해 liveness·포트 프로브 후 clear/quarantine을 명시적 회복으로 수행하거나, (2) claim을 try/finally·시그널 훅으로 줄이고, (3) 런처의 존재 검사 자체는 fail-closed로 남기는 쪽이다. 포트가 살아 있는데 PID만 죽은 경우, PID만 보고 지우면 위험하다.

types.ts/config.ts 분할과 무관하다. 실사용 fail-closed라 우선순위는 높다. 다만 수정은 “편의 자동삭제”가 아니라 #3008 의무 모델을 깨지 않는 회복 경로여야 한다.

bin/ocx.mjs (hasPendingTeardown / teardownOutstanding) - 파일 존재만으로 update를 막는다. #3008 의도된 fail-closed다.
src/config/pending-teardown-names.mjs hasPendingTeardownIn - 이름 패턴만 보고 ownerPid를 읽지 않는다. 런처에 PID 파싱을 넣는 수정은 설계 변경이다.
src/cli/index.ts claimPendingTeardown / clearPendingTeardown / stopWithDeferral - clear 전 중단이 고아 영수증의 직접 원인.
suggested auto-ignore dead PID - 포트 liveness 없이 PID만 보면 위험. stop 경로의 명시적 회복이 더 맞다.
reproduction - 서비스 불안정 시 claim이 반복되면 영수증이 여러 개 쌓일 수 있다.

메인테이너의 판단이 필요한 지점

• #3008 fail-closed를 유지한 채 stop/restore에 dead-receipt 회복을 넣을지, 런처 존재 검사에 제한적 PID/포트 판별을 허용할지
• 시그널/finally로 uncommitted claim을 줄이는 작업을 같은 PR에 묶을지
• 운영 문서에 고아 영수증 수동 정리 절차를 더 드러낼지

너의 추천
버그로 받아 수정 PR을 받는다. 런처의 hasPendingTeardownIn을 단순히 죽은 PID ignore로 바꾸지 말고, ocx stop/recovery가 영수증을 파싱·프로브한 뒤 clear하는 쪽을 우선한다. claim 구간은 finally/시그널로 누수를 줄인다. 회귀 테스트에 “죽은 ownerPid + 죽은 포트 → stop이 회복 / 산 포트 → 여전히 차단”을 넣는다.

이 댓글은 grok-bot이 작성했습니다

[lidge-jun]

This looks fixed on 2.58.0, and the count of receipts in your report is what points at the mechanism. Could you check whether it still reproduces on 2.58.0 or later?

The orphaned receipt was never supposed to block. handleStop reads an inherited receipt, and when endpointSource is exact it probes that endpoint; a dead endpoint is recovered. Yours were exact and the proxy was not running, so all six should have been discharged.

What blocked them is the next branch. If the Codex history preflight refused, nothing was restored, so nothing was discharged — including the receipt the ocx stop had just claimed. Before #4896 that refusal fired on the mere presence of the history_mode column, which every current Codex build has, so it fired every time. That makes one update attempt equal one retained receipt, which is why you had six.

#4896 (dfa02fe96a) changed that reason from a refusal into a degraded restore: the configuration half is written, the history relabel stands down, and the receipt is released. It is an ancestor of 2.58.0 and not of 2.57.0, and you were on 2.56.0 → 2.57.0.

One note on the first suggestion in your report — cleaning up when ownerPid is dead. We are not going to implement that as written, because recovery is decided by endpoint liveness rather than by the owner's age: a dead owner does not prove the proxy died (a Task Scheduler wrapper recreates it after the parent exits), and widening the launcher gate that way would also let quarantined receipts through. The second suggestion — reclaiming an unconfirmed claim — is a real remainder, and it is safe for claims made before a deferral was requested.

[lidge-jun]

Fixed on dev by #4960, squashed as faa31e207b3b519defa4980c8bc4c1946bbc003b.

Pending-teardown ownership is now decided by identity rather than bare PID liveness, so an interrupted stop no longer leaves a receipt that a later ocx update reads as a live owner and refuses against forever.

The shape of the fix is worth stating, because the guard itself was right: it narrows the precondition rather than opening the gate. A receipt whose owner cannot be identified is still refused; what changed is that a recycled PID no longer counts as proof that the original owner is alive.

Shipping in the next release off dev.