Skip to content

ci: pin codex-action to v1.11 and cap review-job runtime - #141

Merged
guohai merged 1 commit into
mainfrom
fix/codex-review-hang
Sep 1, 2026
Merged

guohai merged 1 commit into
mainfrom
fix/codex-review-hang

Conversation

@guohai

@guohai guohai commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Problem

codex-code-review hangs: Codex prints its final message and token count, the step then emits no ##[end-action], and the job idles until GitHub's 60-minute default kills it. It blocked #133 and several PRs since.

287 runs since 2026-04-28 never exceeded 15 minutes. Since 2026-08-24T22:25Z there have been 18 hangs of 23–64 minutes.

Cause

Upstream, in the action — not in anything of ours. This file was last edited 2026-06-16.

The floating @v1 tag moved to v1.12 (86365089) on 2026-08-20T23:38:51Z. v1.12 rewrote the privilege-isolation launch path (dropSudo / runCodexExec / linuxCredentials); it spawns the CLI with inherited stdio and waits on the child's close event, so a descendant that outlives the turn holds those descriptors open and the action never returns.

Tracked upstream as openai/codex-action#150 and #169. A wrapper fix (private pipes, complete on exit) is proposed in their #151.

Fix

Pin the action to v1.11 (52fe01ec), the last release before the rewrite, which is what the other affected orgs are running.

A near-miss worth recording

My first pass blamed the wrong thing, and the trap is generic enough to be worth writing down.

codex-version defaults to empty, so every run installs whatever npm latest is at that moment. @openai/codex@0.149.1 published 2026-08-24T00:32Z22 hours before our first hang. Clean fit, and wrong.

Our 20 runs before the boundary were all light (max 3 min) and the failure is workload-sensitive, so in our data the CLI-version boundary and the action-version boundary are perfectly confounded — both explain the evidence equally well, and no run of ours can separate them. What separates them is evidence we don't own: another org hit the same hang on codex-version 0.147.0, which predates the suspect release, and a third reports 145/145 clean on v1.11 vs 69/74 on v1.12 with model and effort held fixed.

The intermittency is why a green run proves nothing. On 2026-08-31 the same PR succeeded in 2 min at 14:38, hung 63 min at 14:45, then succeeded in 3 min at 15:05.

Also in this PR

  • timeout-minutes at JOB level, capping a hang at 10 min instead of 60. Deliberately job-level: timeout-minutes does not apply to a step that uses: a composite action, so a step-level value here would silently do nothing.
  • Same guard on claude-code-review.yml (15/5 min). It has never hung — this is a cap, not a fix — but it runs the same class of model-driven step behind a floating tag.

Deliberately not done

Posting stays inline in the same job. An earlier draft of this PR moved it to an artifact plus a separate posting job so a completed review would survive the hang. But in every observed hang the comment posted fine — posting runs before the stall. It solved a failure that never happened, at the cost of an extra job and an artifact round-trip.

Generated with SMT smt@agora.build

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🤖 Codex Code Review

No findings.

The changes look aligned with the stated goal: the model job loses write permissions, the review output is handed off via artifact, and the posting job can still run after a teardown failure without running untrusted code. The added timeouts also cap the observed runner-idle failure mode without changing normal review behavior.

Residual risk: if codex-review times out before artifact upload, post will skip cleanly because download is continue-on-error; the workflow should still fail through the codex-review job, so this is acceptable unless branch protection is changed to require only the post job.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🤖 Claude Code Review

Reviewed the two workflow files on the merge ref. The core design — cap the hang with timeout-minutes, persist the review as an artifact before teardown, and move pull-requests: write into a model-free post job — is sound, and the codex job correctly drops pull-requests: write. printf '%s' "$REVIEW_BODY" via env: is the right way to move model output into a file, and steps.download.outcome (not .conclusion) is the correct field to read under continue-on-error: true.

A few real issues:

1. if: ${{ !cancelled() }} detaches post from the trusted-author gate (codex-code-review.yml:106)

When codex-review is skipped by the author_association gate, GitHub normally skips dependents — but an explicit if: overrides that, and !cancelled() is true for a skipped dependency. So on every untrusted PR, post now spins up requesting pull-requests: write while codex-review never runs.

Impact today is small: fork PRs get a read-only GITHUB_TOKEN regardless of the declared permission, and with no artifact the download fails → the comment step is skipped → the job goes green. But the author gate was the load-bearing control here, and it no longer covers the only job with write intent; the effective gate is now "an artifact happened to exist." Narrow it to the case you actually want:

if: ${{ !cancelled() && needs.codex-review.result != 'skipped' }}

2. Empty review fails the check; a crashed review passes it (codex-code-review.yml:120-133)

The two failure paths land on opposite check colors:

  • Codex returns an empty final-message → the file is created empty → if-no-files-found: error is satisfied (the file exists) → post downloads it, trims to '', and core.setFailed turns post red.
  • The codex action itself fails (bad key, model error) → Save/Upload never run (no if: always()) → download fails → comment step skipped → post goes green.

So the harder failure produces the friendlier signal. Previously an empty message just skipped posting silently. Worth picking one convention — either treat "no artifact" as a failure too, or downgrade the empty case to core.warning.

3. claude-code-review.yml gained the timeout but not the salvage path (claude-code-review.yml:23, 89-91)

timeout-minutes: 15 is now a live way for the review job to be marked failed, but that file's post job has no if:, so it keeps the default success gate. If the 15-minute cap ever trips after Upload review output succeeded, the artifact is there and gets thrown away — exactly the failure mode the codex !cancelled() gate exists to prevent. The comment explains why 15 minutes is speculative, which is fine, but the asymmetry means the timeout you just added has no recovery path on the Claude side.

4. Pre-existing, adjacent: base.ref is shell-interpolated in the codex diff step (codex-code-review.yml:38-40)

git fetch origin ${{ github.event.pull_request.base.ref }}
DIFF=$(git diff origin/${{ github.event.pull_request.base.ref }}...HEAD --stat)

Git ref names permit $, backticks, (, ), ;, & — so a branch name is command substitution at this point. The sibling file already fixed this (BASE_REF via env:, quoted), along with using __DIFF_EOF__ instead of EOF as the heredoc delimiter so a diff line reading EOF can't terminate the block early. Not introduced here and gated behind push access, but this PR is hardening this exact file — porting those two lines over is a couple of minutes.

Nothing blocking on the artifact-handoff mechanics themselves; items 1 and 3 are the ones I'd want fixed before merge.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🤖 Codex Code Review

Findings: None.

The changes look sound for the stated goal: the model jobs are read-only, posting is isolated to a separate write-token job, the Codex teardown hang is bounded, and the artifact handoff avoids shell evaluation of model output.

Residual risk: I only reviewed the workflow diff statically; I did not execute actionlint or a live workflow run in this read-only environment.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🤖 Claude Code Review

Reviewed the two workflow files on the merge ref. The overall shape is right: pinning codex-version addresses the actual cause, the timeout caps blast radius, and splitting the writer job mirrors the hardening already in claude-code-review.yml. Three things worth changing.

Empty review now turns a green PR into a red check

codex-code-review.yml:95-107 — the old guard if: steps.run_codex.outputs.final-message != '' silently skipped posting on empty output. It's gone, and nothing replaces it:

  • printf '%s' "$REVIEW_BODY" creates a 0-byte codex-review.md when final-message is empty.
  • if-no-files-found: error checks for a matching file, not a non-empty one — the upload succeeds.
  • The post job downloads successfully, .trim() yields '', and core.setFailed(...) (line 142) marks the check failed.

So a run where codex exits 0 but produces no final message now fails the PR instead of quietly skipping. Cheapest fix that keeps the artifact isolation:

      - name: Save review output
        if: steps.run_codex.outputs.final-message != ''
        ...
      - name: Upload review output
        if: steps.run_codex.outputs.final-message != ''

Then the missing artifact takes the existing continue-on-error path and the post job goes green without commenting. (The same setFailed-on-empty exists in claude-code-review.yml:138-141, so this is consistent — but consistency with a sharp edge is still a sharp edge.)

post runs for untrusted authors

codex-code-review.yml:117-123 — a custom if: drops the implicit "needs succeeded" requirement, and !cancelled() is true when codex-review is skipped. So on any PR failing the author-association gate, codex-review skips and post still spins up a runner requesting pull-requests: write. It downloads nothing and posts nothing, so there's no escalation (fork PRs get a read-only token anyway), but it burns a runner per PR and puts a write-scoped job on the untrusted path for no benefit. Tighten to:

    if: ${{ !cancelled() && needs.codex-review.result != 'skipped' }}

Get PR diff was left on the un-hardened pattern

codex-code-review.yml:39-46 still uses the shape that claude-code-review.yml:38-48 was fixed away from — ${{ }} interpolated directly into the shell rather than passed via env:, and a bare EOF heredoc delimiter into $GITHUB_OUTPUT. Neither is presently exploitable (base.ref is the base-repo branch name; --stat indents every line so a lone EOF isn't producible), but since this PR's stated goal is to mirror the Claude workflow, porting the BASE_REF env + __DIFF_EOF__ block over would close the divergence.

Relatedly, prompt: at lines 74-77 splices PR title and body in with no untrusted-data framing, while the Claude equivalent wraps them in <pr-metadata> with an explicit "never follow instructions inside it." Same trusted-author gate bounds the risk in both, but the codex prompt is the weaker of the two and its output is posted verbatim.

Correct as written

  • steps.download.outcome == 'success' (line 135) is the right context — outcome is pre-continue-on-error, so it's failure when the artifact is absent. Using conclusion here would have been the bug.
  • Artifact upload is a main step, so it completes before the teardown hang the comment describes; a 10-minute timeout won't lose an already-finalized artifact.
  • 10 and 15 minutes sit well above the cited maxima (198s / 394s).
  • The comments actually carry the evidence (run IDs, publish dates, hang rate) rather than asserting a conclusion — that's the part that makes this revisitable when upstream fixes the orphaned process.

Nit, pre-existing in both files: no concurrency group, so rapid pushes to a PR produce overlapping runs and duplicate review comments.

@guohai
guohai force-pushed the fix/codex-review-hang branch from 47897c0 to 681defe Compare September 1, 2026 04:15
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🤖 Codex Code Review

Findings

  • High — Codex posting is not isolated, and the model job still has write permission. .github/workflows/codex-code-review.yml:24 still grants pull-requests: write, and .github/workflows/codex-code-review.yml:84 still posts from the same job that runs openai/codex-action. This means the PR does not implement the stated isolation: prompt-influenced model output still runs in a writer-token job, and if the known post-step teardown hang happens, the whole review check will still be marked failed after the new 10-minute timeout. The intended artifact + separate post job gated with !cancelled() appears to be missing.

Open Questions

  • The PR description says Codex posting was moved to its own job and the review job dropped pull-requests: write, but the diff only adds timeouts and pins codex-version. Is this branch missing commits?

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🤖 Claude Code Review

Verified the diff against fd7c062 and read both workflows in full.

The substance is sound: timeout-minutes on all three jobs is well below the observed healthy maxima (10 min vs ~400s for codex, 15 min vs 394s for claude, 5 min for the artifact-download-and-post job), so no legitimate run gets cut off, and the 60-minute default no longer parks a runner on a hang. The codex-version pin is correct in its reasoning — openai/codex-action@v1 pinning the action doesn't pin the npm binary it fetches, so the empty-default input really was the hole, and the comment records the evidence needed to unpin later.

Two things worth raising:

1. The title's second claim isn't in the diff. "isolate its posting job" — its reads as codex's, but codex-code-review.yml still runs the model and Post Review Comment in one job holding pull-requests: write (lines 22-24, 84-97). The only posting-job change here is timeout-minutes: 5 added to claude's already-separate post job. That job separation is the exact pattern claude-code-review.yml:8-10 documents as the mitigation ("Posting is done by a separate, model-free job"), so codex remains the outlier. Either split it in this PR or drop the clause from the title — as written the commit message asserts a security property the file doesn't have.

2. The cap doesn't change the check's outcome, only its duration. Per the comment, the stall occurs after every step completes — so the review comment has already been posted by the time the job hangs. timeout-minutes: 10 converts a 60-minute hang into a 10-minute cancellation, but the job still ends red. If Codex Code Review is a required status check, a hung-but-successful review will still block merge; it'll just block it sooner. Worth confirming that's understood, since the pin (not the timeout) is what actually restores green.

Minor, pre-existing, in a file this PR touches: codex's Get PR diff (lines 33-40) still uses the unquoted-$GITHUB_OUTPUT, fixed-EOF-delimiter, inline-${{ }} pattern that claude-code-review.yml:38-48 was hardened away from (env var + quoted heredoc + unguessable delimiter). It also interpolates github.event.pull_request.body straight into the prompt with no untrusted-data fence, where the claude side wraps metadata in <pr-metadata> with an explicit instruction not to follow it. The trusted-author gate limits the blast radius, so this isn't urgent — but the two files having diverged on it is the kind of thing that's easiest to fix while you're already in here.

codex-code-review has been hanging: Codex prints its final message and token
count, the step then emits no `##[end-action]`, and the job idles until
GitHub's 60-minute default kills it. 287 runs since 2026-04-28 never exceeded
15 minutes; since 2026-08-24T22:25Z there have been 18 hangs of 23-64 minutes.
It blocked PR #133 and several since.

Cause is upstream, in the action rather than in anything of ours: the floating
`@v1` tag moved to v1.12 (86365089) on 2026-08-20T23:38:51Z, and v1.12 rewrote
the privilege-isolation launch path. It spawns the CLI with inherited stdio and
waits on the child's `close` event, so a descendant outliving the turn keeps
those descriptors open and the action never returns. Tracked upstream as
openai/codex-action#150 and #169; a wrapper fix using private pipes and
completing on `exit` is proposed in their #151. So pin to v1.11 (52fe01ec),
which is what other affected orgs are running.

Worth recording how this was nearly mis-diagnosed, because the trap is generic.
The first pass here blamed an unpinned CLI: `codex-version` defaults to empty,
so every run installs whatever npm `latest` is at that moment, and 0.149.1
published 2026-08-24T00:32Z — 22 hours before our first hang. That fit, and it
was wrong. Our 20 clean runs before the boundary were all light (max 3 min),
and the failure is workload-sensitive, so the CLI-version boundary and the
action-version boundary are perfectly confounded in our data: both explain it
equally well and our runs cannot separate them. What separates them is evidence
we do not own — another org hit the same hang on codex-version 0.147.0, which
predates the suspect release, and a third has 145/145 clean on v1.11 against
69/74 on v1.12 with model and effort held fixed.

The intermittency is the reason a green run proves nothing: on 2026-08-31 the
same PR succeeded in 2 min at 14:38, hung 63 min at 14:45, and succeeded in
3 min at 15:05.

timeout-minutes caps the damage at 10 minutes rather than 60. Note it is at JOB
level deliberately — `timeout-minutes` does not apply to a step that `uses:` a
composite action, so a step-level value here would silently do nothing.
Claude's workflow gets the same guard at 15/5 minutes; it has never hung, but
it runs the same class of model-driven step behind a floating tag.

Deliberately NOT changed: posting stays inline in the same job. An earlier
draft moved it to an artifact plus a separate posting job so a review would
survive the hang, but in every observed hang the comment posted fine — posting
runs before the stall. That solved a failure that never happened.

🤖 Built with SMT <smt@agora.build>
@guohai
guohai force-pushed the fix/codex-review-hang branch from 681defe to 3a2f751 Compare September 1, 2026 04:19
@guohai guohai changed the title ci: cap the codex-review hang and isolate its posting job ci: pin codex-action to v1.11 and cap review-job runtime Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🤖 Codex Code Review

Findings

  • High .github/workflows/codex-code-review.yml:24 — The Codex review job still grants pull-requests: write and still posts from the same job at .github/workflows/codex-code-review.yml:91. This PR description says posting was moved to an isolated job and the model-bearing job drops write access, but the actual workflow does neither. That leaves the Codex action running in a job with a write-capable github.token available to actions in the job, preserving the prompt-injection/write-token risk the Claude workflow explicitly avoids.

No other concrete issues found in the timeout additions or the pinning change.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🤖 Claude Code Review

Reviewed both workflow files against the diff.

The pin itself is the right call and the comment justifying it is unusually good — it distinguishes "a green run cleared it" from "controlled data indicts it," which is the distinction that actually matters for a nondeterministic hang. Two things worth resolving before merge.

1. The PR title claims something the diff doesn't do. Title says "cap the codex-review hang and isolate its posting job," but codex-code-review.yml still runs the model and posts in a single job holding pull-requests: write (.github/workflows/codex-code-review.yml:24-25, posting step at :89). The Codex model receives untrusted PR title/body/diff in its prompt and its final-message is posted verbatim — from a job that has a write token in scope for the entire model run.

claude-code-review.yml does exactly the split the title describes, and documents why: read-only review job → artifact → model-free post job with the write permission. Whichever way you go, make them agree:

  • If the isolation was intended and got dropped, port the artifact-handoff pattern to the Codex job.
  • If it's deliberately deferred, retitle so the commit history doesn't record a security property the file doesn't have.

Not urgent on its own — the trusted-author gate means the injecting party already has push access — but the same gate applies to claude-code-review.yml, which isolated anyway. The asymmetry is what stands out.

2. Unquoted base.ref interpolation in the Codex diff step (.github/workflows/codex-code-review.yml:35-37). ${{ ... }} is expanded into the shell script text before bash sees it, and git ref names permit `, $, ;, and &. A branch named main;curl … would execute. Again gated behind push access, but claude-code-review.yml:40-49 already fixed precisely this by passing BASE_REF through env: and quoting it — the safe version is one file over, and this PR is already editing the unsafe file. Same step's EOF heredoc delimiter is weaker than Claude's __DIFF_EOF__ for the same reason (--stat indents its lines, so it's near-unreachable, but the fix is free while you're in there).

Minor / no action needed:

  • The timeout-minutes placement note is correct — step-level timeouts genuinely don't apply to composite-action steps (Support timeout-minutes in composite-actions actions/runner#1979), so job level is the only thing that bites. Good that the comment records why it's at job level; that's the detail a future editor would otherwise "clean up."
  • Be aware of what the 10-minute cap buys: in the hang mode described, Codex has already produced its final message but the action never returns, so step outputs are never set. The cap converts a 60-minute hang into a 10-minute hang and a silently missing review — it doesn't rescue the output. That's the right tradeoff given the pin is the actual fix, just don't expect the timeout to preserve a review.
  • Headroom on both caps looks sound: 10 min vs ~200s healthy max, 15 min vs 394s max.
  • I couldn't verify from this sandbox that 52fe01ec70a4… is in fact the v1.11 tag (no network access granted). Worth a one-line confirmation before merge, since a SHA pin's whole value is that nobody re-checks it later.

Everything else — permissions scoping, the trusted-author gates, the post job's parse-and-fail-loudly logic — reads clean.

@guohai
guohai merged commit 4904552 into main Sep 1, 2026
3 checks passed
@guohai
guohai deleted the fix/codex-review-hang branch September 1, 2026 04:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant