Skip to content

fix(deploy): move compose_p into a tested lib; handle stale working_dir labels #103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
owine merged 1 commit into from
Oct 10, 2026
Merged

fix(deploy): move compose_p into a tested lib; handle stale working_dir labels #103

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
176 changes: 38 additions & 138 deletions .github/workflows/deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -342,51 +342,21 @@ jobs:
echo "skipped=false" >> "$GITHUB_OUTPUT"
fi

# Shared by every compose ps/logs/down below that runs without `op run`.
# Keep in sync with the copies in the deploy, health-check and rollback jobs.
# Scripts pinned to this workflow's commit, as in the prepare job. The
# target checkout below cleans the workspace, so the helper is copied out
# to $RUNNER_TEMP first (the shared-networks checkout re-fetches later).
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

@sourcery-ai sourcery-ai Bot Oct 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High · Rollback recovery gets skipped

When a required compose-workflow checkout fails after deployment starts, the health-check checkout failure triggers rollback, while a rollback checkout failure prevents the plan and restore steps from running; GitHub skips those recovery steps and leaves the failed deployment unrestored.

Make the helper available to recovery without a required network fetch, or provide a fallback helper path.

Also at .github/workflows/deploy.yml:829-833, .github/workflows/deploy.yml:1013, .github/workflows/deploy.yml:1022, .github/workflows/deploy.yml:1039, .github/workflows/deploy.yml:1081-1082.

Prompt for AI agents 
In `.github/workflows/deploy.yml` at line 348:

**Rollback recovery gets skipped**

When a required compose-workflow checkout fails after deployment starts, the health-check checkout failure triggers rollback, while a rollback checkout failure prevents the plan and restore steps from running; GitHub skips those recovery steps and leaves the failed deployment unrestored.

Make the helper available to recovery without a required network fetch, or provide a fallback helper path.

Also at `.github/workflows/deploy.yml:829-833`, `.github/workflows/deploy.yml:1013`, `.github/workflows/deploy.yml:1022`, `.github/workflows/deploy.yml:1039`, `.github/workflows/deploy.yml:1081-1082`.

if: steps.skip-gate.outputs.skipped == 'false'
with:
repository: owine/compose-workflow
ref: ${{ job.workflow_sha }}
path: .compose-workflow

# compose_p for every compose ps/logs/down below that runs without `op run`
# (scripts/deployment/lib/compose-project.sh).
- name: Install compose project helper
if: steps.skip-gate.outputs.skipped == 'false'
run: |
set -euo pipefail
cat > "$RUNNER_TEMP/compose-project.sh" <<'SCRIPT'
# usage: compose_projects <stack> -> the stack's Compose project name(s), one per line
# Read from the containers' working_dir label, so a top-level `name:` or
# a dir name Compose normalises (dots, uppercase, leading _/-) still
# resolves. Every distinct match is returned (e.g. leftovers from a renamed
# project). With no containers in the dir it falls back to the stack name,
# unless a project of that name runs from another dir (e.g. dockge's own
# /opt/dockge): then it returns nothing, so down is a no-op and the health
# check sees "no containers" rather than acting on a foreign project.
compose_projects() {
local dir="${LIVE_REPO_PATH%/}/$1" p others
p=$(docker ps -a --filter "label=com.docker.compose.project.working_dir=$dir" \
--format '{{.Label "com.docker.compose.project"}}' 2>/dev/null \
| sort -u) || p=""
if [[ -z "$p" ]]; then
# Captured first: piping docker into `grep -q` can SIGPIPE under pipefail.
others=$(docker ps -a --filter "label=com.docker.compose.project=$1" \
--format '{{.Label "com.docker.compose.project.working_dir"}}' 2>/dev/null) || others=""
if [[ -n "$others" ]] && grep -qvxF -- "$dir" <<<"$others"; then
return 0
fi
p="$1"
fi
printf '%s\n' "$p"
}
# usage: compose_p <stack> <compose args...>
# Runs compose against each of the stack's projects by name from / (no
# compose file there), so the stack's compose file is never parsed: these
# calls run without `op run`, and a file whose syntax needs ${VARS} would
# fail to parse. Non-zero if the command failed for any project.
compose_p() {
local stack="$1" proj rc=0; shift
while IFS= read -r proj; do
[[ -n "$proj" ]] || continue
(cd / && docker compose -p "$proj" "$@" </dev/null) || rc=$?
done < <(compose_projects "$stack")
return "$rc"
}
SCRIPT
run: install -m 0644 .compose-workflow/scripts/deployment/lib/compose-project.sh "$RUNNER_TEMP/compose-project.sh"

- name: Teardown removed stacks (uses pre-reset compose files)
if: |
Expand Down Expand Up @@ -855,50 +825,17 @@ jobs:
FAILED_LOG_LINES: ${{ inputs.failed-container-log-lines }}
OP_SERVICE_ACCOUNT_TOKEN: ${{ secrets.OP_SERVICE_ACCOUNT_TOKEN }}
steps:
# Shared by every compose ps/logs/down below that runs without `op run`.
# Keep in sync with the copies in the deploy, health-check and rollback jobs.
# Scripts pinned to this workflow's commit, as in the other jobs.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: owine/compose-workflow
ref: ${{ job.workflow_sha }}
path: .compose-workflow

# compose_p for every compose ps/logs/down below that runs without `op run`
# (scripts/deployment/lib/compose-project.sh).
- name: Install compose project helper
run: |
set -euo pipefail
cat > "$RUNNER_TEMP/compose-project.sh" <<'SCRIPT'
# usage: compose_projects <stack> -> the stack's Compose project name(s), one per line
# Read from the containers' working_dir label, so a top-level `name:` or
# a dir name Compose normalises (dots, uppercase, leading _/-) still
# resolves. Every distinct match is returned (e.g. leftovers from a renamed
# project). With no containers in the dir it falls back to the stack name,
# unless a project of that name runs from another dir (e.g. dockge's own
# /opt/dockge): then it returns nothing, so down is a no-op and the health
# check sees "no containers" rather than acting on a foreign project.
compose_projects() {
local dir="${LIVE_REPO_PATH%/}/$1" p others
p=$(docker ps -a --filter "label=com.docker.compose.project.working_dir=$dir" \
--format '{{.Label "com.docker.compose.project"}}' 2>/dev/null \
| sort -u) || p=""
if [[ -z "$p" ]]; then
# Captured first: piping docker into `grep -q` can SIGPIPE under pipefail.
others=$(docker ps -a --filter "label=com.docker.compose.project=$1" \
--format '{{.Label "com.docker.compose.project.working_dir"}}' 2>/dev/null) || others=""
if [[ -n "$others" ]] && grep -qvxF -- "$dir" <<<"$others"; then
return 0
fi
p="$1"
fi
printf '%s\n' "$p"
}
# usage: compose_p <stack> <compose args...>
# Runs compose against each of the stack's projects by name from / (no
# compose file there), so the stack's compose file is never parsed: these
# calls run without `op run`, and a file whose syntax needs ${VARS} would
# fail to parse. Non-zero if the command failed for any project.
compose_p() {
local stack="$1" proj rc=0; shift
while IFS= read -r proj; do
[[ -n "$proj" ]] || continue
(cd / && docker compose -p "$proj" "$@" </dev/null) || rc=$?
done < <(compose_projects "$stack")
return "$rc"
}
SCRIPT
run: install -m 0644 .compose-workflow/scripts/deployment/lib/compose-project.sh "$RUNNER_TEMP/compose-project.sh"

- name: Health check critical stacks
id: h
Expand Down Expand Up @@ -951,8 +888,9 @@ jobs:
echo "::error::Critical stack $stack: docker compose ps failed"
continue
fi
# `ps -p <name>` exits 0 with no output for an unknown project; an empty
# critical stack must fail, not pass vacuously.
# compose_p prints nothing for a stack with no project (and `ps -p` is
# empty for one with no containers); an empty critical stack must fail,
# not pass vacuously.
if [[ "$(jq 'length' <<<"$services")" -eq 0 ]]; then
failed+=("$stack")
echo "::error::Critical stack $stack has no containers"
Expand Down Expand Up @@ -1072,14 +1010,16 @@ jobs:
# fails on a missing external network. Scripts pinned to this workflow's
# commit, as in the other jobs.
#
# continue-on-error on both: a failure here must not skip the plan and
# restore steps below. Stacks that need a missing network then fail
# their own `up` (with the usual recover-manually warning); every other
# stack still rolls back. The deploy job, by contrast, fails fast here,
# before any stack is touched, which itself triggers this rollback.
# The checkout is unconditional and required: the compose project helper
# comes from it, and every teardown below sources that helper.
#
# continue-on-error on the network step: a failure there must not skip
# the plan and restore steps below. Stacks that need a missing network
# then fail their own `up` (with the usual recover-manually warning);
# every other stack still rolls back. The deploy job, by contrast, fails
# fast there, before any stack is touched, which itself triggers this
# rollback.
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
if: env.SHARED_NETWORKS != ''
continue-on-error: true
with:
repository: owine/compose-workflow
ref: ${{ job.workflow_sha }}
Expand All @@ -1093,50 +1033,10 @@ jobs:
read -ra nets <<< "$SHARED_NETWORKS"
./.compose-workflow/scripts/deployment/ensure-shared-networks.sh "${nets[@]}"

# Shared by every compose ps/logs/down below that runs without `op run`.
# Keep in sync with the copies in the deploy, health-check and rollback jobs.
# compose_p for every compose ps/logs/down below that runs without `op run`
# (scripts/deployment/lib/compose-project.sh).
- name: Install compose project helper
run: |
set -euo pipefail
cat > "$RUNNER_TEMP/compose-project.sh" <<'SCRIPT'
# usage: compose_projects <stack> -> the stack's Compose project name(s), one per line
# Read from the containers' working_dir label, so a top-level `name:` or
# a dir name Compose normalises (dots, uppercase, leading _/-) still
# resolves. Every distinct match is returned (e.g. leftovers from a renamed
# project). With no containers in the dir it falls back to the stack name,
# unless a project of that name runs from another dir (e.g. dockge's own
# /opt/dockge): then it returns nothing, so down is a no-op and the health
# check sees "no containers" rather than acting on a foreign project.
compose_projects() {
local dir="${LIVE_REPO_PATH%/}/$1" p others
p=$(docker ps -a --filter "label=com.docker.compose.project.working_dir=$dir" \
--format '{{.Label "com.docker.compose.project"}}' 2>/dev/null \
| sort -u) || p=""
if [[ -z "$p" ]]; then
# Captured first: piping docker into `grep -q` can SIGPIPE under pipefail.
others=$(docker ps -a --filter "label=com.docker.compose.project=$1" \
--format '{{.Label "com.docker.compose.project.working_dir"}}' 2>/dev/null) || others=""
if [[ -n "$others" ]] && grep -qvxF -- "$dir" <<<"$others"; then
return 0
fi
p="$1"
fi
printf '%s\n' "$p"
}
# usage: compose_p <stack> <compose args...>
# Runs compose against each of the stack's projects by name from / (no
# compose file there), so the stack's compose file is never parsed: these
# calls run without `op run`, and a file whose syntax needs ${VARS} would
# fail to parse. Non-zero if the command failed for any project.
compose_p() {
local stack="$1" proj rc=0; shift
while IFS= read -r proj; do
[[ -n "$proj" ]] || continue
(cd / && docker compose -p "$proj" "$@" </dev/null) || rc=$?
done < <(compose_projects "$stack")
return "$rc"
}
SCRIPT
run: install -m 0644 .compose-workflow/scripts/deployment/lib/compose-project.sh "$RUNNER_TEMP/compose-project.sh"

- name: Resolve rollback plan
id: plan
Expand Down
8 changes: 5 additions & 3 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@ Used by `deploy.yml`. The deploy/health/rollback logic is **inlined as workflow
- **`detect-stack-changes.sh`** — three-method detection (git diff, tree comparison, discovery analysis) for removed/existing/new stacks. Runs locally on the runner against the workspace checkout (which has full history + GitHub creds), then the workflow uses the classifications to drive teardown / sequential deploy. Cleanup of removed stacks happens in the workflow's `Teardown removed stacks` step, not in this script.
- **`detect-critical-stacks.sh`** — scans for `com.compose.tier: infrastructure` labels, emits JSON array
- **`build-pr-comment.sh`** — builds the deploy-status PR comment body (used by the notify job's PR-comment step)
- **`ensure-shared-networks.sh`** — idempotently creates the `shared-networks` bridge networks outside any Compose project. The `deploy` and `rollback` jobs check out compose-workflow (at `job.workflow_sha`) only when that input is set; `deploy`'s own target checkout runs `git clean -ffdx`, so `prepare`'s copy never survives into it
- **`ensure-shared-networks.sh`** — idempotently creates the `shared-networks` bridge networks outside any Compose project. Run by `deploy` and `rollback` when that input is set, from their compose-workflow checkout (at `job.workflow_sha`); `deploy`'s own target checkout runs `git clean -ffdx`, so `deploy` checks compose-workflow out twice (once for the compose project helper, before the target checkout, and again for this script)

The previously-existing `deploy-stacks.sh`, `health-check.sh`, `rollback-stacks.sh`, `cleanup-stack.sh`, and `lib/ssh-helpers.sh` were removed when the SSH-based reusable workflow was retired in favor of the inline-step approach now in `deploy.yml` — their logic lives directly in the workflow file.

Expand Down Expand Up @@ -254,7 +254,7 @@ docker compose -f stack/compose.yaml config
- **Native health verification** — `docker compose up --wait` for atomic readiness
- **Multi-registry auth** — single `1password/load-secrets-action` step pulls all four registry credential pairs, four `docker/login-action` steps with `logout: false` and `continue-on-error: true` so a single misconfigured registry doesn't block deploys that may not pull from it
- **Sequential existing-then-new** — new stacks only deploy if existing stacks succeeded
- **No-env compose calls** — `ps`/`logs`/`down` run without `op run`, so they go through the `compose_p` helper (installed per job into `$RUNNER_TEMP/compose-project.sh`): it resolves the stack's Compose project(s) from the containers' `com.docker.compose.project.working_dir` label (all distinct matches; falls back to the stack name) and runs `docker compose -p <project>` from `/` for each, so the stack's compose file — whose syntax may need `${VARS}` — is never parsed. Keep the three copies (deploy, health-check, rollback) in sync
- **No-env compose calls** — `ps`/`logs`/`down` run without `op run`, so they go through `compose_p` from `scripts/deployment/lib/compose-project.sh` (each of `deploy`, `health-check`, `rollback` checks out compose-workflow at `job.workflow_sha` and copies it to `$RUNNER_TEMP/compose-project.sh`). It resolves the stack's Compose project(s) from the containers' `com.docker.compose.project.working_dir` label (all distinct matches) and runs `docker compose -p <project>` from `/` for each, so the stack's compose file — whose syntax may need `${VARS}` — is never parsed. With no containers under the stack dir it falls back to the normalised stack name only if every container of that project came from a dir also named `<stack>` (a stale label from an older checkout; `::warning::`); a same-named project from any other dir is left alone (`::warning::`). Covered by `scripts/testing/test-compose-project.sh`
- **Failure diagnostics** — on stack failure or health failure, dumps `docker compose ps -a`, `docker inspect` of `.State.Health.Log` (probe history with exit codes + stdout), and `docker compose logs --tail N` scoped to the failing service
- **Automatic rollback** — `git reset --hard <previous_sha>` + redeploy if deploy or health-check failed
- **Discord notifications** — pipeline-status icon line, removed-stacks list, commit link, user mention on failure
Expand Down Expand Up @@ -321,13 +321,15 @@ Follow `docs/superpowers/runbooks/adding-a-new-host.md` (the generic step-by-ste
│ │ └── lint-summary.sh
│ ├── deployment/
│ │ ├── lib/
│ │ │ └── common.sh # logging + validation
│ │ │ ├── common.sh # logging + validation
│ │ │ └── compose-project.sh # compose_p: no-env ps/logs/down by project label
│ │ ├── detect-stack-changes.sh
│ │ ├── detect-critical-stacks.sh
│ │ ├── ensure-shared-networks.sh
│ │ └── build-pr-comment.sh
│ └── testing/
│ ├── test-detect-critical-stacks.sh
│ ├── test-compose-project.sh
│ ├── test-workflow.sh
│ ├── validate-compose.sh
│ └── README.md
Expand Down
Loading