Skip to content

Commit 1f2299d

Browse files
authored
fix(ci): bound a stalled bun install and let the helm test probe wait for routing (#9409)
* fix(ci): bound a stalled bun install and let the helm test probe wait for routing * fix(ci): fit the install retry inside every caller and read a failed probe's log from its kept pod
1 parent 40df163 commit 1f2299d

4 files changed

Lines changed: 41 additions & 12 deletions

File tree

‎.github/actions/setup/action.yml‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,20 @@ runs:
6666
key: ${{ github.repository }}-${{ inputs.turbo-cache-key }}-${{ github.event_name }}${{ github.event.pull_request.head.repo.fork && '-fork' || '' }}
6767
path: ./.turbo
6868

69+
# Bun retries a failed request itself, but a registry connection that stalls
70+
# without failing leaves `bun install` waiting until the job's own timeout
71+
# cancels it, the whole budget spent on one hung step. Healthy installs finish
72+
# in under two and a half minutes cold, so an attempt past three is stalled:
73+
# kill it and retry once, which leaves room inside the shortest caller's
74+
# 10-minute job timeout. Any other failure is real and fails at once.
6975
- name: Install dependencies
7076
shell: bash
71-
run: bun install --frozen-lockfile --ignore-scripts
77+
run: |
78+
for attempt in 1 2; do
79+
timeout --kill-after=15s 3m bun install --frozen-lockfile --ignore-scripts && exit 0
80+
status=$?
81+
if [ "$status" -ne 124 ] && [ "$status" -ne 137 ]; then exit "$status"; fi
82+
echo "::warning title=bun install stalled::attempt ${attempt} of 2 ran past 3 minutes"
83+
done
84+
echo "::error title=bun install stalled::both attempts ran past 3 minutes"
85+
exit 1

‎.github/workflows/helm.yml‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,12 @@ jobs:
188188
--values helm/sim/ci/kind-overlay.yaml \
189189
--wait --timeout 15m
190190
191+
- name: Run helm test
192+
run: helm test sim --namespace sim --timeout 5m
193+
194+
# After the test, so it also explains a failed probe, not only a failed
195+
# install. A failed test pod is kept (the hook deletes it only on success),
196+
# so its log says which URL refused.
191197
- name: Diagnostics on failure
192198
if: failure()
193199
run: |
@@ -196,9 +202,8 @@ jobs:
196202
kubectl -n sim describe pods | tail -100 || true
197203
kubectl -n sim logs deploy/sim-app -c migrations --tail=50 || true
198204
kubectl -n sim logs deploy/sim-app --tail=80 || true
199-
200-
- name: Run helm test
201-
run: helm test sim --namespace sim --timeout 5m
205+
kubectl -n sim logs deploy/sim-realtime --tail=80 || true
206+
kubectl -n sim logs sim-test-connection || true
202207
203208
# Resolved once and shared, so the two publish paths cannot package the same
204209
# immutable chart version with different appVersions -- they run in parallel,

‎helm/sim/Chart.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ apiVersion: v2
22
name: sim
33
description: A Helm chart for Sim - the open-source AI workspace where teams build, deploy, and manage AI agents
44
type: application
5-
version: 1.11.8
5+
version: 1.11.9
66
appVersion: "v0.8.26"
77
kubeVersion: ">=1.25.0-0"
88
home: https://sim.ai

‎helm/sim/templates/tests/test-connection.yaml‎

Lines changed: 17 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -40,15 +40,25 @@ spec:
4040
- -c
4141
- |
4242
set -eu
43-
APP_URL="http://{{ include "sim.fullname" . }}-app:{{ .Values.app.service.port }}/"
44-
echo "Probing $APP_URL"
45-
wget -q -T {{ .Values.tests.timeoutSeconds }} --tries=1 --spider "$APP_URL"
43+
# A pod turns Ready before every node routes its Service to it, so a
44+
# probe sent the moment `helm install --wait` returns can be refused.
45+
# Each URL is retried until a minute has passed, then the test fails.
46+
probe() {
47+
deadline=$(($(date +%s) + 60))
48+
while ! wget -q -T {{ .Values.tests.timeoutSeconds }} --spider "$1"; do
49+
if [ "$(date +%s)" -ge "$deadline" ]; then
50+
echo "FAILED: $1 did not answer within 60s"
51+
return 1
52+
fi
53+
echo "$1 did not answer; retrying in 5s"
54+
sleep 5
55+
done
56+
echo "OK: $1"
57+
}
58+
probe "http://{{ include "sim.fullname" . }}-app:{{ .Values.app.service.port }}/"
4659
{{- if .Values.realtime.enabled }}
47-
RT_URL="http://{{ include "sim.fullname" . }}-realtime:{{ .Values.realtime.service.port }}/health"
48-
echo "Probing $RT_URL"
49-
wget -q -T {{ .Values.tests.timeoutSeconds }} --tries=1 --spider "$RT_URL"
60+
probe "http://{{ include "sim.fullname" . }}-realtime:{{ .Values.realtime.service.port }}/health"
5061
{{- end }}
51-
echo "OK"
5262
resources:
5363
{{- toYaml .Values.tests.resources | nindent 8 }}
5464
{{- end }}

0 commit comments

Comments
 (0)