Summary
The e2e suite decides how long to wait for a pod using a fixed table keyed on
substrings of the workload name:
if [[ "$WORKLOAD" == *"70b"* ]]; then POD_READY_TIMEOUT=1800
elif [[ "$WORKLOAD" == trtllm-* ]]; then POD_READY_TIMEOUT=1800
elif [[ "$WORKLOAD" == *"qwen32b"* ]]; then POD_READY_TIMEOUT=1800
else POD_READY_TIMEOUT=600
fi
There are 13 workloads. Three of those patterns match. The other ten get 600
seconds no matter how large the model is, including every sglang workload,
gemma-sglang, e5-mistral and nim-llama-8b.
Why it keeps costing time
nim-qwen3-32b does not match *qwen32b*, because the name contains
qwen3-32b. It therefore gets 600 seconds for a 61 GB model. Observed on dev1:
The pod was healthy. Left alone it reached Ready and the full run passed in
7m 28s warm. The failure is reported as "Pod ready FAIL", which reads like a
capture or restore defect, so the cost is not the wasted ten minutes, it is the
hour spent disproving a bug that never existed. This has now happened three
times: once on a 70B cold start that needed more than the 1800 second branch
allowed, once here, and once on a workload that never matched a branch at all.
A fixed number cannot work in principle. The same workload takes 35 minutes
cold and 4 minutes warm. Any value large enough for the cold case makes a
genuinely hung pod burn the whole budget before failing.
Proposal
Wait on progress rather than on a deadline.
Poll the pod and keep a fingerprint of everything that changes while startup is
healthy: phase, conditions, per-container state and reason, restart count, and
log size. If the fingerprint moves, reset the clock. If it has not moved for
NVSNAP_STALL_TIMEOUT (default 600s), the pod is hung, so fail with the last
observed state and say plainly that this is a startup stall rather than a
capture failure. Keep NVSNAP_MAX_WAIT (default 5400s) only as a backstop.
Fail immediately, without waiting, on states no amount of time will fix:
ImagePullBackOff, ErrImagePull, InvalidImageName,
CreateContainerConfigError, CrashLoopBackOff, non-zero container exit,
OOMKilled, and pod phase Failed.
The result is strictly better in both directions. A 64 GB download that takes 35
minutes passes because it is making progress throughout. A wedged pod fails in
10 minutes instead of 30, with the reason attached.
Delete the per-workload table. Nothing replaces it, which is the point.
Scope
scripts/test-e2e.sh and scripts/test-bench.sh both need it. test-bench.sh
has the same defect in milder form: a single 1800 second value for every
workload.
Summary
The e2e suite decides how long to wait for a pod using a fixed table keyed on
substrings of the workload name:
There are 13 workloads. Three of those patterns match. The other ten get 600
seconds no matter how large the model is, including every sglang workload,
gemma-sglang, e5-mistral and nim-llama-8b.
Why it keeps costing time
nim-qwen3-32bdoes not match*qwen32b*, because the name containsqwen3-32b. It therefore gets 600 seconds for a 61 GB model. Observed on dev1:The pod was healthy. Left alone it reached Ready and the full run passed in
7m 28s warm. The failure is reported as "Pod ready FAIL", which reads like a
capture or restore defect, so the cost is not the wasted ten minutes, it is the
hour spent disproving a bug that never existed. This has now happened three
times: once on a 70B cold start that needed more than the 1800 second branch
allowed, once here, and once on a workload that never matched a branch at all.
A fixed number cannot work in principle. The same workload takes 35 minutes
cold and 4 minutes warm. Any value large enough for the cold case makes a
genuinely hung pod burn the whole budget before failing.
Proposal
Wait on progress rather than on a deadline.
Poll the pod and keep a fingerprint of everything that changes while startup is
healthy: phase, conditions, per-container state and reason, restart count, and
log size. If the fingerprint moves, reset the clock. If it has not moved for
NVSNAP_STALL_TIMEOUT(default 600s), the pod is hung, so fail with the lastobserved state and say plainly that this is a startup stall rather than a
capture failure. Keep
NVSNAP_MAX_WAIT(default 5400s) only as a backstop.Fail immediately, without waiting, on states no amount of time will fix:
ImagePullBackOff,ErrImagePull,InvalidImageName,CreateContainerConfigError,CrashLoopBackOff, non-zero container exit,OOMKilled, and pod phaseFailed.The result is strictly better in both directions. A 64 GB download that takes 35
minutes passes because it is making progress throughout. A wedged pod fails in
10 minutes instead of 30, with the reason attached.
Delete the per-workload table. Nothing replaces it, which is the point.
Scope
scripts/test-e2e.shandscripts/test-bench.shboth need it.test-bench.shhas the same defect in milder form: a single 1800 second value for every
workload.