Analyze failures skill

.claude/skills/analyze-failures/SKILL.md

@@ -0,0 +1,78 @@
+---
+name: analyze-failures
+description: Triage a failed cardano-node-tests run, and answer questions about where a test run's logs and artifacts live. Use when the user asks to analyze, triage, or explain failures in a regression run, a node-upgrade run, a run_workdir, a saved run directory or artifacts downloaded from a CI run, and also when they ask where to find node logs, cluster logs, test artifacts, allure results or the cluster-management status database after running tests. Accepts an optional run directory argument (default `run_workdir`).
+---
+
+# Analyze test run failures
+
+Triage a finished cardano-node-tests run and report grouped failures with a likely root cause
+for each group.
+
+## 1. Resolve the run directory
+
+The directory passed as the skill argument, or `run_workdir` if none was given. The path may be
+relative to the current working directory or absolute; it can be a fresh run produced by
+`runner/regression.sh`, artifacts downloaded from a CI run, or any saved historical run
+directory.
+
+If the directory does not exist, stop and say so - do not guess another path.
+
+## 2. Unpack tarballs if needed
+
+A run directory produced locally holds both the unpacked result dirs and their `*.tar.xz`
+archives (`runner/create_results.sh` keeps both). Artifacts **downloaded from a CI run** hold
+only the archives. Each archive contains a single top-level directory of the same name, so
+unpack any archive whose directory is missing before reading anything:
+
+```sh
+tar -xf <run_dir>/testing_artifacts.tar.xz -C <run_dir>
+```
+
+## 3. Detect the testsuite
+
+Match on results present as a directory **or** as a tarball:
+
+- `allure-results-step*` -> **node-upgrade run**: follow `agent_docs/upgrade_failure_analysis_prompt.md`.
+- otherwise `allure-results` -> **regression run**: follow `agent_docs/failure_analysis_prompt.md`.
+- neither -> do **not** default to the regression prompt. An upgrade run that died before
+  step1's pytest produces no results at all and would be misread as a regression run. Report
+  what the directory does contain (`testrun-report-step*.html` and three `testing_artifacts/pytest-*`
+  dirs point at an upgrade run) and ask which testsuite it was.
+
+Say which one you detected before you start.
+
+## 4. Run the analysis
+
+Read the detected prompt file and follow it. Substitute the resolved run directory for every
+`{RUN_DIR}` placeholder in it.
+
+Those prompt files are the single source of truth for the analysis - the same ones the CI
+workflows use - so do not restate or reinterpret their instructions here.
+
+## 5. Report
+
+Report the findings in the conversation and stay available for follow-up questions.
+
+Do **not** write `failure_analysis.md`, cap the output length, or apply any other constraint
+from `agent_docs/ci_analysis_prompt.md`. Those are CI-only and the workflows append them
+themselves.
+
+## Answering "where are the logs?" without analyzing
+
+When the question is only *where something lives*, answer it and stop - do not run the
+analysis.
+
+**A saved run directory** (produced by `runner/regression.sh`, downloaded from a CI run, or
+archived by hand) - answer from the `Inputs available under {RUN_DIR}/` list of the matching
+prompt file, which maps every log, results dir and status database. Unpack archives first as in
+step 2.
+
+**A local `./ai_run.sh pytest` run against a dev cluster** - there is no run directory:
+
+- test artifacts are under the pytest temp dir;
+  `${TMPDIR:-${TEMP:-${TMP:-/tmp}}}/pytest-of-${LOGNAME:-$USER}/pytest-current` is a symlink to
+  the newest run, repointed by the next pytest invocation
+- cluster logs are in the *live* state dir of the running cluster, which is the parent dir of
+  `$CARDANO_NODE_SOCKET_PATH` - same file names as in `cluster_artifacts/` above
+- cluster artifacts are **not** copied into the pytest temp dir on a dev cluster unless
+  `FORCE_SAVE_CLUSTER_ARTIFACTS` is set, so read the live state dir instead

README.md

@@ -63,6 +63,10 @@ Or run the upgrade test suite:
 ./runner/load-gh-env.sh runner/env_nightly_upgrade CI_BYRON_CLUSTER=false ./runner/node_upgrade.sh
 ```
 
+### Analyzing Failures
+
+In Claude Code, `/analyze-failures [run_dir]` triages a finished run (`run_workdir` by default, also artifacts downloaded from a CI run) and answers where a run's logs and artifacts live.
+
 ---
 
 ## 🧪 Running Individual Tests with Custom Binaries

agent_docs/failure_analysis_prompt.md

@@ -7,8 +7,9 @@ The run directory is `{RUN_DIR}` (path is relative to the current working direct
 Inputs available under `{RUN_DIR}/` (use only what exists):
 
 - `{RUN_DIR}/allure-results/` — one JSON per test (status, statusDetails.message, statusDetails.trace, stdout/stderr attachments listed by name)
-- `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs with cluster logs, node stdouts, etc.
-- `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh` over cluster logs
+- `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs, cluster instance artifacts and scripts
+- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster<N>_<instance-id>/` — saved state dir of one cluster instance: per-node `*.stdout`/`*.stderr` and their rotated `*.stdout.<n>`/`*.stderr.<n>` (`bft1` for the first node, `pool<N>` for the rest; the pool count comes from the testnet config, don't assume three), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`, plus the `nodes/` and `shelley/` subdirs (per-node dirs, genesis files and keys). There is one such dir per instance start, so a respun instance number has several; `<instance-id>` matches the `started cluster instance '<id>'` line in `scheduling.log` - a name carrying an extra random suffix, or one with no matching `scheduling.log` line, is a fallback name used when the instance id was unavailable or the dir name already existed
+- `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh`: case-insensitive grep for `:error:|failed|failure` in `*.stdout`/`*.stderr` files under `testing_artifacts/`, with paths relative to that dir. Rotated logs (`*.stdout.<n>`, `*.stderr.<n>`) are **not** grepped, so a thin or empty `errors_all.log` doesn't prove there were no errors - grep the rotated logs directly when an early crash is suspected
 - `{RUN_DIR}/scheduling.log` — cluster instance manager log
 - `{RUN_DIR}/testrun-report.xml` — junit XML
 - `{RUN_DIR}/monitor.log` — system resource snapshots every 10 min
@@ -35,7 +36,8 @@ Steps:
 
 Known patterns:
 
-- `All cluster instances are dead.` — no cluster instance could start; usually caused by a `cardano-cli` argument change or a `cardano-node` configuration change. Inspect `scheduling.log` for the cluster startup failure details.
+- `All cluster instances are dead.` — no cluster instance could start; usually caused by a `cardano-cli` argument change or a `cardano-node` configuration change. Inspect `scheduling.log` and the `start-cluster.log` / `supervisord.log` / node `*.stderr` of the newest `cluster_artifacts/state-cluster*` dir for the startup failure details.
+- **Testnet under constant load** — if `tx-firehose.stderr`, `tx-centrifuge.stdout` or `tx-generator.stdout` are present in a `cluster_artifacts/state-cluster*` dir, a load generator was submitting transactions for the whole run. Expect timing-sensitive failures (tx submission timeouts, full mempool, slower block/epoch progress), and attribute errors these tools log to the load generator, not to the node under test.
 
 Constraints:
 

agent_docs/upgrade_failure_analysis_prompt.md

@@ -14,8 +14,9 @@ Inputs available under `{RUN_DIR}/` (use only what exists):
 
 - `{RUN_DIR}/allure-results-step1/`, `{RUN_DIR}/allure-results-step2/`, `{RUN_DIR}/allure-results-step3/` — one JSON per test per step (`status`, `statusDetails.message`, `statusDetails.trace`, stdout/stderr attachments listed by name)
 - `{RUN_DIR}/testrun-report-step1.html`, `…-step2.html`, `…-step3.html` — self-contained HTML reports (large; prefer the per-step allure JSON above)
-- `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs with cluster logs, node stdouts, etc. (shared across all steps)
-- `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh` over cluster logs (covers all steps)
+- `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs, cluster instance artifacts and scripts (shared across all steps)
+- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster<N>_<instance-id>/` — saved state dir of one cluster instance: per-node `*.stdout`/`*.stderr` and their rotated `*.stdout.<n>`/`*.stderr.<n>` (`bft1` for the first node, `pool<N>` for the rest; the pool count comes from the testnet config, don't assume three), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`, plus the `nodes/` and `shelley/` subdirs (per-node dirs, genesis files and keys). There is one such dir per instance start, so a respun instance number has several; `<instance-id>` matches the `started cluster instance '<id>'` line in `scheduling.log` - a name carrying an extra random suffix, or one with no matching `scheduling.log` line, is a fallback name used when the instance id was unavailable or the dir name already existed. Each step has its own `pytest-*` dir, so use the same oldest-first ordering as for `cm-status.db` to tell which step a dir belongs to
+- `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh`: case-insensitive grep for `:error:|failed|failure` in `*.stdout`/`*.stderr` files under `testing_artifacts/`, with paths relative to that dir (covers all steps). Rotated logs (`*.stdout.<n>`, `*.stderr.<n>`) are **not** grepped, so a thin or empty `errors_all.log` doesn't prove there were no errors - grep the rotated logs directly when an early crash is suspected
 - `{RUN_DIR}/scheduling.log` — cluster instance manager log
 - `{RUN_DIR}/testing_artifacts/pytest-*/cm-status.db` — cluster-management SQLite status databases, one per step ("test running", resource and flag records as they were at the end of that step); the `pytest-N` dir numbers don't map to steps, order the databases by modification time instead (oldest = step1); there should normally be three - when fewer are present, don't assume positions and correlate with which `allure-results-stepN/` dirs exist to decide which steps the databases belong to; query with `sqlite3 -readonly -header <db_file> 'SELECT * FROM overview ORDER BY instance_num, kind'`
 
@@ -28,14 +29,15 @@ Steps:
 2. Group failures by likely root cause (same exception class + message head, same node crash, same infra symptom). **Note which step(s) each group hits** — a failure that appears only in step2 or step3 is much more interesting than one that already fails in step1. Treat one node crash that flunks many tests as a single group.
 3. For each group: list affected tests (truncate to ~10 with a "+N more" tail), give the most informative 1–3 lines of error context, mark the step(s) affected, and classify as one of `node-bug | test-bug | infra-flake | env-issue | upgrade-regression | unknown` with a short justification. Use `upgrade-regression` when a test passes in step1 but fails in step2 or step3 — that is the signal this workflow exists to catch.
 4. Skim `{RUN_DIR}/errors_all.log` and `{RUN_DIR}/scheduling.log` for anything corroborating (node crash on restart, hard-fork failure, supervisord errors, OOM, repeated tracebacks). When failures look cluster-management related (dead cluster instances, tests stuck waiting for resources), query the `overview` view of the affected step's status database.
-5. If a whole step is missing its `allure-results-stepN/` dir, that step likely failed before pytest ran — call this out explicitly and check `errors_all.log` / the workflow log group output for the cause (commonly a `start-cluster` / `supervisord` / hard-fork failure).
+5. If a whole step is missing its `allure-results-stepN/` dir, that step likely failed before pytest ran — call this out explicitly and check `errors_all.log`, that step's `cluster_artifacts/state-cluster*/start-cluster.log` and `supervisord.log`, and the workflow log group output for the cause (commonly a `start-cluster` / `supervisord` / hard-fork failure).
 
 Known patterns:
 
 - **Step2 node failed to start** — typically a config-rewrite mismatch (genesis hashes, topology, `ExperimentalHardForksEnabled`) or pool3 still using `cardano-node-step1` against an incompatible config.
 - **Step3 hard-fork test failure** — `test_hardfork` fails or never raises the protocol version; later tests in step3 all then fail with stale protocol params.
 - **Sync stalls after restart** — `Failed to sync node` in workflow log; check whether pool1/pool3 PIDs are 0 or whether `syncProgress` never reaches `100.00`.
 - **`All cluster instances are dead.`** — no cluster instance could start; usually caused by a `cardano-cli` argument change or a `cardano-node` configuration change.
+- **Testnet under constant load** — if `tx-firehose.stderr`, `tx-centrifuge.stdout` or `tx-generator.stdout` are present in a `cluster_artifacts/state-cluster*` dir, a load generator was submitting transactions for the whole run. Expect timing-sensitive failures (tx submission timeouts, full mempool, slower block/epoch progress), and attribute errors these tools log to the load generator, not to the node under test.
 
 Constraints: