diff --git a/.claude/skills/analyze-failures/SKILL.md b/.claude/skills/analyze-failures/SKILL.md new file mode 100644 index 000000000..5b9989c62 --- /dev/null +++ b/.claude/skills/analyze-failures/SKILL.md @@ -0,0 +1,78 @@ +--- +name: analyze-failures +description: Triage a failed cardano-node-tests run, and answer questions about where a test run's logs and artifacts live. Use when the user asks to analyze, triage, or explain failures in a regression run, a node-upgrade run, a run_workdir, a saved run directory or artifacts downloaded from a CI run, and also when they ask where to find node logs, cluster logs, test artifacts, allure results or the cluster-management status database after running tests. Accepts an optional run directory argument (default `run_workdir`). +--- + +# Analyze test run failures + +Triage a finished cardano-node-tests run and report grouped failures with a likely root cause +for each group. + +## 1. Resolve the run directory + +The directory passed as the skill argument, or `run_workdir` if none was given. The path may be +relative to the current working directory or absolute; it can be a fresh run produced by +`runner/regression.sh`, artifacts downloaded from a CI run, or any saved historical run +directory. + +If the directory does not exist, stop and say so - do not guess another path. + +## 2. Unpack tarballs if needed + +A run directory produced locally holds both the unpacked result dirs and their `*.tar.xz` +archives (`runner/create_results.sh` keeps both). Artifacts **downloaded from a CI run** hold +only the archives. Each archive contains a single top-level directory of the same name, so +unpack any archive whose directory is missing before reading anything: + +```sh +tar -xf /testing_artifacts.tar.xz -C +``` + +## 3. Detect the testsuite + +Match on results present as a directory **or** as a tarball: + +- `allure-results-step*` -> **node-upgrade run**: follow `agent_docs/upgrade_failure_analysis_prompt.md`. +- otherwise `allure-results` -> **regression run**: follow `agent_docs/failure_analysis_prompt.md`. +- neither -> do **not** default to the regression prompt. An upgrade run that died before + step1's pytest produces no results at all and would be misread as a regression run. Report + what the directory does contain (`testrun-report-step*.html` and three `testing_artifacts/pytest-*` + dirs point at an upgrade run) and ask which testsuite it was. + +Say which one you detected before you start. + +## 4. Run the analysis + +Read the detected prompt file and follow it. Substitute the resolved run directory for every +`{RUN_DIR}` placeholder in it. + +Those prompt files are the single source of truth for the analysis - the same ones the CI +workflows use - so do not restate or reinterpret their instructions here. + +## 5. Report + +Report the findings in the conversation and stay available for follow-up questions. + +Do **not** write `failure_analysis.md`, cap the output length, or apply any other constraint +from `agent_docs/ci_analysis_prompt.md`. Those are CI-only and the workflows append them +themselves. + +## Answering "where are the logs?" without analyzing + +When the question is only *where something lives*, answer it and stop - do not run the +analysis. + +**A saved run directory** (produced by `runner/regression.sh`, downloaded from a CI run, or +archived by hand) - answer from the `Inputs available under {RUN_DIR}/` list of the matching +prompt file, which maps every log, results dir and status database. Unpack archives first as in +step 2. + +**A local `./ai_run.sh pytest` run against a dev cluster** - there is no run directory: + +- test artifacts are under the pytest temp dir; + `${TMPDIR:-${TEMP:-${TMP:-/tmp}}}/pytest-of-${LOGNAME:-$USER}/pytest-current` is a symlink to + the newest run, repointed by the next pytest invocation +- cluster logs are in the *live* state dir of the running cluster, which is the parent dir of + `$CARDANO_NODE_SOCKET_PATH` - same file names as in `cluster_artifacts/` above +- cluster artifacts are **not** copied into the pytest temp dir on a dev cluster unless + `FORCE_SAVE_CLUSTER_ARTIFACTS` is set, so read the live state dir instead diff --git a/README.md b/README.md index d79ab58b1..25d001a86 100644 --- a/README.md +++ b/README.md @@ -63,6 +63,10 @@ Or run the upgrade test suite: ./runner/load-gh-env.sh runner/env_nightly_upgrade CI_BYRON_CLUSTER=false ./runner/node_upgrade.sh ``` +### Analyzing Failures + +In Claude Code, `/analyze-failures [run_dir]` triages a finished run (`run_workdir` by default, also artifacts downloaded from a CI run) and answers where a run's logs and artifacts live. + --- ## ๐Ÿงช Running Individual Tests with Custom Binaries diff --git a/agent_docs/failure_analysis_prompt.md b/agent_docs/failure_analysis_prompt.md index 5df817617..9208fd395 100644 --- a/agent_docs/failure_analysis_prompt.md +++ b/agent_docs/failure_analysis_prompt.md @@ -7,8 +7,9 @@ The run directory is `{RUN_DIR}` (path is relative to the current working direct Inputs available under `{RUN_DIR}/` (use only what exists): - `{RUN_DIR}/allure-results/` โ€” one JSON per test (status, statusDetails.message, statusDetails.trace, stdout/stderr attachments listed by name) -- `{RUN_DIR}/testing_artifacts/` โ€” per-test artifact dirs with cluster logs, node stdouts, etc. -- `{RUN_DIR}/errors_all.log` โ€” output of `runner/grep_errors.sh` over cluster logs +- `{RUN_DIR}/testing_artifacts/` โ€” per-test artifact dirs, cluster instance artifacts and scripts +- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/` โ€” saved state dir of one cluster instance: per-node `*.stdout`/`*.stderr` and their rotated `*.stdout.`/`*.stderr.` (`bft1` for the first node, `pool` for the rest; the pool count comes from the testnet config, don't assume three), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`, plus the `nodes/` and `shelley/` subdirs (per-node dirs, genesis files and keys). There is one such dir per instance start, so a respun instance number has several; `` matches the `started cluster instance ''` line in `scheduling.log` - a name carrying an extra random suffix, or one with no matching `scheduling.log` line, is a fallback name used when the instance id was unavailable or the dir name already existed +- `{RUN_DIR}/errors_all.log` โ€” output of `runner/grep_errors.sh`: case-insensitive grep for `:error:|failed|failure` in `*.stdout`/`*.stderr` files under `testing_artifacts/`, with paths relative to that dir. Rotated logs (`*.stdout.`, `*.stderr.`) are **not** grepped, so a thin or empty `errors_all.log` doesn't prove there were no errors - grep the rotated logs directly when an early crash is suspected - `{RUN_DIR}/scheduling.log` โ€” cluster instance manager log - `{RUN_DIR}/testrun-report.xml` โ€” junit XML - `{RUN_DIR}/monitor.log` โ€” system resource snapshots every 10 min @@ -35,7 +36,8 @@ Steps: Known patterns: -- `All cluster instances are dead.` โ€” no cluster instance could start; usually caused by a `cardano-cli` argument change or a `cardano-node` configuration change. Inspect `scheduling.log` for the cluster startup failure details. +- `All cluster instances are dead.` โ€” no cluster instance could start; usually caused by a `cardano-cli` argument change or a `cardano-node` configuration change. Inspect `scheduling.log` and the `start-cluster.log` / `supervisord.log` / node `*.stderr` of the newest `cluster_artifacts/state-cluster*` dir for the startup failure details. +- **Testnet under constant load** โ€” if `tx-firehose.stderr`, `tx-centrifuge.stdout` or `tx-generator.stdout` are present in a `cluster_artifacts/state-cluster*` dir, a load generator was submitting transactions for the whole run. Expect timing-sensitive failures (tx submission timeouts, full mempool, slower block/epoch progress), and attribute errors these tools log to the load generator, not to the node under test. Constraints: diff --git a/agent_docs/upgrade_failure_analysis_prompt.md b/agent_docs/upgrade_failure_analysis_prompt.md index e0338e548..c0eace9cf 100644 --- a/agent_docs/upgrade_failure_analysis_prompt.md +++ b/agent_docs/upgrade_failure_analysis_prompt.md @@ -14,8 +14,9 @@ Inputs available under `{RUN_DIR}/` (use only what exists): - `{RUN_DIR}/allure-results-step1/`, `{RUN_DIR}/allure-results-step2/`, `{RUN_DIR}/allure-results-step3/` โ€” one JSON per test per step (`status`, `statusDetails.message`, `statusDetails.trace`, stdout/stderr attachments listed by name) - `{RUN_DIR}/testrun-report-step1.html`, `โ€ฆ-step2.html`, `โ€ฆ-step3.html` โ€” self-contained HTML reports (large; prefer the per-step allure JSON above) -- `{RUN_DIR}/testing_artifacts/` โ€” per-test artifact dirs with cluster logs, node stdouts, etc. (shared across all steps) -- `{RUN_DIR}/errors_all.log` โ€” output of `runner/grep_errors.sh` over cluster logs (covers all steps) +- `{RUN_DIR}/testing_artifacts/` โ€” per-test artifact dirs, cluster instance artifacts and scripts (shared across all steps) +- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/` โ€” saved state dir of one cluster instance: per-node `*.stdout`/`*.stderr` and their rotated `*.stdout.`/`*.stderr.` (`bft1` for the first node, `pool` for the rest; the pool count comes from the testnet config, don't assume three), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`, plus the `nodes/` and `shelley/` subdirs (per-node dirs, genesis files and keys). There is one such dir per instance start, so a respun instance number has several; `` matches the `started cluster instance ''` line in `scheduling.log` - a name carrying an extra random suffix, or one with no matching `scheduling.log` line, is a fallback name used when the instance id was unavailable or the dir name already existed. Each step has its own `pytest-*` dir, so use the same oldest-first ordering as for `cm-status.db` to tell which step a dir belongs to +- `{RUN_DIR}/errors_all.log` โ€” output of `runner/grep_errors.sh`: case-insensitive grep for `:error:|failed|failure` in `*.stdout`/`*.stderr` files under `testing_artifacts/`, with paths relative to that dir (covers all steps). Rotated logs (`*.stdout.`, `*.stderr.`) are **not** grepped, so a thin or empty `errors_all.log` doesn't prove there were no errors - grep the rotated logs directly when an early crash is suspected - `{RUN_DIR}/scheduling.log` โ€” cluster instance manager log - `{RUN_DIR}/testing_artifacts/pytest-*/cm-status.db` โ€” cluster-management SQLite status databases, one per step ("test running", resource and flag records as they were at the end of that step); the `pytest-N` dir numbers don't map to steps, order the databases by modification time instead (oldest = step1); there should normally be three - when fewer are present, don't assume positions and correlate with which `allure-results-stepN/` dirs exist to decide which steps the databases belong to; query with `sqlite3 -readonly -header 'SELECT * FROM overview ORDER BY instance_num, kind'` @@ -28,7 +29,7 @@ Steps: 2. Group failures by likely root cause (same exception class + message head, same node crash, same infra symptom). **Note which step(s) each group hits** โ€” a failure that appears only in step2 or step3 is much more interesting than one that already fails in step1. Treat one node crash that flunks many tests as a single group. 3. For each group: list affected tests (truncate to ~10 with a "+N more" tail), give the most informative 1โ€“3 lines of error context, mark the step(s) affected, and classify as one of `node-bug | test-bug | infra-flake | env-issue | upgrade-regression | unknown` with a short justification. Use `upgrade-regression` when a test passes in step1 but fails in step2 or step3 โ€” that is the signal this workflow exists to catch. 4. Skim `{RUN_DIR}/errors_all.log` and `{RUN_DIR}/scheduling.log` for anything corroborating (node crash on restart, hard-fork failure, supervisord errors, OOM, repeated tracebacks). When failures look cluster-management related (dead cluster instances, tests stuck waiting for resources), query the `overview` view of the affected step's status database. -5. If a whole step is missing its `allure-results-stepN/` dir, that step likely failed before pytest ran โ€” call this out explicitly and check `errors_all.log` / the workflow log group output for the cause (commonly a `start-cluster` / `supervisord` / hard-fork failure). +5. If a whole step is missing its `allure-results-stepN/` dir, that step likely failed before pytest ran โ€” call this out explicitly and check `errors_all.log`, that step's `cluster_artifacts/state-cluster*/start-cluster.log` and `supervisord.log`, and the workflow log group output for the cause (commonly a `start-cluster` / `supervisord` / hard-fork failure). Known patterns: @@ -36,6 +37,7 @@ Known patterns: - **Step3 hard-fork test failure** โ€” `test_hardfork` fails or never raises the protocol version; later tests in step3 all then fail with stale protocol params. - **Sync stalls after restart** โ€” `Failed to sync node` in workflow log; check whether pool1/pool3 PIDs are 0 or whether `syncProgress` never reaches `100.00`. - **`All cluster instances are dead.`** โ€” no cluster instance could start; usually caused by a `cardano-cli` argument change or a `cardano-node` configuration change. +- **Testnet under constant load** โ€” if `tx-firehose.stderr`, `tx-centrifuge.stdout` or `tx-generator.stdout` are present in a `cluster_artifacts/state-cluster*` dir, a load generator was submitting transactions for the whole run. Expect timing-sensitive failures (tx submission timeouts, full mempool, slower block/epoch progress), and attribute errors these tools log to the load generator, not to the node under test. Constraints: diff --git a/scripts/analyze_failures.sh b/scripts/analyze_failures.sh deleted file mode 100755 index 68fd7b15b..000000000 --- a/scripts/analyze_failures.sh +++ /dev/null @@ -1,68 +0,0 @@ -#!/usr/bin/env bash -# Run preliminary failure analysis on a regression-run directory using any -# coding-agent CLI that accepts a prompt as its first positional argument. -# -# Usage: -# scripts/analyze_failures.sh [run_dir] -# -# run_dir defaults to ./run_workdir. It can point at a fresh run produced by -# `runner/regression.sh` or at any saved historical run directory (relative -# or absolute path). -# -# By default the agent is started in interactive mode, seeded with the -# analysis prompt, so you can read the findings and ask follow-up questions. -# The CI-only constraints (write failure_analysis.md, length cap, "Start now.") -# are intentionally NOT included here; those live in agent_docs/ci_analysis_prompt.md -# and are appended only by the CI workflows. -# -# Env vars: -# AGENT_CMD Command used to invoke the agent. Default: "claude". -# The prompt is passed as the last positional argument. -# Examples: -# AGENT_CMD="claude" # interactive (default) -# AGENT_CMD="claude -p" # non-interactive print mode -# AGENT_CMD="claude --model opus" # interactive, model override -# AGENT_CMD="gemini" -# AGENT_CMD="codex" -# PROMPT_FILE Override prompt auto-detection. Path to a prompt template -# containing {RUN_DIR} placeholders. - -set -euo pipefail - -run_dir="${1:-run_workdir}" -AGENT_CMD="${AGENT_CMD:-claude}" - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -repo_root="$(cd "${script_dir}/.." && pwd)" - -if [ ! -d "${run_dir}" ]; then - echo "Run directory not found: ${run_dir}" >&2 - exit 1 -fi - -# Auto-detect testsuite from run_dir contents unless PROMPT_FILE is set. -# Upgrade runs produce per-step allure result dirs/tarballs; regression runs -# produce a single allure-results dir/tarball. -if [ -z "${PROMPT_FILE:-}" ]; then - if compgen -G "${run_dir}/allure-results-step*" > /dev/null; then - PROMPT_FILE="${repo_root}/agent_docs/upgrade_failure_analysis_prompt.md" - echo "Detected node-upgrade run; using upgrade prompt." >&2 - else - PROMPT_FILE="${repo_root}/agent_docs/failure_analysis_prompt.md" - echo "Detected regression run; using regression prompt." >&2 - fi -fi - -if [ ! -f "${PROMPT_FILE}" ]; then - echo "Prompt file not found: ${PROMPT_FILE}" >&2 - exit 1 -fi - -# Substitute {RUN_DIR} in the prompt template, then seed the agent with it. -prompt_content="$(cat "${PROMPT_FILE}")" -prompt_content="${prompt_content//\{RUN_DIR\}/${run_dir}}" - -# Pass the prompt as the last positional argument so interactive agents start -# seeded with it (stdin piping would put claude into non-interactive mode). -# shellcheck disable=SC2086 -exec $AGENT_CMD "${prompt_content}"