From 3871c5f868b98127e79079faeeeb0a7d8b225a45 Mon Sep 17 00:00:00 2001 From: Martin Kourim Date: Fri, 21 Aug 2026 09:46:13 +0200 Subject: [PATCH 1/4] feat: add analyze-failures skill Replace `scripts/analyze_failures.sh` with a Claude Code skill that detects a regression vs node-upgrade run and follows the matching prompt from `agent_docs/`. The skill also answers where a run's logs and artifacts live, both for saved run directories and for local runs against a dev cluster. Sharpen both analysis prompts: describe the `cluster_artifacts/` state dirs and their contents, say what `errors_all.log` actually greps, point at `start-cluster.log` / `supervisord.log` when the cluster fails to start, and note that load generator logs mean the testnet was under constant load during the run. --- .claude/skills/analyze-failures/SKILL.md | 68 +++++++++++++++++++ agent_docs/failure_analysis_prompt.md | 8 ++- agent_docs/upgrade_failure_analysis_prompt.md | 8 ++- scripts/analyze_failures.sh | 68 ------------------- 4 files changed, 78 insertions(+), 74 deletions(-) create mode 100644 .claude/skills/analyze-failures/SKILL.md delete mode 100755 scripts/analyze_failures.sh diff --git a/.claude/skills/analyze-failures/SKILL.md b/.claude/skills/analyze-failures/SKILL.md new file mode 100644 index 000000000..6ad9c0566 --- /dev/null +++ b/.claude/skills/analyze-failures/SKILL.md @@ -0,0 +1,68 @@ +--- +name: analyze-failures +description: Triage a failed cardano-node-tests run, and answer questions about where a test run's logs and artifacts live. Use when the user asks to analyze, triage, or explain failures in a regression run, a node-upgrade run, or a run_workdir / saved run directory, and also when they ask where to find node logs, cluster logs, test artifacts, allure results or the cluster-management status database after running tests. Accepts an optional run directory argument (default `run_workdir`). +--- + +# Analyze test run failures + +Triage a finished cardano-node-tests run and report grouped failures with a likely root cause for each group. + +## 1. Resolve the run directory + +`$1` (the skill argument), or `run_workdir` if no argument was given. The path may be +relative to the current working directory or absolute; it can be a fresh run produced by +`runner/regression.sh` or any saved historical run directory. + +If the directory does not exist, stop and say so - do not guess another path. + +## 2. Detect the testsuite + +```sh +ls -d /allure-results-step* 2>/dev/null +``` + +- Any `allure-results-step*` entries -> **node-upgrade run**: follow `agent_docs/upgrade_failure_analysis_prompt.md`. +- Otherwise -> **regression run**: follow `agent_docs/failure_analysis_prompt.md`. + +Say which one you detected before you start. + +## 3. Run the analysis + +Read the detected prompt file and follow it. Substitute the resolved run directory for every +`{RUN_DIR}` placeholder in it. + +Those prompt files are the single source of truth for the analysis - the same ones the CI +workflows use - so do not restate or reinterpret their instructions here. + +## 4. Report + +Report the findings in the conversation and stay available for follow-up questions. + +Do **not** write `failure_analysis.md`, cap the output length, or apply any other constraint +from `agent_docs/ci_analysis_prompt.md`. Those are CI-only and the workflows append them +themselves. + +## Answering "where are the logs?" without analyzing + +When the question is only *where something lives*, answer it and stop - do not run the +analysis. The `Inputs available under {RUN_DIR}/` list of the matching prompt file is the +authoritative map for a saved run; the two cases below say which map applies. + +**A saved run directory** (produced by `runner/regression.sh`, downloaded from a CI run, or +archived by hand) - use the prompt file's inputs list. The parts people ask for most: + +- test results, one JSON per test -> `allure-results/` (or `allure-results-step/`) +- node logs of a cluster instance -> `testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/{bft1,pool1,pool2,pool3}.{stdout,stderr}` +- why a cluster failed to start -> `start-cluster.log`, `supervisord.log` in the same dir +- all errors at once -> `errors_all.log` +- which test held which cluster resource -> `testing_artifacts/pytest-*/cm-status.db` +- files a test itself produced (tx bodies, keys, ...) -> `testing_artifacts/pytest-*//` + +**A local `./ai_run.sh pytest` run against a dev cluster** - there is no run directory: + +- test artifacts are under the pytest temp dir; `$TMPDIR/pytest-of-$USER/pytest-current` is a + symlink to the newest run (repointed by the next pytest invocation) +- cluster logs are in the *live* state dir of the running cluster, which is the parent dir of + `$CARDANO_NODE_SOCKET_PATH` - same file names as in `cluster_artifacts/` above +- cluster artifacts are **not** copied into the pytest temp dir on a dev cluster unless + `FORCE_SAVE_CLUSTER_ARTIFACTS` is set, so read the live state dir instead diff --git a/agent_docs/failure_analysis_prompt.md b/agent_docs/failure_analysis_prompt.md index 5df817617..d4390db4f 100644 --- a/agent_docs/failure_analysis_prompt.md +++ b/agent_docs/failure_analysis_prompt.md @@ -7,8 +7,9 @@ The run directory is `{RUN_DIR}` (path is relative to the current working direct Inputs available under `{RUN_DIR}/` (use only what exists): - `{RUN_DIR}/allure-results/` — one JSON per test (status, statusDetails.message, statusDetails.trace, stdout/stderr attachments listed by name) -- `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs with cluster logs, node stdouts, etc. -- `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh` over cluster logs +- `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs, cluster instance artifacts and scripts +- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/` — saved state dir of one cluster instance: node `*.stdout`/`*.stderr` (`bft1`, `pool1`..`pool3`), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`. There is one such dir per instance start, so a respun instance number has several; `` matches the `started cluster instance ''` line in `scheduling.log` +- `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh`: case-insensitive grep for `:error:|failed|failure` in every `*.stdout`/`*.stderr` under `testing_artifacts/`, with paths relative to that dir - `{RUN_DIR}/scheduling.log` — cluster instance manager log - `{RUN_DIR}/testrun-report.xml` — junit XML - `{RUN_DIR}/monitor.log` — system resource snapshots every 10 min @@ -35,7 +36,8 @@ Steps: Known patterns: -- `All cluster instances are dead.` — no cluster instance could start; usually caused by a `cardano-cli` argument change or a `cardano-node` configuration change. Inspect `scheduling.log` for the cluster startup failure details. +- `All cluster instances are dead.` — no cluster instance could start; usually caused by a `cardano-cli` argument change or a `cardano-node` configuration change. Inspect `scheduling.log` and the `start-cluster.log` / `supervisord.log` / node `*.stderr` of the newest `cluster_artifacts/state-cluster*` dir for the startup failure details. +- **Testnet under constant load** — if `tx-firehose.stderr`, `tx-centrifuge.stdout` or `tx-generator.stdout` are present in a `cluster_artifacts/state-cluster*` dir, a load generator was submitting transactions for the whole run. Expect timing-sensitive failures (tx submission timeouts, full mempool, slower block/epoch progress), and attribute errors these tools log to the load generator, not to the node under test. Constraints: diff --git a/agent_docs/upgrade_failure_analysis_prompt.md b/agent_docs/upgrade_failure_analysis_prompt.md index e0338e548..8d4319165 100644 --- a/agent_docs/upgrade_failure_analysis_prompt.md +++ b/agent_docs/upgrade_failure_analysis_prompt.md @@ -14,8 +14,9 @@ Inputs available under `{RUN_DIR}/` (use only what exists): - `{RUN_DIR}/allure-results-step1/`, `{RUN_DIR}/allure-results-step2/`, `{RUN_DIR}/allure-results-step3/` — one JSON per test per step (`status`, `statusDetails.message`, `statusDetails.trace`, stdout/stderr attachments listed by name) - `{RUN_DIR}/testrun-report-step1.html`, `…-step2.html`, `…-step3.html` — self-contained HTML reports (large; prefer the per-step allure JSON above) -- `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs with cluster logs, node stdouts, etc. (shared across all steps) -- `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh` over cluster logs (covers all steps) +- `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs, cluster instance artifacts and scripts (shared across all steps) +- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/` — saved state dir of one cluster instance: node `*.stdout`/`*.stderr` (`bft1`, `pool1`..`pool3`), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`. There is one such dir per instance start, so a respun instance number has several; `` matches the `started cluster instance ''` line in `scheduling.log`. Each step has its own `pytest-*` dir, so use the same oldest-first ordering as for `cm-status.db` to tell which step a dir belongs to +- `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh`: case-insensitive grep for `:error:|failed|failure` in every `*.stdout`/`*.stderr` under `testing_artifacts/`, with paths relative to that dir (covers all steps) - `{RUN_DIR}/scheduling.log` — cluster instance manager log - `{RUN_DIR}/testing_artifacts/pytest-*/cm-status.db` — cluster-management SQLite status databases, one per step ("test running", resource and flag records as they were at the end of that step); the `pytest-N` dir numbers don't map to steps, order the databases by modification time instead (oldest = step1); there should normally be three - when fewer are present, don't assume positions and correlate with which `allure-results-stepN/` dirs exist to decide which steps the databases belong to; query with `sqlite3 -readonly -header 'SELECT * FROM overview ORDER BY instance_num, kind'` @@ -28,7 +29,7 @@ Steps: 2. Group failures by likely root cause (same exception class + message head, same node crash, same infra symptom). **Note which step(s) each group hits** — a failure that appears only in step2 or step3 is much more interesting than one that already fails in step1. Treat one node crash that flunks many tests as a single group. 3. For each group: list affected tests (truncate to ~10 with a "+N more" tail), give the most informative 1–3 lines of error context, mark the step(s) affected, and classify as one of `node-bug | test-bug | infra-flake | env-issue | upgrade-regression | unknown` with a short justification. Use `upgrade-regression` when a test passes in step1 but fails in step2 or step3 — that is the signal this workflow exists to catch. 4. Skim `{RUN_DIR}/errors_all.log` and `{RUN_DIR}/scheduling.log` for anything corroborating (node crash on restart, hard-fork failure, supervisord errors, OOM, repeated tracebacks). When failures look cluster-management related (dead cluster instances, tests stuck waiting for resources), query the `overview` view of the affected step's status database. -5. If a whole step is missing its `allure-results-stepN/` dir, that step likely failed before pytest ran — call this out explicitly and check `errors_all.log` / the workflow log group output for the cause (commonly a `start-cluster` / `supervisord` / hard-fork failure). +5. If a whole step is missing its `allure-results-stepN/` dir, that step likely failed before pytest ran — call this out explicitly and check `errors_all.log`, that step's `cluster_artifacts/state-cluster*/start-cluster.log` and `supervisord.log`, and the workflow log group output for the cause (commonly a `start-cluster` / `supervisord` / hard-fork failure). Known patterns: @@ -36,6 +37,7 @@ Known patterns: - **Step3 hard-fork test failure** — `test_hardfork` fails or never raises the protocol version; later tests in step3 all then fail with stale protocol params. - **Sync stalls after restart** — `Failed to sync node` in workflow log; check whether pool1/pool3 PIDs are 0 or whether `syncProgress` never reaches `100.00`. - **`All cluster instances are dead.`** — no cluster instance could start; usually caused by a `cardano-cli` argument change or a `cardano-node` configuration change. +- **Testnet under constant load** — if `tx-firehose.stderr`, `tx-centrifuge.stdout` or `tx-generator.stdout` are present in a `cluster_artifacts/state-cluster*` dir, a load generator was submitting transactions for the whole run. Expect timing-sensitive failures (tx submission timeouts, full mempool, slower block/epoch progress), and attribute errors these tools log to the load generator, not to the node under test. Constraints: diff --git a/scripts/analyze_failures.sh b/scripts/analyze_failures.sh deleted file mode 100755 index 68fd7b15b..000000000 --- a/scripts/analyze_failures.sh +++ /dev/null @@ -1,68 +0,0 @@ -#!/usr/bin/env bash -# Run preliminary failure analysis on a regression-run directory using any -# coding-agent CLI that accepts a prompt as its first positional argument. -# -# Usage: -# scripts/analyze_failures.sh [run_dir] -# -# run_dir defaults to ./run_workdir. It can point at a fresh run produced by -# `runner/regression.sh` or at any saved historical run directory (relative -# or absolute path). -# -# By default the agent is started in interactive mode, seeded with the -# analysis prompt, so you can read the findings and ask follow-up questions. -# The CI-only constraints (write failure_analysis.md, length cap, "Start now.") -# are intentionally NOT included here; those live in agent_docs/ci_analysis_prompt.md -# and are appended only by the CI workflows. -# -# Env vars: -# AGENT_CMD Command used to invoke the agent. Default: "claude". -# The prompt is passed as the last positional argument. -# Examples: -# AGENT_CMD="claude" # interactive (default) -# AGENT_CMD="claude -p" # non-interactive print mode -# AGENT_CMD="claude --model opus" # interactive, model override -# AGENT_CMD="gemini" -# AGENT_CMD="codex" -# PROMPT_FILE Override prompt auto-detection. Path to a prompt template -# containing {RUN_DIR} placeholders. - -set -euo pipefail - -run_dir="${1:-run_workdir}" -AGENT_CMD="${AGENT_CMD:-claude}" - -script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -repo_root="$(cd "${script_dir}/.." && pwd)" - -if [ ! -d "${run_dir}" ]; then - echo "Run directory not found: ${run_dir}" >&2 - exit 1 -fi - -# Auto-detect testsuite from run_dir contents unless PROMPT_FILE is set. -# Upgrade runs produce per-step allure result dirs/tarballs; regression runs -# produce a single allure-results dir/tarball. -if [ -z "${PROMPT_FILE:-}" ]; then - if compgen -G "${run_dir}/allure-results-step*" > /dev/null; then - PROMPT_FILE="${repo_root}/agent_docs/upgrade_failure_analysis_prompt.md" - echo "Detected node-upgrade run; using upgrade prompt." >&2 - else - PROMPT_FILE="${repo_root}/agent_docs/failure_analysis_prompt.md" - echo "Detected regression run; using regression prompt." >&2 - fi -fi - -if [ ! -f "${PROMPT_FILE}" ]; then - echo "Prompt file not found: ${PROMPT_FILE}" >&2 - exit 1 -fi - -# Substitute {RUN_DIR} in the prompt template, then seed the agent with it. -prompt_content="$(cat "${PROMPT_FILE}")" -prompt_content="${prompt_content//\{RUN_DIR\}/${run_dir}}" - -# Pass the prompt as the last positional argument so interactive agents start -# seeded with it (stdin piping would put claude into non-interactive mode). -# shellcheck disable=SC2086 -exec $AGENT_CMD "${prompt_content}" From a1cf9108f04501a169c82b2eedc6dc6c4844a73f Mon Sep 17 00:00:00 2001 From: Martin Kourim Date: Fri, 21 Aug 2026 09:55:28 +0200 Subject: [PATCH 2/4] fix: address review of analyze-failures skill Unpack step: artifacts downloaded from a CI run contain only the `*.tar.xz` archives, while a locally produced run dir keeps both the archives and the unpacked dirs, so unpack what is missing first. Detection: match results as dir or tarball, and stop asking instead of falling back to the regression prompt - an upgrade run that dies before step1's pytest produces no results at all. Drop the duplicated path map in favour of the prompt files' inputs list, use the same temp dir fallbacks as `runner/copy_artifacts.sh`, and stop implying that `$1` gets substituted in the skill body. Prompts: don't hardcode a 3-pool testnet, list the rotated logs and the `nodes/` and `shelley/` subdirs that `_copy_state_dir_content` also saves, note the fallback names of the state dirs, and warn that `grep_errors.sh` skips rotated logs. --- .claude/skills/analyze-failures/SKILL.md | 54 +++++++++++-------- agent_docs/failure_analysis_prompt.md | 4 +- agent_docs/upgrade_failure_analysis_prompt.md | 4 +- 3 files changed, 36 insertions(+), 26 deletions(-) diff --git a/.claude/skills/analyze-failures/SKILL.md b/.claude/skills/analyze-failures/SKILL.md index 6ad9c0566..5b9989c62 100644 --- a/.claude/skills/analyze-failures/SKILL.md +++ b/.claude/skills/analyze-failures/SKILL.md @@ -1,32 +1,47 @@ --- name: analyze-failures -description: Triage a failed cardano-node-tests run, and answer questions about where a test run's logs and artifacts live. Use when the user asks to analyze, triage, or explain failures in a regression run, a node-upgrade run, or a run_workdir / saved run directory, and also when they ask where to find node logs, cluster logs, test artifacts, allure results or the cluster-management status database after running tests. Accepts an optional run directory argument (default `run_workdir`). +description: Triage a failed cardano-node-tests run, and answer questions about where a test run's logs and artifacts live. Use when the user asks to analyze, triage, or explain failures in a regression run, a node-upgrade run, a run_workdir, a saved run directory or artifacts downloaded from a CI run, and also when they ask where to find node logs, cluster logs, test artifacts, allure results or the cluster-management status database after running tests. Accepts an optional run directory argument (default `run_workdir`). --- # Analyze test run failures -Triage a finished cardano-node-tests run and report grouped failures with a likely root cause for each group. +Triage a finished cardano-node-tests run and report grouped failures with a likely root cause +for each group. ## 1. Resolve the run directory -`$1` (the skill argument), or `run_workdir` if no argument was given. The path may be +The directory passed as the skill argument, or `run_workdir` if none was given. The path may be relative to the current working directory or absolute; it can be a fresh run produced by -`runner/regression.sh` or any saved historical run directory. +`runner/regression.sh`, artifacts downloaded from a CI run, or any saved historical run +directory. If the directory does not exist, stop and say so - do not guess another path. -## 2. Detect the testsuite +## 2. Unpack tarballs if needed + +A run directory produced locally holds both the unpacked result dirs and their `*.tar.xz` +archives (`runner/create_results.sh` keeps both). Artifacts **downloaded from a CI run** hold +only the archives. Each archive contains a single top-level directory of the same name, so +unpack any archive whose directory is missing before reading anything: ```sh -ls -d /allure-results-step* 2>/dev/null +tar -xf /testing_artifacts.tar.xz -C ``` -- Any `allure-results-step*` entries -> **node-upgrade run**: follow `agent_docs/upgrade_failure_analysis_prompt.md`. -- Otherwise -> **regression run**: follow `agent_docs/failure_analysis_prompt.md`. +## 3. Detect the testsuite + +Match on results present as a directory **or** as a tarball: + +- `allure-results-step*` -> **node-upgrade run**: follow `agent_docs/upgrade_failure_analysis_prompt.md`. +- otherwise `allure-results` -> **regression run**: follow `agent_docs/failure_analysis_prompt.md`. +- neither -> do **not** default to the regression prompt. An upgrade run that died before + step1's pytest produces no results at all and would be misread as a regression run. Report + what the directory does contain (`testrun-report-step*.html` and three `testing_artifacts/pytest-*` + dirs point at an upgrade run) and ask which testsuite it was. Say which one you detected before you start. -## 3. Run the analysis +## 4. Run the analysis Read the detected prompt file and follow it. Substitute the resolved run directory for every `{RUN_DIR}` placeholder in it. @@ -34,7 +49,7 @@ Read the detected prompt file and follow it. Substitute the resolved run directo Those prompt files are the single source of truth for the analysis - the same ones the CI workflows use - so do not restate or reinterpret their instructions here. -## 4. Report +## 5. Report Report the findings in the conversation and stay available for follow-up questions. @@ -45,23 +60,18 @@ themselves. ## Answering "where are the logs?" without analyzing When the question is only *where something lives*, answer it and stop - do not run the -analysis. The `Inputs available under {RUN_DIR}/` list of the matching prompt file is the -authoritative map for a saved run; the two cases below say which map applies. +analysis. **A saved run directory** (produced by `runner/regression.sh`, downloaded from a CI run, or -archived by hand) - use the prompt file's inputs list. The parts people ask for most: - -- test results, one JSON per test -> `allure-results/` (or `allure-results-step/`) -- node logs of a cluster instance -> `testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/{bft1,pool1,pool2,pool3}.{stdout,stderr}` -- why a cluster failed to start -> `start-cluster.log`, `supervisord.log` in the same dir -- all errors at once -> `errors_all.log` -- which test held which cluster resource -> `testing_artifacts/pytest-*/cm-status.db` -- files a test itself produced (tx bodies, keys, ...) -> `testing_artifacts/pytest-*//` +archived by hand) - answer from the `Inputs available under {RUN_DIR}/` list of the matching +prompt file, which maps every log, results dir and status database. Unpack archives first as in +step 2. **A local `./ai_run.sh pytest` run against a dev cluster** - there is no run directory: -- test artifacts are under the pytest temp dir; `$TMPDIR/pytest-of-$USER/pytest-current` is a - symlink to the newest run (repointed by the next pytest invocation) +- test artifacts are under the pytest temp dir; + `${TMPDIR:-${TEMP:-${TMP:-/tmp}}}/pytest-of-${LOGNAME:-$USER}/pytest-current` is a symlink to + the newest run, repointed by the next pytest invocation - cluster logs are in the *live* state dir of the running cluster, which is the parent dir of `$CARDANO_NODE_SOCKET_PATH` - same file names as in `cluster_artifacts/` above - cluster artifacts are **not** copied into the pytest temp dir on a dev cluster unless diff --git a/agent_docs/failure_analysis_prompt.md b/agent_docs/failure_analysis_prompt.md index d4390db4f..33e6669f1 100644 --- a/agent_docs/failure_analysis_prompt.md +++ b/agent_docs/failure_analysis_prompt.md @@ -8,8 +8,8 @@ Inputs available under `{RUN_DIR}/` (use only what exists): - `{RUN_DIR}/allure-results/` — one JSON per test (status, statusDetails.message, statusDetails.trace, stdout/stderr attachments listed by name) - `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs, cluster instance artifacts and scripts -- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/` — saved state dir of one cluster instance: node `*.stdout`/`*.stderr` (`bft1`, `pool1`..`pool3`), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`. There is one such dir per instance start, so a respun instance number has several; `` matches the `started cluster instance ''` line in `scheduling.log` -- `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh`: case-insensitive grep for `:error:|failed|failure` in every `*.stdout`/`*.stderr` under `testing_artifacts/`, with paths relative to that dir +- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/` — saved state dir of one cluster instance: per-node `*.stdout`/`*.stderr` and their rotated `*.stdout.`/`*.stderr.` (node names come from the testnet template, e.g. `bft1` and `pool` - don't assume a fixed pool count), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`, plus the `nodes/` and `shelley/` subdirs (per-node dirs, genesis files and keys). There is one such dir per instance start, so a respun instance number has several; `` matches the `started cluster instance ''` line in `scheduling.log` - a name carrying an extra random suffix, or one with no matching `scheduling.log` line, is a fallback name used when the instance id was unavailable or the dir name already existed +- `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh`: case-insensitive grep for `:error:|failed|failure` in `*.stdout`/`*.stderr` files under `testing_artifacts/`, with paths relative to that dir. Rotated logs (`*.stdout.`, `*.stderr.`) are **not** grepped, so a thin or empty `errors_all.log` doesn't prove there were no errors - grep the rotated logs directly when an early crash is suspected - `{RUN_DIR}/scheduling.log` — cluster instance manager log - `{RUN_DIR}/testrun-report.xml` — junit XML - `{RUN_DIR}/monitor.log` — system resource snapshots every 10 min diff --git a/agent_docs/upgrade_failure_analysis_prompt.md b/agent_docs/upgrade_failure_analysis_prompt.md index 8d4319165..b6124a116 100644 --- a/agent_docs/upgrade_failure_analysis_prompt.md +++ b/agent_docs/upgrade_failure_analysis_prompt.md @@ -15,8 +15,8 @@ Inputs available under `{RUN_DIR}/` (use only what exists): - `{RUN_DIR}/allure-results-step1/`, `{RUN_DIR}/allure-results-step2/`, `{RUN_DIR}/allure-results-step3/` — one JSON per test per step (`status`, `statusDetails.message`, `statusDetails.trace`, stdout/stderr attachments listed by name) - `{RUN_DIR}/testrun-report-step1.html`, `…-step2.html`, `…-step3.html` — self-contained HTML reports (large; prefer the per-step allure JSON above) - `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs, cluster instance artifacts and scripts (shared across all steps) -- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/` — saved state dir of one cluster instance: node `*.stdout`/`*.stderr` (`bft1`, `pool1`..`pool3`), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`. There is one such dir per instance start, so a respun instance number has several; `` matches the `started cluster instance ''` line in `scheduling.log`. Each step has its own `pytest-*` dir, so use the same oldest-first ordering as for `cm-status.db` to tell which step a dir belongs to -- `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh`: case-insensitive grep for `:error:|failed|failure` in every `*.stdout`/`*.stderr` under `testing_artifacts/`, with paths relative to that dir (covers all steps) +- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/` — saved state dir of one cluster instance: per-node `*.stdout`/`*.stderr` and their rotated `*.stdout.`/`*.stderr.` (node names come from the testnet template, e.g. `bft1` and `pool` - don't assume a fixed pool count), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`, plus the `nodes/` and `shelley/` subdirs (per-node dirs, genesis files and keys). There is one such dir per instance start, so a respun instance number has several; `` matches the `started cluster instance ''` line in `scheduling.log` - a name carrying an extra random suffix, or one with no matching `scheduling.log` line, is a fallback name used when the instance id was unavailable or the dir name already existed. Each step has its own `pytest-*` dir, so use the same oldest-first ordering as for `cm-status.db` to tell which step a dir belongs to +- `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh`: case-insensitive grep for `:error:|failed|failure` in `*.stdout`/`*.stderr` files under `testing_artifacts/`, with paths relative to that dir (covers all steps). Rotated logs (`*.stdout.`, `*.stderr.`) are **not** grepped, so a thin or empty `errors_all.log` doesn't prove there were no errors - grep the rotated logs directly when an early crash is suspected - `{RUN_DIR}/scheduling.log` — cluster instance manager log - `{RUN_DIR}/testing_artifacts/pytest-*/cm-status.db` — cluster-management SQLite status databases, one per step ("test running", resource and flag records as they were at the end of that step); the `pytest-N` dir numbers don't map to steps, order the databases by modification time instead (oldest = step1); there should normally be three - when fewer are present, don't assume positions and correlate with which `allure-results-stepN/` dirs exist to decide which steps the databases belong to; query with `sqlite3 -readonly -header 'SELECT * FROM overview ORDER BY instance_num, kind'` From 8fae5641ba8b03c6d50c938fb05104baea328902 Mon Sep 17 00:00:00 2001 From: Martin Kourim Date: Fri, 21 Aug 2026 09:58:59 +0200 Subject: [PATCH 3/4] docs: mention the analyze-failures skill in README --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index d79ab58b1..25d001a86 100644 --- a/README.md +++ b/README.md @@ -63,6 +63,10 @@ Or run the upgrade test suite: ./runner/load-gh-env.sh runner/env_nightly_upgrade CI_BYRON_CLUSTER=false ./runner/node_upgrade.sh ``` +### Analyzing Failures + +In Claude Code, `/analyze-failures [run_dir]` triages a finished run (`run_workdir` by default, also artifacts downloaded from a CI run) and answers where a run's logs and artifacts live. + --- ## 🧪 Running Individual Tests with Custom Binaries From 244682e395cb042a600a69271ed1631e4340606a Mon Sep 17 00:00:00 2001 From: Martin Kourim Date: Fri, 21 Aug 2026 12:07:05 +0200 Subject: [PATCH 4/4] fix: don't imply a fixed pool count in the prompts --- agent_docs/failure_analysis_prompt.md | 2 +- agent_docs/upgrade_failure_analysis_prompt.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/agent_docs/failure_analysis_prompt.md b/agent_docs/failure_analysis_prompt.md index 33e6669f1..9208fd395 100644 --- a/agent_docs/failure_analysis_prompt.md +++ b/agent_docs/failure_analysis_prompt.md @@ -8,7 +8,7 @@ Inputs available under `{RUN_DIR}/` (use only what exists): - `{RUN_DIR}/allure-results/` — one JSON per test (status, statusDetails.message, statusDetails.trace, stdout/stderr attachments listed by name) - `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs, cluster instance artifacts and scripts -- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/` — saved state dir of one cluster instance: per-node `*.stdout`/`*.stderr` and their rotated `*.stdout.`/`*.stderr.` (node names come from the testnet template, e.g. `bft1` and `pool` - don't assume a fixed pool count), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`, plus the `nodes/` and `shelley/` subdirs (per-node dirs, genesis files and keys). There is one such dir per instance start, so a respun instance number has several; `` matches the `started cluster instance ''` line in `scheduling.log` - a name carrying an extra random suffix, or one with no matching `scheduling.log` line, is a fallback name used when the instance id was unavailable or the dir name already existed +- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/` — saved state dir of one cluster instance: per-node `*.stdout`/`*.stderr` and their rotated `*.stdout.`/`*.stderr.` (`bft1` for the first node, `pool` for the rest; the pool count comes from the testnet config, don't assume three), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`, plus the `nodes/` and `shelley/` subdirs (per-node dirs, genesis files and keys). There is one such dir per instance start, so a respun instance number has several; `` matches the `started cluster instance ''` line in `scheduling.log` - a name carrying an extra random suffix, or one with no matching `scheduling.log` line, is a fallback name used when the instance id was unavailable or the dir name already existed - `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh`: case-insensitive grep for `:error:|failed|failure` in `*.stdout`/`*.stderr` files under `testing_artifacts/`, with paths relative to that dir. Rotated logs (`*.stdout.`, `*.stderr.`) are **not** grepped, so a thin or empty `errors_all.log` doesn't prove there were no errors - grep the rotated logs directly when an early crash is suspected - `{RUN_DIR}/scheduling.log` — cluster instance manager log - `{RUN_DIR}/testrun-report.xml` — junit XML diff --git a/agent_docs/upgrade_failure_analysis_prompt.md b/agent_docs/upgrade_failure_analysis_prompt.md index b6124a116..c0eace9cf 100644 --- a/agent_docs/upgrade_failure_analysis_prompt.md +++ b/agent_docs/upgrade_failure_analysis_prompt.md @@ -15,7 +15,7 @@ Inputs available under `{RUN_DIR}/` (use only what exists): - `{RUN_DIR}/allure-results-step1/`, `{RUN_DIR}/allure-results-step2/`, `{RUN_DIR}/allure-results-step3/` — one JSON per test per step (`status`, `statusDetails.message`, `statusDetails.trace`, stdout/stderr attachments listed by name) - `{RUN_DIR}/testrun-report-step1.html`, `…-step2.html`, `…-step3.html` — self-contained HTML reports (large; prefer the per-step allure JSON above) - `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs, cluster instance artifacts and scripts (shared across all steps) -- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/` — saved state dir of one cluster instance: per-node `*.stdout`/`*.stderr` and their rotated `*.stdout.`/`*.stderr.` (node names come from the testnet template, e.g. `bft1` and `pool` - don't assume a fixed pool count), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`, plus the `nodes/` and `shelley/` subdirs (per-node dirs, genesis files and keys). There is one such dir per instance start, so a respun instance number has several; `` matches the `started cluster instance ''` line in `scheduling.log` - a name carrying an extra random suffix, or one with no matching `scheduling.log` line, is a fallback name used when the instance id was unavailable or the dir name already existed. Each step has its own `pytest-*` dir, so use the same oldest-first ordering as for `cm-status.db` to tell which step a dir belongs to +- `{RUN_DIR}/testing_artifacts/pytest-*/cluster_artifacts/state-cluster_/` — saved state dir of one cluster instance: per-node `*.stdout`/`*.stderr` and their rotated `*.stdout.`/`*.stderr.` (`bft1` for the first node, `pool` for the rest; the pool count comes from the testnet config, don't assume three), `start-cluster.log`, `supervisord.log`, `config-*.json`, `topology-*.json`, `pparams.json`, plus the `nodes/` and `shelley/` subdirs (per-node dirs, genesis files and keys). There is one such dir per instance start, so a respun instance number has several; `` matches the `started cluster instance ''` line in `scheduling.log` - a name carrying an extra random suffix, or one with no matching `scheduling.log` line, is a fallback name used when the instance id was unavailable or the dir name already existed. Each step has its own `pytest-*` dir, so use the same oldest-first ordering as for `cm-status.db` to tell which step a dir belongs to - `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh`: case-insensitive grep for `:error:|failed|failure` in `*.stdout`/`*.stderr` files under `testing_artifacts/`, with paths relative to that dir (covers all steps). Rotated logs (`*.stdout.`, `*.stderr.`) are **not** grepped, so a thin or empty `errors_all.log` doesn't prove there were no errors - grep the rotated logs directly when an early crash is suspected - `{RUN_DIR}/scheduling.log` — cluster instance manager log - `{RUN_DIR}/testing_artifacts/pytest-*/cm-status.db` — cluster-management SQLite status databases, one per step ("test running", resource and flag records as they were at the end of that step); the `pytest-N` dir numbers don't map to steps, order the databases by modification time instead (oldest = step1); there should normally be three - when fewer are present, don't assume positions and correlate with which `allure-results-stepN/` dirs exist to decide which steps the databases belong to; query with `sqlite3 -readonly -header 'SELECT * FROM overview ORDER BY instance_num, kind'`