diff --git a/.github/workflows/regression_reusable.yaml b/.github/workflows/regression_reusable.yaml index 932f6a574..df772cafa 100644 --- a/.github/workflows/regression_reusable.yaml +++ b/.github/workflows/regression_reusable.yaml @@ -225,7 +225,6 @@ jobs: run_workdir/requirements_coverage.json run_workdir/monitor.log run_workdir/failure_analysis.md - run_workdir/cm-status-*.db* - name: ↟ Upload CLI coverage uses: actions/upload-artifact@v7 if: success() || failure() diff --git a/.github/workflows/upgrade_reusable.yaml b/.github/workflows/upgrade_reusable.yaml index 2ad0e4467..fc47eaea1 100644 --- a/.github/workflows/upgrade_reusable.yaml +++ b/.github/workflows/upgrade_reusable.yaml @@ -172,7 +172,6 @@ jobs: run_workdir/scheduling.log run_workdir/errors_all.log run_workdir/failure_analysis.md - run_workdir/cm-status-*.db* - name: ↟ Upload CLI coverage uses: actions/upload-artifact@v7 if: success() || failure() diff --git a/agent_docs/failure_analysis_prompt.md b/agent_docs/failure_analysis_prompt.md index 45301f7fb..8c5acfb4c 100644 --- a/agent_docs/failure_analysis_prompt.md +++ b/agent_docs/failure_analysis_prompt.md @@ -12,7 +12,7 @@ Inputs available under `{RUN_DIR}/` (use only what exists): - `{RUN_DIR}/scheduling.log` — cluster instance manager log - `{RUN_DIR}/testrun-report.xml` — junit XML - `{RUN_DIR}/monitor.log` — system resource snapshots every 10 min -- `{RUN_DIR}/cm-status-1.db` — cluster-management SQLite status database ("test running", resource and flag records as they were at the end of the run); query with `sqlite3 -readonly -header {RUN_DIR}/cm-status-1.db 'SELECT * FROM overview ORDER BY instance_num, kind'` +- `{RUN_DIR}/testing_artifacts/pytest-*/cm-status.db` — cluster-management SQLite status database ("test running", resource and flag records as they were at the end of the run); query with `sqlite3 -readonly -header 'SELECT * FROM overview ORDER BY instance_num, kind'` Counting tests (IMPORTANT): @@ -40,7 +40,7 @@ Steps: 2. Group failures by likely root cause (same exception class + message head, same node crash, same infra symptom). Treat one node crash that flunks many tests as a single group. 3. For each group: list affected tests (truncate to ~10 with a "+N more" tail), give the most informative 1–3 lines of error context, and classify as one of `node-bug | test-bug | infra-flake | env-issue | unknown` with a short justification. 4. Skim `{RUN_DIR}/errors_all.log` and `{RUN_DIR}/monitor.log` for anything corroborating (OOM, disk pressure, repeated tracebacks). -5. When failures look cluster-management related (dead cluster instances, tests stuck waiting for resources), query the `overview` view of `{RUN_DIR}/cm-status-1.db` — leftover records show which tests were running, which resources were locked and which flags (e.g. `cluster_dead`, `respin_needed`) were set when the run ended. +5. When failures look cluster-management related (dead cluster instances, tests stuck waiting for resources), query the `overview` view of the status database — leftover records show which tests were running, which resources were locked and which flags (e.g. `cluster_dead`, `respin_needed`) were set when the run ended. Known patterns: diff --git a/agent_docs/upgrade_failure_analysis_prompt.md b/agent_docs/upgrade_failure_analysis_prompt.md index 38197d691..d888c3c0c 100644 --- a/agent_docs/upgrade_failure_analysis_prompt.md +++ b/agent_docs/upgrade_failure_analysis_prompt.md @@ -17,7 +17,7 @@ Inputs available under `{RUN_DIR}/` (use only what exists): - `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs with cluster logs, node stdouts, etc. (shared across all steps) - `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh` over cluster logs (covers all steps) - `{RUN_DIR}/scheduling.log` — cluster instance manager log -- `{RUN_DIR}/cm-status-1.db`, `{RUN_DIR}/cm-status-2.db`, `{RUN_DIR}/cm-status-3.db` — cluster-management SQLite status databases, one per step ("test running", resource and flag records as they were at the end of that step); query with `sqlite3 -readonly -header {RUN_DIR}/cm-status-1.db 'SELECT * FROM overview ORDER BY instance_num, kind'` +- `{RUN_DIR}/testing_artifacts/pytest-*/cm-status.db` — cluster-management SQLite status databases, one per step ("test running", resource and flag records as they were at the end of that step); the `pytest-N` dir numbers don't map to steps, order the databases by modification time instead (oldest = step1); there should normally be three - when fewer are present, don't assume positions and correlate with which `allure-results-stepN/` dirs exist to decide which steps the databases belong to; query with `sqlite3 -readonly -header 'SELECT * FROM overview ORDER BY instance_num, kind'` Steps: @@ -28,7 +28,7 @@ Steps: `grep -E '"status": "(failed|broken)"' {RUN_DIR}/allure-results-step*/*result.json | cut -c1-200`. 3. Group failures by likely root cause (same exception class + message head, same node crash, same infra symptom). **Note which step(s) each group hits** — a failure that appears only in step2 or step3 is much more interesting than one that already fails in step1. Treat one node crash that flunks many tests as a single group. 4. For each group: list affected tests (truncate to ~10 with a "+N more" tail), give the most informative 1–3 lines of error context, mark the step(s) affected, and classify as one of `node-bug | test-bug | infra-flake | env-issue | upgrade-regression | unknown` with a short justification. Use `upgrade-regression` when a test passes in step1 but fails in step2 or step3 — that is the signal this workflow exists to catch. -5. Skim `{RUN_DIR}/errors_all.log` and `{RUN_DIR}/scheduling.log` for anything corroborating (node crash on restart, hard-fork failure, supervisord errors, OOM, repeated tracebacks). When failures look cluster-management related (dead cluster instances, tests stuck waiting for resources), query the `overview` view of the affected step's `cm-status-N.db`. +5. Skim `{RUN_DIR}/errors_all.log` and `{RUN_DIR}/scheduling.log` for anything corroborating (node crash on restart, hard-fork failure, supervisord errors, OOM, repeated tracebacks). When failures look cluster-management related (dead cluster instances, tests stuck waiting for resources), query the `overview` view of the affected step's status database. 6. If a whole step is missing its `allure-results-stepN/` dir, that step likely failed before pytest ran — call this out explicitly and check `errors_all.log` / the workflow log group output for the cause (commonly a `start-cluster` / `supervisord` / hard-fork failure). Known patterns: diff --git a/runner/node_upgrade.sh b/runner/node_upgrade.sh index b654fabfa..7aa5bcb68 100755 --- a/runner/node_upgrade.sh +++ b/runner/node_upgrade.sh @@ -190,13 +190,8 @@ fi _last_cleanup -# Copy cluster status databases to workdir -./runner/status_dbs.sh "$ARTIFACTS_DIR" "$WORKDIR" || : - -# prepare artifacts for upload in GitHub Actions -if [ -n "${GITHUB_ACTIONS:-}" ]; then - # save testing artifacts - ./runner/save_artifacts.sh "$ARTIFACTS_DIR" "$WORKDIR" -fi +# Save testing artifacts. Best-effort - a failure here must not override the testrun +# exit code. +./runner/save_artifacts.sh "$ARTIFACTS_DIR" "$WORKDIR" || : exit "$retval" diff --git a/runner/regression.sh b/runner/regression.sh index 78b778323..f118891aa 100755 --- a/runner/regression.sh +++ b/runner/regression.sh @@ -438,9 +438,6 @@ fi # Create results archive ./runner/create_results.sh "$REPORTS_DIR" "$WORKDIR" || : -# Copy cluster status databases to workdir -./runner/status_dbs.sh "$ARTIFACTS_DIR" "$WORKDIR" || : - # Save testing artifacts ./runner/save_artifacts.sh "$ARTIFACTS_DIR" "$WORKDIR" || : diff --git a/runner/status_dbs.sh b/runner/status_dbs.sh deleted file mode 100755 index d442fdb0a..000000000 --- a/runner/status_dbs.sh +++ /dev/null @@ -1,54 +0,0 @@ -#!/usr/bin/env bash - -set -euo pipefail - -if [ "$#" -ne 2 ]; then - echo "Usage: $0 " >&2 - exit 1 -fi - -artifacts_dir="$1" -output_dir="$2" - -mkdir -p "$output_dir" || { echo "Cannot create $output_dir" >&2; exit 1; } - -# The status database was copied from the pytest temp dir to the artifacts dir together -# with the other testing artifacts. Copy the database of each pytest run to the output -# dir. The output databases are numbered in the run order (e.g. one per node upgrade -# step). The run order is derived from the database modification times (preserved by -# the copy to the artifacts dir) - the `pytest-N` component of the artifacts subdir -# names cannot be used, as pytest's numbered basetemp counter is per temp root and is -# shared by all pytest invocations that use the root, including invocations whose -# artifacts are not collected. The mtime ordering holds no matter how the launcher -# distributes the pytest invocations over temp roots. -# Copy also the WAL sidecar. As artifacts are copied only after pytest exits, the WAL -# is normally checkpointed into the database and the sidecar removed. It can still be -# left behind when the run was killed - copy it in that case, otherwise the database -# copy would miss the not-yet-checkpointed writes. The `-shm` file is skipped on -# purpose, it is transient and not needed for an offline copy. -found=0 -num=0 -while IFS= read -r db; do - # The file could disappear before it is copied - [ -f "$db" ] || continue - num=$((num + 1)) - cp "$db" "${output_dir}/cm-status-${num}.db" \ - || { echo "Failed to copy $db" >&2; continue; } - found=1 - if [ -e "${db}-wal" ]; then - cp "${db}-wal" "${output_dir}/cm-status-${num}.db-wal" \ - || echo "Failed to copy ${db}-wal" >&2 - fi -done < <( - # Hidden dirs are excluded explicitly - a hidden dir can be a leftover staging dir - # of an interrupted `copy_artifacts.sh` run - find "$artifacts_dir" -mindepth 2 -maxdepth 2 ! -path '*/.*' \ - -path '*/pytest-*/cm-status.db' -printf '%T@\t%p\n' | sort -n | cut -f2- -) - -if [ "$found" -eq 0 ]; then - echo "No status database copied from $artifacts_dir" >&2 - if [ -n "${GITHUB_ACTIONS:-}" ]; then - echo "::warning::No cluster status database was copied from the testing artifacts, none uploaded." - fi -fi