Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .github/workflows/regression_reusable.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,6 @@ jobs:
run_workdir/requirements_coverage.json
run_workdir/monitor.log
run_workdir/failure_analysis.md
run_workdir/cm-status-*.db*
- name: ↟ Upload CLI coverage
uses: actions/upload-artifact@v7
if: success() || failure()
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/upgrade_reusable.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,6 @@ jobs:
run_workdir/scheduling.log
run_workdir/errors_all.log
run_workdir/failure_analysis.md
run_workdir/cm-status-*.db*
- name: ↟ Upload CLI coverage
uses: actions/upload-artifact@v7
if: success() || failure()
Expand Down
4 changes: 2 additions & 2 deletions agent_docs/failure_analysis_prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Inputs available under `{RUN_DIR}/` (use only what exists):
- `{RUN_DIR}/scheduling.log` — cluster instance manager log
- `{RUN_DIR}/testrun-report.xml` — junit XML
- `{RUN_DIR}/monitor.log` — system resource snapshots every 10 min
- `{RUN_DIR}/cm-status-1.db` — cluster-management SQLite status database ("test running", resource and flag records as they were at the end of the run); query with `sqlite3 -readonly -header {RUN_DIR}/cm-status-1.db 'SELECT * FROM overview ORDER BY instance_num, kind'`
- `{RUN_DIR}/testing_artifacts/pytest-*/cm-status.db` — cluster-management SQLite status database ("test running", resource and flag records as they were at the end of the run); query with `sqlite3 -readonly -header <db_file> 'SELECT * FROM overview ORDER BY instance_num, kind'`

Counting tests (IMPORTANT):

Expand Down Expand Up @@ -40,7 +40,7 @@ Steps:
2. Group failures by likely root cause (same exception class + message head, same node crash, same infra symptom). Treat one node crash that flunks many tests as a single group.
3. For each group: list affected tests (truncate to ~10 with a "+N more" tail), give the most informative 1–3 lines of error context, and classify as one of `node-bug | test-bug | infra-flake | env-issue | unknown` with a short justification.
4. Skim `{RUN_DIR}/errors_all.log` and `{RUN_DIR}/monitor.log` for anything corroborating (OOM, disk pressure, repeated tracebacks).
5. When failures look cluster-management related (dead cluster instances, tests stuck waiting for resources), query the `overview` view of `{RUN_DIR}/cm-status-1.db` — leftover records show which tests were running, which resources were locked and which flags (e.g. `cluster_dead`, `respin_needed`) were set when the run ended.
5. When failures look cluster-management related (dead cluster instances, tests stuck waiting for resources), query the `overview` view of the status database — leftover records show which tests were running, which resources were locked and which flags (e.g. `cluster_dead`, `respin_needed`) were set when the run ended.

Known patterns:

Expand Down
4 changes: 2 additions & 2 deletions agent_docs/upgrade_failure_analysis_prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Inputs available under `{RUN_DIR}/` (use only what exists):
- `{RUN_DIR}/testing_artifacts/` — per-test artifact dirs with cluster logs, node stdouts, etc. (shared across all steps)
- `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh` over cluster logs (covers all steps)
- `{RUN_DIR}/scheduling.log` — cluster instance manager log
- `{RUN_DIR}/cm-status-1.db`, `{RUN_DIR}/cm-status-2.db`, `{RUN_DIR}/cm-status-3.db` — cluster-management SQLite status databases, one per step ("test running", resource and flag records as they were at the end of that step); query with `sqlite3 -readonly -header {RUN_DIR}/cm-status-1.db 'SELECT * FROM overview ORDER BY instance_num, kind'`
- `{RUN_DIR}/testing_artifacts/pytest-*/cm-status.db` — cluster-management SQLite status databases, one per step ("test running", resource and flag records as they were at the end of that step); the `pytest-N` dir numbers don't map to steps, order the databases by modification time instead (oldest = step1); there should normally be three - when fewer are present, don't assume positions and correlate with which `allure-results-stepN/` dirs exist to decide which steps the databases belong to; query with `sqlite3 -readonly -header <db_file> 'SELECT * FROM overview ORDER BY instance_num, kind'`

Steps:

Expand All @@ -28,7 +28,7 @@ Steps:
`grep -E '"status": "(failed|broken)"' {RUN_DIR}/allure-results-step*/*result.json | cut -c1-200`.
3. Group failures by likely root cause (same exception class + message head, same node crash, same infra symptom). **Note which step(s) each group hits** — a failure that appears only in step2 or step3 is much more interesting than one that already fails in step1. Treat one node crash that flunks many tests as a single group.
4. For each group: list affected tests (truncate to ~10 with a "+N more" tail), give the most informative 1–3 lines of error context, mark the step(s) affected, and classify as one of `node-bug | test-bug | infra-flake | env-issue | upgrade-regression | unknown` with a short justification. Use `upgrade-regression` when a test passes in step1 but fails in step2 or step3 — that is the signal this workflow exists to catch.
5. Skim `{RUN_DIR}/errors_all.log` and `{RUN_DIR}/scheduling.log` for anything corroborating (node crash on restart, hard-fork failure, supervisord errors, OOM, repeated tracebacks). When failures look cluster-management related (dead cluster instances, tests stuck waiting for resources), query the `overview` view of the affected step's `cm-status-N.db`.
5. Skim `{RUN_DIR}/errors_all.log` and `{RUN_DIR}/scheduling.log` for anything corroborating (node crash on restart, hard-fork failure, supervisord errors, OOM, repeated tracebacks). When failures look cluster-management related (dead cluster instances, tests stuck waiting for resources), query the `overview` view of the affected step's status database.
6. If a whole step is missing its `allure-results-stepN/` dir, that step likely failed before pytest ran — call this out explicitly and check `errors_all.log` / the workflow log group output for the cause (commonly a `start-cluster` / `supervisord` / hard-fork failure).

Known patterns:
Expand Down
11 changes: 3 additions & 8 deletions runner/node_upgrade.sh
Original file line number Diff line number Diff line change
Expand Up @@ -190,13 +190,8 @@ fi

_last_cleanup

# Copy cluster status databases to workdir
./runner/status_dbs.sh "$ARTIFACTS_DIR" "$WORKDIR" || :

# prepare artifacts for upload in GitHub Actions
if [ -n "${GITHUB_ACTIONS:-}" ]; then
# save testing artifacts
./runner/save_artifacts.sh "$ARTIFACTS_DIR" "$WORKDIR"
fi
# Save testing artifacts. Best-effort - a failure here must not override the testrun
# exit code.
./runner/save_artifacts.sh "$ARTIFACTS_DIR" "$WORKDIR" || :

exit "$retval"
3 changes: 0 additions & 3 deletions runner/regression.sh
Original file line number Diff line number Diff line change
Expand Up @@ -438,9 +438,6 @@ fi
# Create results archive
./runner/create_results.sh "$REPORTS_DIR" "$WORKDIR" || :

# Copy cluster status databases to workdir
./runner/status_dbs.sh "$ARTIFACTS_DIR" "$WORKDIR" || :

# Save testing artifacts
./runner/save_artifacts.sh "$ARTIFACTS_DIR" "$WORKDIR" || :

Expand Down
54 changes: 0 additions & 54 deletions runner/status_dbs.sh

This file was deleted.