Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
173 changes: 173 additions & 0 deletions test/workflows/auto-rebase/approval-survival.bats
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
#!/usr/bin/env bats
# Integration regression test for issue #929 (part of #926, AC8):
# an APPROVED PR stays approved/mergeable after an eligible auto-rebase update.
Comment on lines +2 to +3

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

1. approval-survival.bats not under integration 📘 Rule violation ▣ Testability

This file is explicitly described as an integration regression test but is located under
test/workflows/... instead of a directory whose name includes integration. This violates the
required unit/integration test separation by directory naming and can make CI test selection/gating
unreliable.
Agent Prompt
## Issue description
`test/workflows/auto-rebase/approval-survival.bats` is written/labelled as an integration regression test, but it is not placed under a directory whose name includes `integration`, as required by the compliance rule.

## Issue Context
The test header comment states it is an “Integration regression test”, and it drives the reusable workflow run-block end-to-end (integration-style). The compliance checklist requires integration tests to live under a dedicated `integration` directory.

## Fix Focus Areas
- test/workflows/auto-rebase/approval-survival.bats[1-172]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

#
# Unlike eligibility.bats / comments.bats (which unit-test the pure library
# helpers), this drives the reusable workflow's actual "Update behind
# non-Dependabot PRs" run-block end to end against a stubbed `gh`. The stub
# models the approval-survival semantics under this org's default ruleset
# configuration (dismiss_stale_reviews_on_push: false, require_last_push_approval: false):
#
# - update-branch with update_method=merge preserves the existing commits
# (no SHA rewrite) → under this org's ruleset, an existing APPROVED review
# survives and the PR stays mergeable.
# - update-branch with update_method=rebase rewrites SHAs → GitHub dismisses
# the approval (review decision drops back to REVIEW_REQUIRED).
#
# This test proves the #929 scenario (behind + conflict-free + non-draft
# + APPROVED → updated + still APPROVED/mergeable) and confirms the workflow
# uses merge method. The "still-approved" assertion is non-vacuous: a regression
# to rebase would flip the modeled approval and fail this suite. However, this
# test cannot detect a future reversion to a strict ruleset (require_last_push_approval: true)
# — that would require a live integration check against the real ruleset config.

load 'helpers/setup'

REUSABLE="${TT_REPO_ROOT}/.github/workflows/auto-rebase-reusable.yml"

setup() {
tt_make_tmpdir

# Working dir for the run-block. The reusable sources its tooling from a
# checkout at ./.auto-rebase-tooling; point that at this repo so the real
# eligibility.sh / comments.sh get sourced.
TT_WORK="${TT_TMP}/work"
mkdir -p "$TT_WORK"
ln -s "$TT_REPO_ROOT" "${TT_WORK}/.auto-rebase-tooling"

# Modeled GitHub state, mutated by the gh stub.
STATE_DIR="${TT_TMP}/state"
mkdir -p "$STATE_DIR"
export STATE_DIR

# `gh` stub on PATH — never touches the network.
TT_BIN="${TT_TMP}/bin"
mkdir -p "$TT_BIN"
_install_gh_stub
Comment on lines +43 to +46

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

2. _install_gh_stub bypasses shared helper 📘 Rule violation ▣ Testability

The new test implements a bespoke gh stub and PATH setup inline instead of using the repo’s
established tt_install_gh_stub helper pattern used in other test suites. This violates the
requirement to use project-provided test helpers for mocking external services when such helpers
exist, and increases duplication/inconsistency.
Agent Prompt
## Issue description
`approval-survival.bats` adds an ad-hoc `_install_gh_stub` implementation to mock `gh`, rather than using the repo’s standard `tt_install_gh_stub` helper approach used elsewhere.

## Issue Context
Other workflow test suites provide a reusable `tt_install_gh_stub` helper (and a `stubs/gh` script) to ensure consistent behavior and avoid duplicating stub logic across tests. This new test currently embeds a custom stub generator and PATH wiring.

## Fix Focus Areas
- test/workflows/auto-rebase/approval-survival.bats[42-115]
- test/workflows/auto-rebase/helpers/setup.bash[1-21]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

PATH="${TT_BIN}:${PATH}"
export PATH

# Extract the reusable's update run-block verbatim so we exercise the real
# workflow logic, not a hand-copied paraphrase of it.
RUN_SCRIPT="${TT_TMP}/run.sh"
yq -r '.jobs.auto-rebase.steps[]
| select(.name == "Update behind non-Dependabot PRs")
| .run' "$REUSABLE" > "$RUN_SCRIPT"
}
Comment on lines +50 to +56

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: The test setup requires yq, but the auto-rebase test workflow installs only bats, shellcheck, and jq. On the configured CI runner, yq is unavailable, so every test fails during setup before the workflow run-block is exercised. Install yq explicitly or extract the YAML using an already-provisioned tool. [possible bug]

Severity Level: Major ⚠️
- ❌ CI fails before approval-survival tests execute.
- ⚠️ Auto-rebase regression coverage is unavailable.

Fix in Cursor Fix in VSCode Claude

(Use Cmd/Ctrl + Click for best experience)

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** test/workflows/auto-rebase/approval-survival.bats
**Line:** 49:55
**Comment:**
	*Possible Bug: The test setup requires `yq`, but the auto-rebase test workflow installs only `bats`, `shellcheck`, and `jq`. On the configured CI runner, `yq` is unavailable, so every test fails during setup before the workflow run-block is exercised. Install `yq` explicitly or extract the YAML using an already-provisioned tool.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎


teardown() {
tt_cleanup_tmpdir
}

# Seed one open, non-Dependabot, same-repo PR that is behind, conflict-free,
# non-draft, and already APPROVED.
_seed_approved_behind_pr() {
printf '1 feature-branch\n' > "${STATE_DIR}/pr_list"
printf 'main\n' > "${STATE_DIR}/base_ref"
printf '3\n' > "${STATE_DIR}/behind"
printf 'APPROVED\n' > "${STATE_DIR}/review_decision"
printf 'true\n' > "${STATE_DIR}/mergeable"
}
Comment on lines +64 to +70

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: The seeded PR-list response is already reduced to an eligible PR, while the gh stub ignores the workflow's --jq filter. Consequently, a regression removing the Dependabot, fork, or same-repository filters would still pass because the fixture has pre-applied those filters and never supplies the fields needed to evaluate them. Return a JSON PR object containing those attributes and validate that the filtering is performed by the extracted run-block. [incomplete implementation]

Severity Level: Major ⚠️
- ⚠️ Workflow eligibility-filter regressions can escape this suite.
- ⚠️ Dependabot or fork branches could be updated unintentionally.

Fix in Cursor Fix in VSCode Claude

(Use Cmd/Ctrl + Click for best experience)

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** test/workflows/auto-rebase/approval-survival.bats
**Line:** 63:69
**Comment:**
	*Incomplete Implementation: The seeded PR-list response is already reduced to an eligible PR, while the `gh` stub ignores the workflow's `--jq` filter. Consequently, a regression removing the Dependabot, fork, or same-repository filters would still pass because the fixture has pre-applied those filters and never supplies the fields needed to evaluate them. Return a JSON PR object containing those attributes and validate that the filtering is performed by the extracted run-block.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎


_install_gh_stub() {
cat > "${TT_BIN}/gh" <<'STUB'
#!/usr/bin/env bash
# Minimal `gh` modeling the subset of the GitHub API the auto-rebase run-block
# calls, plus the approval-survival semantics of merge vs rebase.
set -euo pipefail

printf '%s\n' "$*" >> "${STATE_DIR}/gh-calls.log"

args="$*"
case "$args" in
*update-branch*)
method="merge"
for a in "$@"; do
case "$a" in
update_method=*) method="${a#update_method=}" ;;
esac
done
printf '%s\n' "$method" >> "${STATE_DIR}/update-methods.log"
# The branch is now up to date regardless of method.
printf '0\n' > "${STATE_DIR}/behind"
if [ "$method" != "merge" ]; then
# rebase rewrites SHAs → GitHub dismisses the existing approval.
printf 'REVIEW_REQUIRED\n' > "${STATE_DIR}/review_decision"
printf 'false\n' > "${STATE_DIR}/mergeable"
fi
exit 0
;;
*compare/*)
cat "${STATE_DIR}/behind"
exit 0
;;
Comment on lines +100 to +103

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

3. Stub bypasses --jq logic 🐞 Bug ⚙ Maintainability

The gh stub returns canned, already-filtered text for gh api ... --jq ... calls without
validating or exercising the workflow’s jq expressions, so a broken/changed --jq filter (or
API-shape expectation) in the real reusable could still pass this “end-to-end” test. This reduces
the test’s effectiveness as a regression guard for the actual run-block behavior.
Agent Prompt
### Issue description
The new integration test stubs `gh` by matching on URL substrings and printing pre-shaped outputs, but it does not validate (or execute) the `--jq` expressions used by the real reusable workflow. This means regressions to the jq filters in `.github/workflows/auto-rebase-reusable.yml` may not be detected.

### Issue Context
The workflow relies on `gh api ... --jq ...` to shape outputs that are then parsed/used by bash control flow. The stub currently ignores the jq argument entirely and returns the final shaped text.

### Fix Focus Areas
- test/workflows/auto-rebase/approval-survival.bats[71-112]

Suggested implementation options:
1) **Add explicit assertions on the recorded gh calls** (simplest): after `_run_workflow`, assert `${STATE_DIR}/gh-calls.log` contains the expected `--jq` snippets for:
   - PR list call (`pulls?state=open...`)
   - base branch call (`--jq '.base.ref'`)
   - compare call (`--jq '.behind_by'`)

2) **Make the stub enforce jq presence/values**: in each case arm, require `--jq` and check its value matches the expected expression; otherwise `exit 1` so the test fails.

3) **Higher fidelity**: have the stub output minimal JSON and apply the passed `--jq` using `jq` (if available) so jq breakage is truly exercised.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

*state=open*)
cat "${STATE_DIR}/pr_list"
exit 0
;;
*/pulls/*)
cat "${STATE_DIR}/base_ref"
exit 0
;;
esac
exit 0
STUB
chmod +x "${TT_BIN}/gh"
}

# Run the extracted reusable run-block with the same shell flags GitHub Actions
# uses for a default `run:` step (bash -eo pipefail).
_run_workflow() {
run env \
STATE_DIR="$STATE_DIR" \
GH_TOKEN="stub-token" \
HAS_PAT="false" \
REPO="owner/repo" \
ELIGIBILITY="all" \
bash --noprofile --norc -eo pipefail -c \
"cd '${TT_WORK}' && exec bash --noprofile --norc -eo pipefail '${RUN_SCRIPT}'"
}

# ── #929: the approved-behind PR gets updated and stays approved ─────────────

@test "approval-survival: an APPROVED behind PR is updated via merge" {
_seed_approved_behind_pr
_run_workflow
[ "$status" -eq 0 ]
[[ "$output" == *"#1"* ]]
[[ "$output" == *"Branch updated"* ]]
# It updated with the approval-preserving method, never rebase.
grep -qx "merge" "${STATE_DIR}/update-methods.log"
! grep -qx "rebase" "${STATE_DIR}/update-methods.log"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

When using grep in BATS tests to assert the absence of a pattern, assert that the exit status is exactly 1 ([ "$status" -eq 1 ]) instead of using ! or checking for any non-zero status. This prevents false positives where other errors (such as a missing file or grep syntax error, which return exit status 2) cause the test to pass incorrectly.

  run grep -qx "rebase" "${STATE_DIR}/update-methods.log"
  [ "$status" -eq 1 ]
References
  1. When using grep in BATS tests to assert the absence of a pattern, assert that the exit status is exactly 1 ([ "$status" -eq 1 ]) instead of any non-zero status ([ "$status" -ne 0 ]). This prevents false positives where other errors (such as a missing file or grep syntax error, which return exit status 2) cause the test to pass incorrectly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in commit 4344a79: replaced ! grep -qx "rebase" with run grep -qx "rebase" / [ "$status" -eq 1 ] at line 145–146. This explicitly asserts exit status 1 (not-found) rather than any non-zero status, so a grep error (exit 2) no longer produces a false-positive pass.

}

@test "approval-survival: the PR ends up-to-date after the update" {
_seed_approved_behind_pr
_run_workflow
[ "$status" -eq 0 ]
[ "$(cat "${STATE_DIR}/behind")" = "0" ]
}

@test "approval-survival: the PR remains APPROVED and mergeable afterward" {
_seed_approved_behind_pr
_run_workflow
[ "$status" -eq 0 ]
[ "$(cat "${STATE_DIR}/review_decision")" = "APPROVED" ]
[ "$(cat "${STATE_DIR}/mergeable")" = "true" ]
}

# ── teeth: the "stays APPROVED" claim depends on merge, not luck ─────────────
#
# Directly exercise the modeled endpoint with the rebase method to confirm the
# stub actually dismisses the approval on a SHA rewrite. This proves the
# assertions above are non-vacuous: had the workflow regressed to rebase, the
# approval would drop to REVIEW_REQUIRED and the suite would go red.

@test "approval-survival: a rebase update would dismiss the approval (control)" {
_seed_approved_behind_pr
run env STATE_DIR="$STATE_DIR" \
gh api "repos/owner/repo/pulls/1/update-branch" -X PUT -f update_method=rebase
[ "$status" -eq 0 ]
[ "$(cat "${STATE_DIR}/review_decision")" = "REVIEW_REQUIRED" ]
[ "$(cat "${STATE_DIR}/mergeable")" = "false" ]
}
Loading