Skip to content

fix(stage0): land #162 — let stage-1 roll back a hung image instead of recovering (rebased) - #167

Merged
Aswin-V merged 2 commits into
masterfrom
loop/2026-11-02-eboot162-watchdog-rollback
Nov 2, 2026
Merged

Aswin-V merged 2 commits into
masterfrom
loop/2026-11-02-eboot162-watchdog-rollback

Conversation

@Aswin-V

@Aswin-V Aswin-V commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Lands the work of #162 (by @Greeva48) on current master.

Why a separate PR: #162 is from a fork branch. It was approved on substance on 2026-11-01 and is blocked only by a tests/CMakeLists.txt anchor conflict: #158's merge added a test registration at the same anchor where #162 adds its own. Maintainer edits are enabled on #162, but the GitHub API does not permit pushing commits to fork branches (verified 2026-11-02: git-data writes to the fork return 404), so the rebase could not be done in #162's own branch.

What this PR is: a merge commit whose FIRST PARENT is #162's head (43279bf) unchanged — the contributor's commits land exactly as authored — and whose second parent is current master (09fbc81). The only new content is the tests/CMakeLists.txt conflict resolution, which keeps both test registrations (test_stage0_boot_log from master, test_watchdog_rollback from #162). No other file differs from the combination of #162 + master.

Fixes #161 (via #162's change).

On merge, #162 will be closed as superseded by this PR with a pointer here.

Summary by Sourcery

Fix watchdog handling so exhausted boot attempts reach stage-1 rollback policy while retaining recovery fallback behavior.

Bug Fixes:

  • Allow stage-1 to roll back a hung test image after watchdog boot attempts are exhausted instead of prematurely entering recovery.
  • Preserve recovery behavior when no valid rollback image exists and for explicit recovery conditions.

Tests:

  • Add host-side coverage for watchdog rollback, no-slot recovery fallback, and unchanged explicit recovery triggers.

Summary by CodeRabbit

  • Bug Fixes
    • Exhausted watchdog boot attempts now allow the system to try the other boot slot instead of immediately entering recovery.
    • Recovery is entered if no valid boot slot is available. Recovery-pin, forced-recovery, and factory-reset triggers continue to work as before.

RetriggerView in GreptileConfidence Score: 4/5

Add a stopping condition for a failed fallback before merging.

Findings

  1. P1 Both images keep rebooting ▶
Fix with agent prompt
### Issue 1
stage0/recovery_entry.c:27-32
Removing this check leaves devices rebooting forever when both slots contain valid images that hang. `eos_boot_policy_select()` switches slots and clears `boot_attempts` whenever the limit is reached. It does not remember that the alternate image already failed, so the two images keep getting fresh attempts instead of reaching recovery. Previously, stage-0 stopped this loop.

Keep the first rollback, but enter recovery if the fallback also exhausts its attempts. Add a test for that sequence.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Stage-0 now leaves exhausted boot attempts to stage-1 instead of entering recovery immediately.

  • Exhausted watchdog attempts can now roll back to another slot.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Watchdog reset after attempt limit] --> B[Stage-0 continues to stage-1]
    B --> C{Alternate image valid?}
    C -->|No| D[Enter recovery]
    C -->|Yes| E[Switch slots and clear attempts]
    E --> F[Run alternate image]
    F -->|Also hangs until attempt limit| A
Loading

Reviews (1) · Last reviewed commit: "Merge PR #162 head with master — maintai..." · Reviewed by Greptile

Greeva Patel and others added 2 commits October 31, 2026 16:13
A test-boot image that hangs is reset by the watchdog, and stage-1
increments boot_attempts before every jump. When boot_attempts reaches
max_attempts, stage-1's eos_boot_policy_select() rolls back to the other
slot (docs/architecture.md, docs/secure_boot_chain.md). But stage-0's
ebldr_recovery_triggered() tested the same condition first -- watchdog
reset with boot_attempts >= max_attempts -- and entered recovery, so the
rollback never ran and the device sat in recovery with a good image in
the other slot.

Drop that clause from the stage-0 trigger. Exhausted attempts are a
rollback condition, owned by the stage-1 policy, which still falls
through to recovery itself when there is no valid slot to roll back to.
The recovery pin, FORCE_RECOVERY and FACTORY_RESET triggers are
unchanged.

test_watchdog_rollback: new suite running the real stage-0 trigger check
(stage0/recovery_entry.c compiled for the host) and then the real boot
policy, with slot validity mocked at the slot-manager boundary: a hung
test boot of B after a watchdog reset rolls back to A; with no valid
alternate the policy still refuses (stage-1 enters recovery); the
explicit recovery triggers still fire.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…esolution only; contributor commits unchanged as first parent)
@sourcery-ai

sourcery-ai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

Removes stage-0's premature recovery path for watchdog-exhausted boot attempts so stage-1 can roll back a hung image, while preserving recovery for explicit triggers and unavailable fallback images. Adds an integrated host regression test with mocked board and slot-manager boundaries, and registers it alongside the existing stage-0 boot-log test.

Sequence diagram for watchdog rollback in stage-1

sequenceDiagram
    participant Stage0
    participant Stage1
    participant SlotManager

    Stage0->>Stage0: ebldr_recovery_triggered(bctl)
    Stage0-->>Stage1: return false
    Stage1->>Stage1: eos_boot_policy_select()
    alt fallback slot is valid
        Stage1->>SlotManager: select fallback slot
        SlotManager-->>Stage1: rollback target
    else no valid slot
        Stage1-->>Stage0: enter recovery
    end
Loading

File-Level Changes

Change Details Files
Allow exhausted watchdog attempts to proceed to stage-1 rollback instead of forcing stage-0 recovery.
  • Remove the watchdog-reset/max-attempts recovery condition from the stage-0 trigger check.
  • Preserve explicit recovery-pin, force-recovery, and factory-reset triggers.
  • Retain stage-1 recovery behavior when no valid rollback slot exists.
stage0/recovery_entry.c
Add host-side regression coverage for watchdog rollback and conflict-resolved test registration.
  • Build and register a test that links the real stage-0 recovery check and stage-1 boot policy.
  • Mock slot validity and board flash/HAL operations without requiring signed images.
  • Verify rollback to the confirmed slot, failure with no fallback image, and unchanged explicit triggers.
  • Keep both the existing boot-log test and the new watchdog-rollback test registered.
tests/CMakeLists.txt
tests/unit/test_watchdog_rollback.c

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@kody-ai

kody-ai Bot commented Nov 2, 2026

Copy link
Copy Markdown

You've used all your free Kodus-paid PR reviews 🎁

Your trial is still active — this just means the PR reviews we cover during the trial are used up.

Connect your own AI key to keep Kody reviewing — unlimited reviews, on any plan (Free included).

Want more trial reviews to finish evaluating before adding a key? Talk to our founders. 😎

@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

Walkthrough

Stage-0 no longer triggers recovery when watchdog boot attempts reach the maximum. New host-side tests verify that stage-1 rolls back to a confirmed slot, returns EOS_ERR_NO_IMAGE when no valid slot is available, and preserves the other recovery triggers.

Changes: Watchdog rollback

Layer / File(s) Summary
Recovery decision and rollback validation
stage0/recovery_entry.c, tests/unit/test_watchdog_rollback.c, tests/CMakeLists.txt
Stage-0 leaves exhausted watchdog attempts to stage-1 boot policy. The tests check rollback to confirmed slot A, the no-valid-slot result, and recovery-pin, force-recovery, and factory-reset triggers. CMake registers the test with CTest and the unit-test list.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: srpatcha, hshanmug12

Merge Risk: Moderate · up to 4a1b5

A hung new image now rolls back to the known-good slot as intended. If both images hang, the device can loop between them forever instead of entering recovery. Add persisted failed-slot tracking, or an equivalent limit, before merging unless this gap is explicitly accepted.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Let stage-1 roll back after watchdog boot attempts are exhausted

🐞 Bug fix 🧪 Tests 🕐 20-40 Minutes

Grey Divider

AI Description

• Stop stage-0 from treating exhausted watchdog boot attempts as an immediate recovery trigger.
• Let stage-1 roll back to a valid alternate image, or recover when none exists.
• Test rollback, no-slot fallback, and unchanged explicit recovery triggers.
Diagram

graph TD
    W["Reset and bootctl"] --> A{"Explicit recovery?"} -->|no| C["Stage-1 policy"] -->|attempts exhausted| D{"Valid alternate?"} -->|yes| E["Roll back slot"]
    A -->|yes| B["Stage-0 recovery"]
    D -->|no| F["Stage-1 recovery"]
Loading
High-Level Assessment

Keep rollback decisions in the existing stage-1 boot policy and limit stage-0 to explicit recovery triggers. Duplicating slot-selection logic in stage-0 would create competing decisions without improving the fallback.

Files changed (3) +216 / -5

Bug fix (1) +6 / -5
recovery_entry.cDefer exhausted boot attempts to stage-1 +6/-5

Defer exhausted boot attempts to stage-1

• Removes the watchdog-plus-exhausted-attempts recovery trigger so a hung image can reach stage-1 rollback policy. Recovery pin, forced recovery, and factory-reset checks remain intact.

stage0/recovery_entry.c

Tests (2) +210 / -0
CMakeLists.txtRegister the host watchdog rollback suite +10/-0

Register the host watchdog rollback suite

• Builds the new test with the real stage-0 recovery check and links it to the core boot policy. Registers it for CTest and the shared Valgrind test list while retaining the existing stage-0 boot-log registration.

tests/CMakeLists.txt

test_watchdog_rollback.cCover rollback and recovery handoff +200/-0

Cover rollback and recovery handoff

• Uses a simulated board and mocked slot validity to exercise the real stage-0 trigger check followed by boot policy. Verifies rollback to a good slot, a no-alternate failure for stage-1 recovery, and unchanged explicit recovery triggers.

tests/unit/test_watchdog_rollback.c

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo


View medium (1)
Remediation recommended
1. Damaged stage-one prevents recovery 🐞
Description
ebldr_recovery_triggered() no longer uses exhausted watchdog attempts as a recovery escape, even
when stage-one verification is disabled. If attempts are exhausted and the stage-one image is
subsequently damaged, stage-zero jumps to it without checking it, so neither stage-one rollback nor
its recovery fallback can run.
Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Removing the exhausted-watchdog recovery trigger makes recovery depend on stage one running, but builds with stage-one verification disabled can jump to a damaged stage-one image.

## Fix Focus Areas
- stage0/recovery_entry.c[27-31]
- stage0/jump_stage1.c[74-76]
- stage0/jump_stage1.c[126-140]

## Recommended Fix
Add a persistent stage-one progress marker: clear it before the stage-zero jump, set it when stage one reaches boot selection, and retain a watchdog recovery escape when attempts are exhausted and the previous stage-one run never reached that marker. Continue handing watchdog resets that reached stage one to its rollback policy.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


Comment thread stage0/recovery_entry.c
Comment on lines -29 to -31
if (reason == EOS_RESET_WATCHDOG &&
bctl->boot_attempts >= bctl->max_attempts)
return true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

1. Damaged stage-one prevents recovery 🐞 Bug ☼ Reliability

ebldr_recovery_triggered() no longer uses exhausted watchdog attempts as a recovery escape, even
when stage-one verification is disabled. If attempts are exhausted and the stage-one image is
subsequently damaged, stage-zero jumps to it without checking it, so neither stage-one rollback nor
its recovery fallback can run.
Agent Prompt
## Issue description
Removing the exhausted-watchdog recovery trigger makes recovery depend on stage one running, but builds with stage-one verification disabled can jump to a damaged stage-one image.

## Fix Focus Areas
- stage0/recovery_entry.c[27-31]
- stage0/jump_stage1.c[74-76]
- stage0/jump_stage1.c[126-140]

## Recommended Fix
Add a persistent stage-one progress marker: clear it before the stage-zero jump, set it when stage one reaches boot selection, and retain a watchdog recovery escape when attempts are exhausted and the previous stage-one run never reached that marker. Continue handing watchdog resets that reached stage one to its rollback policy.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

Comment thread stage0/recovery_entry.c
Comment on lines +27 to +32
/* Exhausted boot attempts are not a recovery trigger. An image that
* hangs is reset by the watchdog until boot_attempts reaches
* max_attempts, and that is the condition on which stage-1's
* eos_boot_policy_select() rolls back to the other slot. Diverting it
* to recovery here meant the rollback never ran. Stage-1 still enters
* recovery itself when there is no valid slot to roll back to. */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Both images keep rebooting

Removing this check leaves devices rebooting forever when both slots contain valid images that hang. eos_boot_policy_select() switches slots and clears boot_attempts whenever the limit is reached. It does not remember that the alternate image already failed, so the two images keep getting fresh attempts instead of reaching recovery. Previously, stage-0 stopped this loop.

Keep the first rollback, but enter recovery if the fallback also exhausts its attempts. Add a test for that sequence.

Prompt To Fix With AI
This is a comment left during a code review.
Path: stage0/recovery_entry.c
Line: 27-32

Comment:
**Both images keep rebooting**

Removing this check leaves devices rebooting forever when both slots contain valid images that hang. `eos_boot_policy_select()` switches slots and clears `boot_attempts` whenever the limit is reached. It does not remember that the alternate image already failed, so the two images keep getting fresh attempts instead of reaching recovery. Previously, stage-0 stopped this loop.

Keep the first rollback, but enter recovery if the fallback also exhausts its attempts. Add a test for that sequence.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!

Linked issues

#161 A hung new image goes to recovery after a watchdog reset instead of rolling back to the good slot · 2 of 2 met

# Requirement Status Notes
1 A hung new image with a valid good slot in the other slot rolls back to that slot. ✅ Met Stage-0 no longer diverts exhausted attempts to recovery; the stage-1 policy rolls back to a valid alternate. The new test covers a hung test boot of B rolling back to A.
2 If no valid slot is available for rollback, the device enters recovery. ✅ Met The policy returns EOS_ERR_NO_IMAGE when the alternate is invalid, and stage-1 enters recovery when no boot target is selected. The new test covers this case.

Sourcery assessment

Needs a human reviewer. After the watchdog-attempt limit, stage-0 now hands control to stage-1, whose policy may change the active slot and boot-control rollback state. Those changes can survive reverting this patch, but they are bounded and repairable; if no slot is valid, stage-1 still enters recovery.


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

@Aswin-V
Aswin-V merged commit 3babea1 into master Nov 2, 2026
37 of 38 checks passed
@Aswin-V
Aswin-V deleted the loop/2026-11-02-eboot162-watchdog-rollback branch November 2, 2026 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A hung new image goes to recovery after a watchdog reset instead of rolling back to the good slot

1 participant