Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/spec/SPEC-011-daemon-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,13 @@ The daemon writes its PID to `~/.netclaw/netclaw.pid` for lifecycle management.
shutdown. The daemon handles SIGTERM by draining active sessions and stopping
the actor system cleanly.

The session journals each accepted input before it acknowledges the source.
The record retains the text, media, source message ID, and original authority.
A completed reply, a started tool batch, or a terminal failure consumes the
input ID. The actor restores unconsumed records from the journal after a cold
start. A retry with the same stable source message ID does not add a second
record. A source without a stable ID cannot use this deduplication rule.

During drain, a session can stop a tool task that waits only for durable
approval prompts. The session waits for the tool task to stop before it
acknowledges drain. Its journal retains the prompts and completed sibling
Expand Down
2 changes: 2 additions & 0 deletions openspec/changes/resume-interrupted-sessions/.openspec.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-09-18
85 changes: 85 additions & 0 deletions openspec/changes/resume-interrupted-sessions/design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
## Context

The actor stores completed turns and tool work. It does not store a model only request before its acknowledgment. Its buffer is also actor local.

The reminder manager already persists schedules, routes a `current_session` turn, deduplicates delivery, and records the result.

## Goals

- Store accepted input before acknowledgment.
- Stop an eligible model call during graceful drain.
- Wake the session within ten minutes through the reminder manager.
- Restore pending input under its recorded authority.
- Add no channel adapter or second delivery path.

## Non Goals

- Resume after an ungraceful crash.
- Replay a turn after a tool starts or partial text reaches a user.
- Create a daemon restart command or a configuration property.

## Decisions

### D1. The session journal owns accepted input

`InputAdmitted` stores an `InputId`, content, media, source ID, executable text, and `TurnContextRecord`. The actor persists it before acknowledgment.

`TurnRecorded` and `ToolBatchStarted` close the input IDs that they consume. `InputClosed` closes input after a terminal path without either event.

`SessionState` keeps the ordered pending input ledger. A bounded source ID ledger rejects a retry after a lost acknowledgment.

### D2. Drain produces a standard reminder

The actor gives the model call a short completion grace. It cancels the call and waits for its task to stop when the grace ends.

The actor creates a standard one shot `ReminderDefinition` only when these conditions hold:

- pending input exists;
- no tool batch started;
- no partial text reached a subscriber;
- the stored turn context is valid;
- the existing reminder path supports the session channel.

The reminder expires ten minutes after the interruption. The restart manifest stores the definition with the active session list.

### D3. The reminder manager owns wakeup and delivery

Startup registers each fresh definition through `SaveReminderCommand`. The reminder uses `DeliveryKind.CurrentSession` and the existing gateway path.

The daemon adds no route binder, channel state, retry loop, or resume candidate protocol. The reminder manager owns persistence, delivery retries, and deduplication.

### D4. The reminder is a trigger

The reminder text is generic: `Resume the work that was interrupted by the daemon restart.`

When this internal reminder arrives, the actor restores its pending input and original `TurnContextRecord`. The model sees the stored input and the restart notice.

The reminder does not replace the original authority. A missing or incompatible context causes a visible operator warning and no model call.

## Ordered Flow

This flow is schematic. It omits persistence callbacks and reminder delivery acknowledgments.

```text
input -> session: SendUserMessage
session -> journal: InputAdmitted
journal -> session: stored
session -> source: CommandAck
stop -> session: PrepareForDaemonRestart
session -> model: cancel and await stop
session -> stop: ReminderDefinition or no reminder
stop -> manifest: active sessions and reminders
start -> reminder manager: SaveReminderCommand
reminder manager -> existing gateway: current_session reminder
gateway -> session: SendUserMessage
session -> journal: restore pending input and authority
session -> model: resume prior work
```

## Risks

- A stale reminder can start old work. The one shot definition has an absolute expiration.
- A tool can have an uncertain effect. `ToolBatchStarted` closes its input before execution and blocks this path.
- A partial reply can repeat text. The actor records transient text emission and does not create a reminder.
- A reminder can register twice after a process failure. Its stored ID makes `CreateOnly` registration idempotent.
- A channel can lack `current_session` support. The actor creates no reminder for that session.
28 changes: 28 additions & 0 deletions openspec/changes/resume-interrupted-sessions/proposal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
## Why

A session acknowledges user input before the journal stores it. A graceful stop can therefore lose an active request or its queued input.

Source: [PRD-001 FR-003 and FR-016](../../../docs/prd/PRD-001-netclaw-mvp.md).

## What Changes

- Persist each accepted input before its acknowledgment.
- Retain its content, order, source ID, media, and original authority.
- Cancel an eligible model call during graceful drain.
- Put a short lived `current_session` reminder in the restart manifest.
- Register that reminder through the existing reminder manager after startup.
- Restore the pending input under its original authority when the reminder arrives.
- Keep approvals, partial replies, and turns with possible tool effects quiet.

This change adds no channel code and no configuration property. It excludes crash recovery and replay of uncertain tool effects.

## Capabilities

### Modified Capabilities

- `session-resume`: Durable input admission and a bounded restart reminder.
- `daemon-container`: The state volume retains the restart manifest across a graceful pod replacement.

## Impact

This change affects session persistence, graceful drain, the restart manifest, and reminder registration. Existing channel gateways deliver the reminder without new adapters.
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
## MODIFIED Requirements

### Requirement: Image entrypoint auto-starts netclawd

The image SHALL start `tini` as PID 1. Its supervisor SHALL start `netclawd` and forward a container stop signal to it. The supervisor SHALL wait for the daemon to finish graceful drain before it exits.

#### Scenario: docker run starts the daemon

- **GIVEN** the image is present locally with valid configuration and identity files
- **WHEN** an operator starts the container
- **THEN** `tini` is PID 1 and the supervisor starts `netclawd`
- **AND** the daemon binds its HTTP port within 60 seconds

#### Scenario: Pod stop preserves a restart reminder

- **GIVEN** the container has a persistent operator state volume and an eligible interrupted session
- **WHEN** the pod sends a graceful stop signal with enough termination time
- **THEN** the supervisor forwards the signal and waits for the daemon to exit
- **AND** the state volume retains the reminder for the next container start

### Requirement: Operator state mounts at /home/netclaw/.netclaw

The image SHALL declare `VOLUME /home/netclaw/.netclaw`. The volume SHALL hold identity, configuration, session data, and restart reminders. The image SHALL not include operator credentials or identity files.

#### Scenario: Operator bind-mounts an initialized home

- **GIVEN** an operator has an initialized Netclaw home on the host
- **WHEN** they mount it at `/home/netclaw/.netclaw` and start the container
- **THEN** the daemon reads identity and configuration from that directory
- **AND** it writes session state and restart reminders to the same directory

## RENAMED Requirements

- FROM: `Operator state mounts at /root/.netclaw`
- TO: `Operator state mounts at /home/netclaw/.netclaw`
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
## ADDED Requirements

### Requirement: Accepted input survives a graceful stop

The session SHALL store each accepted input before acknowledgment. The record SHALL retain its identity, order, content, media, source identity, and original authority.

Use the [engineering glossary](../../../../../docs/spec/GLOSSARY.md) for shared terms.

#### Scenario: Input acknowledgment follows its journal record

- **GIVEN** a session receives user input
- **WHEN** the journal stores its admission record
- **THEN** the session acknowledges the input
- **AND** cold recovery restores the pending input and its authority

#### Scenario: Journal failure rejects input

- **GIVEN** the journal cannot store an admission record
- **WHEN** the session receives input
- **THEN** the session rejects that input
- **AND** it starts no model call for that input

#### Scenario: A lost acknowledgment does not duplicate input

- **GIVEN** the journal stores input with a stable source ID
- **WHEN** the source retries that input
- **THEN** the session acknowledges the stored input
- **AND** it does not add a second pending record

### Requirement: Graceful drain creates only a safe restart reminder

The session SHALL create a restart reminder only after an eligible model task stops. It SHALL use the existing reminder definition and `current_session` delivery contract.

#### Scenario: An interrupted model call creates a reminder

- **GIVEN** a model call has pending admitted input
- **AND** no tool batch or partial reply exists
- **WHEN** graceful drain cancels the call and confirms its task stopped
- **THEN** the restart manifest stores one reminder for that session
- **AND** the reminder expires ten minutes after interruption

#### Scenario: A completed turn stays quiet

- **GIVEN** a model call completes during drain
- **AND** no admitted input remains pending
- **WHEN** the daemon starts again
- **THEN** it registers no restart reminder for that session

#### Scenario: A possible effect blocks the reminder

- **GIVEN** a tool batch started or partial text reached a subscriber
- **WHEN** graceful drain stops the session
- **THEN** the manifest contains no restart reminder for that turn
- **AND** the daemon reports the blocked session

### Requirement: A fresh restart reminder resumes stored work

The reminder manager SHALL deliver a fresh restart reminder through its existing `current_session` path. The session SHALL restore pending input under its recorded authority.

#### Scenario: A fresh reminder resumes the pending input

- **GIVEN** the restart manifest contains a reminder that has not expired
- **WHEN** the daemon starts
- **THEN** startup registers the reminder through `SaveReminderCommand`
- **AND** the session resumes the stored input without a user prompt

#### Scenario: The original authority remains in force

- **GIVEN** a restart reminder wakes a session with pending input
- **WHEN** the session starts the model call
- **THEN** it restores the recorded requester, audience, and trust boundary
- **AND** reminder automation authority does not replace that context

#### Scenario: An expired reminder stays quiet

- **GIVEN** the reminder expiration is in the past
- **WHEN** startup reads the restart manifest
- **THEN** it does not register or deliver that reminder
- **AND** it logs one warning

#### Scenario: A channel lacks current session delivery

- **GIVEN** an interrupted session uses a channel that the reminder manager cannot address
- **WHEN** graceful drain classifies the session
- **THEN** the actor creates no restart reminder
- **AND** no channel adapter is added by this change
24 changes: 24 additions & 0 deletions openspec/changes/resume-interrupted-sessions/tasks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
## 1. Durable input

- [x] 1.1 Persist accepted input before acknowledgment.
- [x] 1.2 Store pending input and recent source IDs in snapshots.
- [x] 1.3 Close input from completed, failed, and tool started turns.
- [x] 1.4 Verify journal, snapshot, order, and source retry behavior.

## 2. Graceful drain

- [ ] 2.1 Retain and cancel the active model task after a short grace.
- [ ] 2.2 Return one standard reminder definition for eligible pending input.
- [ ] 2.3 Exclude approvals, tool work, partial replies, and unsupported channels.

## 3. Existing reminder path

- [ ] 3.1 Store reminder definitions in the restart manifest.
- [ ] 3.2 Register fresh reminders through the reminder manager after startup.
- [ ] 3.3 Restore pending input under its original context when the reminder arrives.
- [ ] 3.4 Verify expiration, duplicate registration, and a cold session wakeup.

## 4. Verification

- [ ] 4.1 Update SPEC-011 and the operations skill.
- [ ] 4.2 Run actor and daemon tests, evals, Slopwatch, headers, and OpenSpec validation.
63 changes: 63 additions & 0 deletions src/Netclaw.Actors.Tests/Protocol/SerializationRoundTripTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
using Netclaw.Actors.Reminders;
using Netclaw.Actors.Serialization;
using Netclaw.Actors.Sessions;
using Netclaw.Configuration;
using Netclaw.Tools;
using Xunit;
using static Netclaw.Actors.Sessions.SessionProtocol;
Expand Down Expand Up @@ -132,6 +133,68 @@ public void TurnRecorded_round_trips()
Assert.Equal(original.RecordedAtMs, result.RecordedAtMs);
}

[Fact]
public void Admitted_input_and_terminal_ids_survive_journal_and_snapshot_round_trips()
{
var sessionId = new SessionId("C99999/1708531200.000100");
var inputId = new InputId("input-1");
var admitted = new InputAdmitted
{
SessionId = sessionId,
InputId = inputId,
SourceMessageId = "event-1",
UserMessage = new SerializableChatMessage { Role = ChatRole.User, Content = "Continue the task" },
ExecutableText = "Continue the task",
TurnContext = new TurnContextRecord
{
SessionId = sessionId,
TurnId = "turn-1",
Audience = TrustAudience.Personal,
Boundary = new TrustBoundary("slack:C99999"),
ChannelType = "slack",
RequesterSenderId = new SenderId("U123"),
RequesterPrincipal = PrincipalClassification.Operator
},
AdmittedAtMs = 1_700_000_000_000
};

var restored = RoundTrip(admitted);
Assert.Equal(admitted.InputId, restored.InputId);
Assert.Equal(admitted.UserMessage.Content, restored.UserMessage.Content);
Assert.Equal(admitted.TurnContext?.Boundary, restored.TurnContext?.Boundary);
Assert.Equal(admitted.TurnContext?.RequesterSenderId, restored.TurnContext?.RequesterSenderId);

var snapshot = RoundTrip(new SessionSnapshot
{
PendingInputs = [admitted],
RecentSourceMessageKeys = ["slack:event-1"]
});
Assert.Single(snapshot.PendingInputs);
Assert.Equal(inputId, snapshot.PendingInputs[0].InputId);
Assert.Equal("slack:event-1", Assert.Single(snapshot.RecentSourceMessageKeys));

var completed = RoundTrip(new TurnRecorded
{
SessionId = sessionId,
UserMessage = admitted.UserMessage,
AssistantReply = new SerializableChatMessage { Role = ChatRole.Assistant, Content = "Done" },
ConsumedInputIds = [inputId]
});
Assert.Equal(inputId, Assert.Single(completed.ConsumedInputIds));

var closed = RoundTrip(new InputClosed { SessionId = sessionId, InputIds = [inputId] });
Assert.Equal(inputId, Assert.Single(closed.InputIds));

var toolStarted = RoundTrip(new ToolBatchStarted
{
SessionId = sessionId,
UserMessage = admitted.UserMessage,
AssistantMessage = new SerializableChatMessage { Role = ChatRole.Assistant },
ConsumedInputIds = [inputId]
});
Assert.Equal(inputId, Assert.Single(toolStarted.ConsumedInputIds));
}

[Fact]
public void TurnRecorded_round_trips_preserving_value_object_source_ids()
{
Expand Down
Loading
Loading