fix: persist immediate agent termination and emit lifecycle event

[arnabnandy7]

External termination of a WAITING, PAUSED, STUCK, or COMPLETED process currently changes its runtime status without updating the repository. A stored WAITING snapshot can therefore survive termination and be restored after runtime state is lost.

This change calls the repository update in the immediate-termination branch, allowing the existing lifecycle checkpoint policy to persist TERMINATED. It adds AgentProcessTerminatedEvent at the status transition for observability and cache invalidation, emitting once per transition. Interrupted action delays now return AGENT_TERMINATED so the caller preserves the terminal status instead of overwriting it with FAILED.

The repository checkpoint policy remains the single persistence path. No event-driven checkpoint listener or snapshot deduplication is added.

Follow-up to #1988, replacing the approach in #2008 as requested in #2005 (comment).

Validation covers immediate termination from all four states, snapshot version advancement, repeated termination, restoration after runtime loss, signal/policy event status ordering, action-requested termination, and interrupted delays.
Validation on JDK 21:

• Production and test compilation passed; mvn -B spotless:check -pl embabel-agent-api passed.
• Selected suites: PersistentAgentProcessRepositoryTest, SimpleAgentProcessTest, AbstractAgentProcessTerminationStatusOrderingTest, TerminationAgenticTest, and AgentActionDelayTest.
• 61 of 63 tests passed on the separate surefire:test run. Every new regression and all termination/persistence tests passed in both runs.
• Two existing AgentActionDelayTest zero-delay assertions failed their strict < 50 ms wall-clock threshold (66–163 ms observed on the separate run). The full lifecycle run also hit these assertions, with one passing on retry. These timing failures remain unresolved; the PR is draft for review.

Reproduction:

mvn -B test -pl embabel-agent-api -Dkotlin.compiler.daemon=false -Dtest=PersistentAgentProcessRepositoryTest,SimpleAgentProcessTest,AbstractAgentProcessTerminationStatusOrderingTest,TerminationAgenticTest,AgentActionDelayTest
mvn -B spotless:check surefire:test -pl embabel-agent-api -Dtest=PersistentAgentProcessRepositoryTest,SimpleAgentProcessTest,AbstractAgentProcessTerminationStatusOrderingTest,TerminationAgenticTest,AgentActionDelayTest

[simeshev]

Thank you for the fix. The approach is correct: the immediate-termination branch must persist, because it has no guaranteed next tick. Three items block approval.

1. AbstractAgentProcess.kt - ephemeral processes log a spurious ERROR

                setStatus(AgentProcessStatusCode.TERMINATED)
                platformServices.agentProcessRepository.update(this)

AbstractAgentProcessRepository.update logs at ERROR level for an ephemeral process:

                Attempted to update ephemeral AgentProcess [id={}].
                Ephemeral processes are not persisted.
                Operation skipped.

Termination of an ephemeral process is legal. Each call now writes an error line. Please guard the new call:

if (!processOptions.ephemeral) {
    platformServices.agentProcessRepository.update(this)
}

2. AbstractAgentProcess.kt - the event precedes the persist

        val previousStatus = _status.getAndSet(status)
        if (status == AgentProcessStatusCode.TERMINATED && previousStatus != status) {
            platformServices.eventListener.onProcessEvent(AgentProcessTerminatedEvent(this))
        }

setStatus runs before update. A listener that reads the process back from the repository still gets the old stored status. The PR names cache invalidation as a goal, so this order defeats it. Please emit the event after the persist, or document that the event gives no persistence guarantee.

3. AgentProcessEvent.kt - the new type changes existing listener behaviour

class AgentProcessTerminatedEvent(
    agentProcess: AgentProcess,
) : AgentProcessFinishedEvent(agentProcess)

LoggingAgenticEventListener.kt:370 matches is AgentProcessFinishedEvent, and EmbabelMetricsEventListener, MdcPropagationEventListener, and EmbabelSpanEventListener switch on the sibling types. Please confirm that no listener counts a process twice, and that the else branches handle the new type. A test for the metrics listener would help.

Minor - SimpleAgentProcessTest.kt

The new imports break the existing order. com.embabel.agent.api.common.PlatformServices and java.time.Duration sit above com.embabel.agent.api.common.StuckHandlerResult.

[arnabnandy7]

@simeshev thanks for the detailed review. I’ve addressed the points you raised:

• Immediate termination now skips repository updates for ephemeral processes.
• AgentProcessTerminatedEvent is emitted only after the terminated state has been persisted.
• Signal-, policy-, action-, and immediate-termination paths now follow the correct ordering without duplicate events.
• Added termination handling and coverage for metrics, MDC cleanup, and tracing. The logging listener already handles the new finished event safely, so no change was needed there.
• Fixed the import ordering.

[igordayen]

I'm adding the problem statement:

1. Agent process is running, waiting for human input — status in datastore: WAITING
2. Code calls terminateAgent() or throws TerminateAgentException — status changes to TERMINATED in memory only
3. PROBLEM HERE: datastore is not updated, still shows WAITING
4. Client makes a REST call to resume the agent
5. Agent reloads its state from the datastore — gets WAITING; termination is lost
6. Agent resumes as if it was never terminated

Fix: step 3 must also write TERMINATED to the database.

[arnabnandy7]

I'm adding the problem statement:

1. Agent process is running, waiting for human input — status in datastore: WAITING
2. Code calls terminateAgent() or throws TerminateAgentException — status changes to TERMINATED in memory only
3. PROBLEM HERE: datastore is not updated, still shows WAITING
4. Client makes a REST call to resume the agent
5. Agent reloads its state from the datastore — gets WAITING; termination is lost
6. Agent resumes as if it was never terminated

Fix: step 3 must also write TERMINATED to the database.

Thanks for adding the concrete scenario. The updated immediate-termination path now persists TERMINATED before emitting the lifecycle event, so a subsequent reload cannot restore the previous WAITING state.