[Feature]: BackgroundAgentsProvider cannot tell a task that is running on another process from one whose owner died

Description

Problem. BackgroundAgentsProvider splits a background task into two halves. The record (BackgroundTaskInfo: id, status, description, result_text, error_text) is serialized into session.state, so it travels with the session to whichever process loads it. The runtime (_RuntimeState.in_flight_tasks, the asyncio.Task objects, and _RuntimeState.background_sessions, the child AgentSessions) lives in self._runtime, a process-local dict keyed by session_id. The comment in BackgroundAgentsProvider.__init__ says this runtime "cannot survive process restarts. If the provider instance is lost, _refresh_task_state() marks orphaned tasks as LOST." _refresh_task_state does exactly that for any RUNNING record without a local in-flight task, and then saves the LOST status back into session state. before_run calls _refresh_task_state on every turn, before the model sees anything.

That is correct when the process that started the task is dead. It is wrong when that process is alive and simply is not the one serving this turn, and the provider has no way to tell the two apart. Tasks outliving a run are also a case the provider supports: before_run refreshes "to get accurate statuses for any tasks that completed between turns". So the question is not whether a task can span turns, but whether the next turn lands on the same process.

Our deployment: a web host running two replicas behind a load balancer without session affinity, with conversation state in Redis. Any turn can land on either replica. agent-framework-core: 1.19.0 (tracking main).

Scenarios that fail today

Healthy task marked LOST, then duplicated. Turn N runs on process A; the model calls background_agents_start_task, and the run ends while the task is still running (see scenario 3 for how). Turn N+1 is routed to process B. B loads the session, before_run → _refresh_task_state finds the task RUNNING with no entry in B's in_flight_tasks, marks it LOST and persists that. The model's injected status block now reads Task 1 [lost]; background_agents_get_task_results returns "Task state was lost (reference unavailable)." The model does the reasonable thing and starts the same work again as task 2 on B. Meanwhile task 1 is still running on A. Two copies of the same delegated work now run concurrently against the same tools, possibly writing the same files, and the user pays for both.
The original's result is orphaned. When task 1 finishes on A, nothing records it: finalization only happens inside a run on A, and on A's next turn for this session _refresh_task_state skips the record because the persisted status is now LOST, not RUNNING (the loop continues on anything that isn't RUNNING). The completed work is never delivered. It sits in A's in_flight_tasks until release_session or process exit.
Runs end with live tasks despite the instruction. DEFAULT_BACKGROUND_AGENTS_INSTRUCTIONS says "Always wait for outstanding tasks to finish before you finish processing." That is a request to the model, and several paths end the run no matter what the model intends. (a) A tool that requires approval ends the run with a pending approval request, and the user may answer hours later, typically on a different process. (b) max_function_calls sets tool_choice="none" (_disable_tools_at_function_call_limit), and max_duration_seconds takes the same path, so the model cannot call background_agents_wait_for_first_completion even if it wants to. (c) Provider errors abort the run. (d) The user presses stop. Each of these leaves RUNNING records in session state, and on a multi-instance host each one becomes scenario 1 on the next turn with probability roughly (N-1)/N.
background_agents_continue_task fails across processes. Even when a task completed and was finalized on A (so its record is COMPLETED, result_text persisted, and results are readable anywhere), continuing it on B returns "Error: Session for task 1 is no longer available." The child AgentSession exists only in A's background_sessions.
Restarts and rolling deploys. Same sequence as scenario 1, except A is gone. Here LOST is the right answer. We are not asking to hide it. The point is that the provider currently reaches the same verdict for "owner dead" (correct) and "owner alive elsewhere" (wrong), because the only signal it has is its own process's memory.
release_session is process-local. A host that deletes a conversation calls release_session on whichever process handles the delete. If the tasks run on another process, it finds no runtime and returns without doing anything. The tasks keep running, and keep calling tools, for a conversation that no longer exists.
Waiting and listing cannot see other processes. On B, background_agents_wait_for_first_completion([1]) finds nothing in in_flight_tasks, refreshes (marking LOST), and returns "Task 1 is not running; current status: lost." background_agents_get_all_tasks reports the same. B cannot observe a task on A at all, so it cannot even choose to wait for it.

Expected behavior. A host with a shared store should be able to give the provider enough information to (a) mark LOST only when the owning process is actually gone, (b) report a task running on another instance as running, not lost, (c) deliver a task's result from whichever process serves the next turn, and ideally (d) signal cancellation to the owner. Hosts without a shared store keep today's single-process behavior unchanged.

Proposed shape (upstream's call). A small pluggable seam on BackgroundAgentsProvider, e.g. a BackgroundTaskRuntimeStore / liveness protocol with an in-memory default that reproduces current semantics:

The host provides an opaque qualified session key (namespace + session identity) and the owner process records a lease per (qualified_session_key, task_id) when it starts or continues a task, renews it while the task runs, and clears it on finalization.
_refresh_task_state asks the store before marking LOST: lease present and fresh → leave RUNNING (or a distinct status such as RUNNING_ELSEWHERE, which the tools render as "running on another instance"); lease missing or expired → LOST as today.
When a task finishes, the owner writes its outcome (status, result_text/error_text) to the store from a done-callback, not only inside a later run. Any process's _refresh_task_state then picks it up, which fixes scenario 2 and makes wait_for_first_completion able to poll a remote task instead of returning immediately.
Optionally, a cancel signal the store propagates to the owner, so release_session on one process cancels tasks owned by another (scenario 6).
Continuing a task on a non-owner (scenario 4) is harder, since it needs the child session. Serializing the child AgentSession into the store on finalization would cover it where the child agent's session is serializable. Otherwise a clear "continue is only available on the instance that ran the task" error is still better than today's generic message.

Nothing here requires the framework to ship a distributed store. A protocol plus the in-memory default is enough; hosts bring Redis or anything else. A reference Redis implementation would fit naturally next to the existing agent-framework-redis package, but that is optional.

Alternatives considered

Rely on "always wait before finishing". This is an instruction, not enforcement, and scenario 3 lists four paths where the model cannot comply. The provider itself already expects tasks to span turns.
Enforce it: cancel or await outstanding tasks at run end. Awaiting blocks the run, and so the user, for the full task duration, which defeats the point of background work. Cancelling at an approval interrupt throws away delegated work right before the user approves the step that needed it.
Session affinity at the load balancer. Fixes scenarios 1, 2, 4 and 7 while every process is up, but breaks exactly on restarts and rolling deploys, and many hosts (serverless containers, some managed ingresses) cannot pin reliably. It also does nothing for scenario 6 when the delete request lands elsewhere.
Durable execution (Durable Agents). Solves durability for a different hosting model and is much heavier than a liveness lease. It does not integrate with BackgroundAgentsProvider, which is what harness-based hosts use.
Host-side wrapper. Possible, but only by reaching into the private _runtime / _RuntimeState / _refresh_task_state, which every framework bump can break. The decision point (_refresh_task_state choosing LOST) is inside the provider, so the seam belongs there.

Code Sample

from typing import Protocol

class BackgroundTaskRuntimeStore(Protocol):
    async def acquire(self, qualified_session_key: str, task_id: int, *, ttl_seconds: float) -> None: ...
    async def renew(self, qualified_session_key: str, task_id: int, *, ttl_seconds: float) -> None: ...
    async def is_alive(self, qualified_session_key: str, task_id: int) -> bool: ...
    async def publish_outcome(self, qualified_session_key: str, task_id: int, info: BackgroundTaskInfo) -> None: ...
    async def fetch_outcome(self, qualified_session_key: str, task_id: int) -> BackgroundTaskInfo | None: ...
    async def request_cancel(self, qualified_session_key: str) -> None: ...

provider = BackgroundAgentsProvider(
    agents=[researcher, writer],
    runtime_store=MyRedisBackgroundTaskRuntimeStore(redis),  # default: in-memory, today's behavior
)

Language/SDK

Both

Eduard van Valkenburg (@eavanvalkenburg)

Eduard van Valkenburg (@eavanvalkenburg) , I dug into this and can confirm the race exists exactly as described in _refresh_task_state (_background_agents.py, lines ~235–260): in_flight is looked up only in the process-local _RuntimeState.in_flight_tasks, so any RUNNING record with no local entry gets marked LOST regardless of whether the owning process is dead or just not the one serving this turn. I traced all seven scenarios against the current code and they all check out.

I'd like to take this on, implementing the BackgroundTaskRuntimeStore protocol as proposed:

class BackgroundTaskRuntimeStore(Protocol):
    async def acquire(self, qualified_session_key: str, task_id: int, *, ttl_seconds: float) -> None: ...
    async def renew(self, qualified_session_key: str, task_id: int, *, ttl_seconds: float) -> None: ...
    async def is_alive(self, qualified_session_key: str, task_id: int) -> bool: ...
    async def publish_outcome(self, qualified_session_key: str, task_id: int, info: BackgroundTaskInfo) -> None: ...
    async def fetch_outcome(self, qualified_session_key: str, task_id: int) -> BackgroundTaskInfo | None: ...
    async def request_cancel(self, qualified_session_key: str) -> None: ...

Implementation plan:

BackgroundAgentsProvider.__init__ gains an optional runtime_store: BackgroundTaskRuntimeStore | None = None, defaulting to an in-memory implementation that reproduces today's single-process semantics exactly — no behavior change for hosts that don't configure one. This mirrors the existing SessionStore protocol already in _sessions.py, so a RedisBackgroundTaskRuntimeStore could sit alongside RedisStore in agent-framework-redis.
background_agents_start_task calls acquire() on start; a wrapper coroutine around _run_agent(...) renews the lease on a configurable interval (defaulting to ttl_seconds / 2) for the life of the task, rather than a separate heartbeat task, so renewal is tied to the task's own lifecycle. A done-callback on the underlying asyncio.Task calls publish_outcome the instant the task settles, from whichever process owns it — today _finalize_task only runs inside a run on the owner, and never at all once a record's been marked LOST, so this is what actually delivers the orphaned result.
_refresh_task_state calls is_alive() before marking LOST: lease present and fresh → leave RUNNING; lease missing/expired → check fetch_outcome() and finalize from that if present, LOST otherwise, same as today.
release_session calls request_cancel() before the current early return at self._runtime.get(session_id) is None (lines 374–378) — today that early return is exactly why a delete on a non-owning process silently no-ops, so the cancel call has to happen ahead of it, not alongside it.
request_cancel(qualified_session_key) is session-granular by design, matching release_session's own scope (cancel everything for a session, not one task). Happy to add an optional task_id: int | None = None if partial cancellation turns out to matter.
For continue_task on a non-owner, since the child AgentSession isn't serialized into the store here, the error becomes "Task {task_id} was completed on another instance; continue is not available cross-process in this release." — more specific than today's generic "session no longer available" message.
qualified_session_key will be session.session_id used as-is, with no built-in namespacing in the protocol — hosts running multiple namespaced environments would wrap or subclass the store rather than the protocol handling it. Flagging this as an open question below rather than assuming it.
fetch_outcome/publish_outcome return and accept BackgroundTaskInfo directly, which couples any external store implementation (Redis, etc.) to importing that framework class. This is the same shape SessionStore already uses with AgentSession, so it's precedented, but worth naming since BackgroundTaskInfo is currently @experimental.

A few things I'd like alignment on before opening the draft PR:

qualified_session_key composition — I plan to use session.session_id directly, with no namespacing built into the protocol itself. Is that the right default, or should __init__ also accept something like session_key_prefix: str = "" or a key factory for hosts that need namespacing?
Lease renewal mechanism — I plan to implement renew via a wrapper coroutine around _run_agent(...) rather than a separate heartbeat task, so renewal lifecycle is tied to the task itself. Does that match what you had in mind, or would you rather see an independent heartbeat loop?
request_cancel granularity — the protocol as sketched cancels everything for a session key. release_session will call it ahead of the current early-return so the signal reaches the owner even when this process has no local runtime. Let me know if task-granular cancellation is needed instead.
TTL clock source — time.monotonic() is process-local and unusable across processes, so time.time() is the only semantically correct option here; flagging this as confirmation rather than an open preference, unless you'd rather the store manage its own timestamps.

Happy to open a draft PR once we're aligned on these four.

/plan I want to split this into two pieces 1) fix any issues with the existing and 2) start creating a design for broader ways of doing this, which might include the proposed protocol but might also leverage other infrastructure and so needs to be even more generic and shared across .Net and python

Summary

I read the full issue, the proposed BackgroundTaskRuntimeStore protocol from the earlier comment, and the current code in _harness/_background_agents.py. I independently traced the seven failure scenarios against the code rather than taking the earlier analysis at face value, and they hold up:

_refresh_task_state (around line 243) only checks the process-local runtime.in_flight_tasks dict. If a RUNNING task isn't found there, it is unconditionally marked LOST, with no way to distinguish "owner process died" from "owner process is alive but didn't serve this turn." before_run calls this on every turn (near the end of the file), so this runs before the model sees anything.
_finalize_task (the function that records a completed result) is only reachable from code paths running on the same process that started the task (background_agents_wait_for_first_completion, or _refresh_task_state finding a locally-tracked task done). Once a record has been marked LOST elsewhere, the continue at the top of the refresh loop means that process's own later finalization never updates it — exactly scenario 2.
release_session (around line 345) returns immediately if there is no local runtime for the session — exactly scenario 6, the cross-process delete no-op.
background_agents_continue_task depends on runtime.background_sessions.get(task_id), a process-local dict of child AgentSession objects — exactly scenario 4.
max_function_calls/max_duration_seconds do disable tools (_tools.py _disable_tools_at_function_call_limit), so a model mid-way through background delegation can have its tool access cut off while tasks are still RUNNING — supporting scenario 3(b).

I also confirmed .NET has a parallel, independently-implemented provider (dotnet/src/Microsoft.Agents.AI/Harness/BackgroundAgents/BackgroundAgentsProvider.cs, BackgroundAgentRuntimeState.cs) with the same per-process runtime-state shape, so this is not Python-only, which supports your framing that a shared design matters.

I did not look at the open PR's (#9040) implementation or diff, per the workflow's instructions; my assessment is based only on the current main code.

Recommendation

Splitting this into two tracks makes sense, and both are warranted — but for different reasons and with different scopes.

Track 1 — fix bugs in the existing single-process semantics. Independent of any cross-process design, the current code has at least one bug with no multi-process angle at all: scenario 2 (orphaned result never delivered) can happen on a single long-running process too, any time _refresh_task_state has already stamped a task LOST for some other reason and a later finalization is silently dropped by the continue. That's a plain logic defect worth fixing on its own, with a regression test, regardless of what the broader design becomes.

Track 2 — a shared, generic design for cross-process/distributed task ownership. This is appropriately scoped as a design exercise, not a direct implementation of the BackgroundTaskRuntimeStore protocol as proposed. Reasons:

The protocol as sketched is Python-shaped (asyncio.Task, AgentSession as a direct payload type) and would need deliberate rethinking to also fit .NET's BackgroundAgentRuntimeState.cs/BackgroundAgentsProvider.cs, which has its own task/session model.
Your comment notes it "might also leverage other infrastructure" — e.g., this provider's problem (process ownership, liveness, handoff of continuation state) overlaps with concerns SessionStore/FileSessionStore and checkpoint/session-store work already address elsewhere in the framework (see _sessions.py SessionStore and ADR 0039-shared-agent-session-store.md). A design pass should check whether a generic "ownership/liveness lease" concept is better placed as shared infrastructure other providers could reuse, rather than a one-off protocol bolted onto BackgroundAgentsProvider.
A distributed-store protocol touches the public API surface for an @experimental class (BackgroundTaskInfo/BackgroundAgentsProvider), so the design should reach a documented decision (likely an ADR, per this repo's docs/decisions/ process) before an implementation PR is accepted, rather than deciding the shape via a single contributor PR.

Proposed plan

Track 1 — targeted bug fix (Python, and check .NET for the analogous issue):
- Fix _refresh_task_state so a task whose status was previously set to LOST (or any non-RUNNING terminal status) doesn't permanently block a later finalization from a still-live owner process — e.g., don't continue past tasks that are locally in-flight and done, regardless of their currently persisted status.
- Add regression tests in test_harness_background_agents.py covering: a task finalizing after being marked LOST on the same process, and confirm background_agents_get_task_results returns the real result/error rather than "state was lost" once finalized.
- Check whether dotnet/src/Microsoft.Agents.AI/Harness/BackgroundAgents/BackgroundAgentsProviderTests.cs has the same gap and needs a parallel fix.
- Resolution check: new tests fail on current main and pass after the fix; existing tests for single-process LOST behavior (owner genuinely gone) keep passing unchanged.
Track 2 — generic cross-process design (no code yet):
- Write a short design note (likely as an ADR in docs/decisions/, following adr-template.md/adr-short-template.md) that frames the problem independent of the specific protocol already proposed: what "task ownership," "liveness," and "result handoff" need to mean generically, and where that concept should live (owned by BackgroundAgentsProvider alone vs. shared with SessionStore/session-state infrastructure).
- Evaluate at least the already-proposed BackgroundTaskRuntimeStore protocol as one option, alongside alternatives such as representing task ownership purely through existing session/state primitives, or a narrower "liveness lease" abstraction usable by both languages.
- Confirm with maintainers whether the .NET team should co-author this given the "shared across .NET and Python" requirement.
- Resolution check: the ADR is reviewed and merged (or explicitly rejected with rationale) before any implementation PR for the distributed store lands; the earlier contributor's draft PR would then be re-scoped against the agreed design rather than merged as-is.

This is a proposal only — no code, tests, or ADR have been written yet.

Generated by Issue Planning for #8760 · copilot · auto · 51.9 AIC · ⌖ 7.1 AIC · ⊞ 10.9K · ◷

Eduard van Valkenburg (@eavanvalkenburg)

Thanks Eduard van Valkenburg (@eavanvalkenburg), and thanks for the plan. I checked both tracks against current main.

Track 1: there is a real single-process gap. _finalize_task updates the record and removes the task from in_flight_tasks, but the session state is persisted by the host only after the run completes. If the run then fails, is stopped, or is resumed from an older snapshot, the persisted record still says running while the in-memory task is gone, so the next refresh marks it lost and drops a result that already exists. I reproduced it with a snapshot taken before finalization: main returns "Task state was lost", and keeping finished tasks in in_flight_tasks until clear_completed_task or release_session returns the real result, with the existing 47 tests passing. The trade-off is that finished Task objects stay in memory until cleared or released. I'd like to open this as a small Track 1 PR (marked as part of #8760, not closing it) with a regression test, if you agree. .NET has the same pattern in FinalizeTask, so it may need a parallel fix.

Track 2: .NET reproduces all seven scenarios, I found no existing lease or ownership abstraction in either language, ADR-0039 covers the .NET session store only, and the persisted task records differ between languages (casing and status values). I'd suggest the ADR define a language-neutral contract for the store entries (lease and outcome), not for the session record. I'm glad to draft it, with #9040 as a reference prototype, ideally with a .NET co-author. Should I mark #9040 as a draft meanwhile so reviewers don't treat it as a merge candidate?

Python: Record generation token on background tasks to prevent wrong …

Python: Simplify background task retention without generation tracking (

Python: Track execution generation on background tasks (microsoft#8760)

Eduard van Valkenburg (@eavanvalkenburg)

Eduard van Valkenburg (@eavanvalkenburg) Since the broader design here is meant to be shared across .NET and Python, MessageInjectingChatClient also has a process-local state that I think should also be addressed.

Its pending queue lives in AgentSession.StateBag, guarded by a per-process ConditionalWeakTable<AgentSession, SemaphoreSlim>, and the class is sealed. In a host with several replicas and no session affinity, the run streams from one replica while the user's mid-run message can arrive at another in a load-balanced architecture. EnqueueMessagesAsync on the receiving replica only reaches a deserialized copy of the session, so the live run never sees the message.

It would help if the design covered injected-message delivery as well as background-task runtime state, for example:

A pluggable pending-message source that MessageInjectingChatClient drains before each inner call, alongside its in-memory queue, so a shared store can feed it.
An option to yield drained messages as response updates, so streaming clients that own the transcript learn where each message was inserted.

Happy to contribute the .NET side of (1) and (2) once the shape is agreed.

Navigation Menu

Repository navigation

[Feature]: BackgroundAgentsProvider cannot tell a task that is running on another process from one whose owner died #8760

Description

Code Sample

Language/SDK

Summary

Recommendation

Proposed plan

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature]: BackgroundAgentsProvider cannot tell a task that is running on another process from one whose owner died #8760

Description

Description

Code Sample

Language/SDK

Activity

karthik-0306 commented on Sep 25, 2026

eavanvalkenburg commented on Oct 14, 2026

github-actions commented on Oct 14, 2026

Summary

Recommendation

Proposed plan

karthik-0306 commented on Oct 14, 2026

JeffinSiby commented on Oct 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions