Skip to content

[agentserver-core] Fix steering future leaks and acknowledgment misrouting - #49233

Closed
Eduard van Valkenburg (eavanvalkenburg) wants to merge 6 commits into
Azure:mainfrom
eavanvalkenburg:agentserver-steering-queue
Closed

Eduard van Valkenburg (eavanvalkenburg) wants to merge 6 commits into
Azure:mainfrom
eavanvalkenburg:agentserver-steering-queue

Conversation

@eavanvalkenburg

@eavanvalkenburg Eduard van Valkenburg (eavanvalkenburg) commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fix request-owned steering acknowledgments for resilient multi-turn tasks. Rejected queue appends previously left unreturned futures registered, so repeated SteeringQueueFull responses could grow per-chain memory without a durable queue slot.

The fix persists opaque acknowledgment IDs alongside durable queue slots and binds append, drain, cancellation, and result delivery by ID rather than list position. Append and drain retries reconcile only the exact transition whose conditional PATCH may have committed before returning a conflict. A later drain cannot redispatch stale active state or consume the next queue head.

Turn completion now acts as a persistence barrier: return, failure, and cancellation are acknowledged, and queued successors are dispatched, only after the terminal task-state PATCH succeeds. If persistence fails, the current caller receives that persistence error, local queued handles receive TaskConflictError(current_status="in_progress"), and durable state remains available for recovery. Waiter-aware write-lock reclamation also prevents rejected remote steers from retaining per-task locks.

The branch is rebased onto the current Core 2.3.0b2 preview line. It no longer creates a stable 2.2.1 release entry; a Chronus fix change records the work for the next preview release.

Tests use the real local file provider and deterministic failure injection for queue saturation, pre/post-commit append failures, committed drain conflicts, failed Phase-3 cleanup plus failed terminal persistence, duplicate inputs/public IDs, legacy/remote backlogs, attachment drains, cancellation, return/raise paths, and waiter-safe lock cleanup. The full TaskManager suite passes (740 tests on Python 3.11), focused fault regressions pass on Python 3.10 and 3.14, and pylint, mypy, pyright, and CI-equivalent cspell pass.

Downstream dependency: microsoft/agent-framework#8794; steering review thread. The downstream dependency floor must not change until a patched AgentServer core release is officially published and verified.

All SDK Contribution checklist:

  • The pull request does not introduce [breaking changes]
  • CHANGELOG is updated for new features, bug fixes or other significant changes.
  • I have read the contribution guidelines.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

Copilot AI balanced review requested due to automatic review settings September 28, 2026 12:32
@github-actions github-actions Bot added Community Contribution Community members are working on the issue customer-reported Issues that are reported by GitHub users external to the Azure organization. Hosted Agents sdk/agentserver/* labels Sep 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution Eduard van Valkenburg (@eavanvalkenburg)! We will review the pull request and get back to you soon.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Ambiguous hosted PATCH outcomes can still remove a future for an input that was durably appended, corrupting acknowledgment ordering.

Review effort: Balanced
Findings: 1 High severity

Open (1)
What changed in this PR

Fixes steering acknowledgment-future leaks when queue appends fail.

Changes:

  • Serializes future registration and queue append.
  • Cleans up futures after failed or cancelled appends.
  • Adds failure, cancellation, capacity, and acknowledgment tests.
File Description
_manager.py Adds targeted future cleanup.
_decorator.py Makes registration and append atomic locally.
test_steering.py Expands steering failure coverage.
CHANGELOG.md Documents the fix.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Copilot AI review requested due to automatic review settings September 28, 2026 13:05
@eavanvalkenburg Eduard van Valkenburg (eavanvalkenburg) changed the title [agentserver-core] Prevent steering future leaks on failed queue appends [agentserver-core] Fix steering future leaks and acknowledgment misrouting Sep 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The concurrency-sensitive durable queue migration and cross-worker compatibility require final human validation.

Review effort: Balanced
Findings: None

Resolved since last review (1)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

A committed PATCH retry can still duplicate queued inputs and leave cancelled work executable.

Review effort: Balanced
Findings: 1 High severity · 1 Medium severity

Open (2)

Copilot AI balanced review requested due to automatic review settings October 5, 2026 05:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Durable queue and concurrency changes require human validation of hosted behavior and coordinated worker upgrades.

Review effort: Balanced
Findings: 1 High severity · 1 Medium severity

Open (2)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

A committed drain followed by a retry-induced conflict can skip an accepted input and strand its acknowledgment future.

Review effort: Balanced
Findings: 1 High severity

Open (1)
Resolved since last review (2)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Failed turn-start cleanup can leave stale drain state and dispatch a phantom input.

1 open finding
1 resolved since last review

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 12, 2026 08:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Rejected remote steers still accumulate unbounded entries in the manager’s etag cache.

0 open findings

1 resolved since last review
Previously missed (1)

In code that hasn't changed since last review

Medium severity Clear stale etags when reclaiming inactive remote task locks

sdk/​agentserver/​azure-ai-agentserver-core/​azure/​ai/​agentserver/​core/​tasks/​_manager.py:3192

The final-user path reclaims the remote task's lock but clears its etag only when _active_tasks_pop() previously marked cleanup pending. A rejected remote steer has no local active entry, while the initial _provider_get_tracked() still caches its etag, so the new 20-task regression empties _task_write_locks but leaves 20 entries in _task_etag_cache; repeated distinct remote IDs still grow manager memory indefinitely. Clear the etag whenever this path removes a no-longer-active lock, and extend the regression to assert the cache is empty.

🧠 Review effort: Balanced

Comment thread sdk/agentserver/azure-ai-agentserver-core/CHANGELOG.md Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Community Contribution Community members are working on the issue customer-reported Issues that are reported by GitHub users external to the Azure organization. Hosted Agents sdk/agentserver/*

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants