Python: [Feature]: Python per-tool request-rate limiting and admission control for production AI agents #9239

Problem

Microsoft Agent Framework's Python SDK provides automatic function invocation, concurrent tool execution, and FunctionMiddleware for intercepting tool calls.

However, the framework does not currently provide an official, reusable mechanism for enforcing independent request-rate limits on locally executed tools before execution.

This creates a practical production reliability limitation.

Enterprise AI agents routinely invoke external services with different request quotas, including:

• REST APIs
• CRM and enterprise applications
• Payment and financial systems
• Database services
• Search and retrieval APIs
• MCP-backed tools
• Cloud infrastructure services

When an LLM requests multiple tools, the framework may execute those calls concurrently. A batch of otherwise valid requests can exceed the downstream service's requests-per-second or requests-per-minute quota.

This can cause:

1. HTTP 429 responses and unnecessary failures.
2. Increased end-to-end execution latency.
3. Excessive traffic to already rate-limited services.
4. Unpredictable agent behavior under concurrent workloads.
5. Duplicated rate-limit handling across applications.
6. Higher operational costs and wasted model interactions.
7. Unnecessary reduction in throughput when developers disable concurrency globally.

Existing allow_concurrent_invocation controls execution concurrency, not request frequency over time.

The existing model-client retry sample demonstrates recovering from rate-limit errors after they occur. It does not prevent locally executed tools from exceeding independent request quotas.

Core limitation: There is no official per-tool admission-control policy that determines whether an invocation may start based on the individual tool's configured request-rate quota.

The desired capability should be opt-in, preserve existing behavior by default, and work with existing function invocation middleware.

Use Case and Evidence

Real-world scenario: Enterprise multi-tool customer service agent

Consider a customer-service agent connected to three services:

These quotas are illustrative application configuration values, not published limits of any particular provider.

A customer asks the agent to retrieve several customer records, check recent payments, and search supporting documentation.

The LLM generates the following operations:

• 5 calls to lookup_customer
• 2 calls to get_payment_status
• 3 calls to search_documents

The model may request these operations within the same response.

The current function invocation scheduler can execute independent calls concurrently.

Consequently, several calls to the same external service may begin together, exceeding that service's rate limit.

Why current controls are insufficient

1. Global sequential execution

Setting:

allow_concurrent_invocation=False

can execute function calls sequentially.

However, sequential execution does not guarantee adherence to requests-per-second limits. Several short calls can still execute within the same second.

It also sacrifices useful concurrency between unrelated services.

2. Function invocation limits

max_function_calls limits the number of function executions, not their rate over time.

A configuration permitting 100 calls does not prevent 100 calls from being concentrated within a short interval.

3. Model-client retry middleware

The repository's existing:

python/samples/02-agents/auto_retry.py

demonstrates handling model-provider rate-limit errors.

It does not implement proactive rate limiting for locally executed tools.

4. Existing function middleware

Developers can build their own rate limiter using FunctionMiddleware, but the framework does not provide a focused official implementation or reference demonstrating:

• Atomic capacity reservations
• Independent per-tool quotas
• Safe concurrent invocation
• Burst management
• Cancellation handling
• Limit-exceeded responses
• Policy state lifetime
• Deterministic validation

These concerns must be addressed separately by application developers.

Reproducible demonstration

A minimal local test could define two asynchronous tools:

• slow_customer_lookup
• fast_document_search

Configure the first tool with a limit of two requests per second and a burst capacity of two.

Then:

1. Initiate five concurrent customer-lookup invocations.
2. Verify that only the permitted number are immediately admitted.
3. Confirm that additional requests are rejected before tool execution in fail-fast mode.
4. Verify that the independent document-search tool continues normally.
5. Advance a controlled monotonic clock.
6. Verify that admission capacity refills according to the configured policy.
7. Verify that simultaneous calls cannot bypass the limit through race conditions.

This demonstrates the capability without requiring external cloud accounts or API credentials.

Relevant implementation evidence

The following repository locations provide the existing integration points:

python/packages/core/agent_framework/_middleware.py

Contains FunctionMiddleware and FunctionInvocationContext.

python/packages/core/agent_framework/_tools.py

Contains function invocation scheduling and execution handling.

python/samples/02-agents/auto_retry.py

Provides model-provider rate-limit retry examples.

The requested capability is distinct from retry, batch serialization, and total function-call limits.

Desired Outcome

Provide an opt-in, concurrency-safe per-tool request-rate admission-control capability for locally executed Python tools.

A successful outcome should allow application developers to configure different request-rate quotas for individual tools while preserving concurrency between independent services.

Required behaviors

1. Independent per-tool quotas

Different tools should support independent rate policies.

For example:

• Customer lookup: 2 requests/second
• Payment status: 1 request/second
• Document search: 10 requests/second

Tools without configured policies must continue behaving normally.

2. Pre-execution admission control

Before invoking a protected tool, the policy must determine whether capacity is available.

If admission is denied, the underlying tool must not execute.

3. Concurrency-safe quota enforcement

Multiple simultaneous invocations must not exceed the available admission capacity because of race conditions.

4. Configurable burst capacity

Applications should be able to define permitted request bursts.

Burst and refill semantics should be explicit, predictable, and testable.

5. Independent execution

A rate-limited tool must not unnecessarily block unrelated tools with available capacity.

6. Clear limit-exceeded behavior

Applications should receive an identifiable outcome when a tool call is rejected by the admission policy.

The initial feature may use fail-fast rejection.

Bounded waiting could be supported later if maintainers prefer a staged implementation.

7. Correct cancellation handling

Caller cancellation must propagate correctly.

If bounded waiting is supported, cancelled waiters must be removed safely without corrupting accounting.

8. No implicit retries

The admission policy must not automatically replay rejected operations.

This is particularly important for non-idempotent operations that may produce external side effects.

9. Observability

Applications should be able to identify:

• Admitted invocations
• Rejected invocations
• Tool identity
• Applicable quota
• Waiting duration, when relevant

Sensitive tool arguments and authentication information should not be logged unnecessarily.

10. Backward compatibility

Existing tool invocation behavior must remain unchanged unless the policy is explicitly enabled.

Alternatives Considered

Alternative 1: Existing model-client rate-limit handling

Issue #1362 and merged PR #4223 address rate-limit retry behavior for model inference requests.

This helps recover from provider throttling but does not proactively restrict the rate of locally executed tools.

Alternative 2: Disable concurrent invocation

The existing allow_concurrent_invocation=False option serializes supported function execution.

However:

• Sequential calls may still exceed time-based quotas.
• Independent tools lose parallelism.
• No per-tool rate policy is expressed.
• Burst behavior remains undefined.

Alternative 3: Per-tool concurrency groups

Issue #7914 discusses per-tool concurrency controls.

Concurrency and rate limiting solve distinct problems.

Concurrency limits the number of simultaneous executions.

Rate limiting restricts the frequency at which requests are admitted over time.

Both capabilities may be useful together.

Alternative 4: Implement rate limiting inside each tool

Developers can manually implement rate limiting using asyncio synchronization primitives and a token-bucket algorithm.

However, this results in duplicated logic across multiple tools and applications.

Common concerns include:

• Race conditions
• Policy configuration
• State sharing
• Time accounting
• Cancellation
• Consistent error handling
• Observability

A reusable reference implementation would reduce this duplication.

Alternative 5: External API gateways

External gateways can enforce request quotas on supported network traffic.

However, not all tools execute through the same gateway, and local Python tools or locally invoked MCP integrations may require application-level admission control.

Alternative 6: Retry with exponential backoff

Retry policies operate after a failure has occurred.

They do not prevent excessive calls from being initiated.

Furthermore, automatic retries may be unsafe for non-idempotent operations.

The proposed capability focuses on proactive admission control rather than retry execution.

Proposed Direction

Preferred implementation

Introduce an optional Python per-tool admission-control policy using the existing FunctionMiddleware architecture.

The first contribution should preferably be a reusable middleware implementation or official example with tests, without requiring changes to the framework's public API.

Illustrative configuration

rate_limit_policy = ToolRateLimitPolicy(
    limits={
        "lookup_customer": RateLimit(
            requests=2,
            per_seconds=1.0,
            burst=2,
        ),
        "get_payment_status": RateLimit(
            requests=1,
            per_seconds=1.0,
            burst=1,
        ),
        "search_documents": RateLimit(
            requests=10,
            per_seconds=1.0,
            burst=10,
        ),
    },
    on_limit="reject",
)

This is an illustrative design, not an existing or finalized public SDK API.

Suggested technical approach

A token-bucket algorithm could provide predictable rate enforcement.

Each configured tool would maintain:

• Maximum token capacity
• Available tokens
• Token refill rate
• Last refill timestamp
• Appropriate asynchronous synchronization state

When a tool invocation begins:

1. Identify the tool and its configured policy.
2. Calculate available capacity using a monotonic clock.
3. Atomically update bucket state and attempt an admission reservation.
4. Admit the tool if capacity is available.
5. Otherwise reject the invocation before execution.
6. Preserve the existing function middleware pipeline behavior.
7. Return a clearly identifiable admission rejection without automatically retrying the tool.

Concurrency and correctness requirements

• Quota accounting must remain consistent under concurrent invocations.
• An admission decision must not be based on stale shared state.
• Independent tools must maintain independent quotas.
• The state-sharing scope must be explicitly documented.
• A cancellation must not corrupt bucket state.
• Already admitted operations must not be re-executed.
• Framework cancellation and failure semantics must be preserved.
• Tool identities must be unambiguous when tools originate from different integrations.
• Rate-limit configuration must reject invalid values.

Implementation boundaries

An in-process policy may share state across invocations and agent runs that use the same middleware instance.

It should not claim to enforce global quotas across separate processes or distributed workers.

Distributed rate limiting would require a separate shared-state or coordination mechanism and should remain outside the initial scope.

Proposed acceptance criteria

• Independent rate policies are configurable for different locally invoked tools.
• Calls within available capacity execute normally.
• Excess calls are rejected before the underlying tool executes.
• Concurrent calls cannot bypass quota accounting.
• Configurable burst capacity behaves as documented.
• Refill calculations use monotonic elapsed time.
• Unconfigured tools retain normal behavior.
• One rate-limited tool does not unnecessarily block unrelated tools.
• Cancellation propagates correctly.
• No implicit retries are introduced.
• Admission rejection is distinguishable from tool execution failure.
• Invalid policy configurations produce clear errors.
• The feature is opt-in and backward compatible.
• Deterministic local tests cover concurrency and refill behavior.
• Documentation explains state scope and distributed limitations.
• No new framework-level dependencies are required for the initial implementation.

Suggested deterministic tests

The tests should run without network access, external APIs, or paid cloud services.

Related issues and pull requests

Issue #1362 — Model-client rate limiting:

#1362

Issue #7914 — Per-tool concurrency controls:

#7914

Merged PR #4223 — Model-client auto-retry sample:

#4223

Merged PR #8453 — Sequential function invocation:

#8453

These address adjacent concerns but do not directly provide proactive per-tool request-rate admission control.

I am open to maintainer feedback regarding whether the initial contribution should be a sample-only implementation or a more reusable framework component.

Scope and Non-Goals

In Scope

• Python implementation
• Locally executed asynchronous tools
• Configurable per-tool request-rate quotas
• Pre-execution admission control
• Concurrency-safe accounting
• Configurable burst limits
• Fail-fast rejection
• Cancellation correctness
• Opt-in behavior
• Deterministic unit tests
• Documentation and examples

Out of Scope

• Model-provider HTTP 429 retry handling
• Distributed or cluster-wide rate limiting
• Redis-backed shared quota management
• Automatic retry and exponential backoff
• Provider-hosted tools outside the local invocation pipeline
• Modification of upstream API quotas
• Billing and token-cost accounting
• Whole-agent execution budgets
• Forced cancellation of already-running operations
• Guaranteed rollback of external side effects
• Breaking changes to existing APIs

These boundaries are intended to keep the initial contribution focused, testable, and maintainable.

Language

Python

Contribution Intent

I am willing to implement this

Acknowledgements

• I searched existing issues and did not find a duplicate.
• I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.