perf(infinity-agent-core)!: cut idle-agent memory ~30% (154 → ~107 KB/agent) by shadaj · Pull Request #105 · hydro-project/infinity

shadaj · 2026-08-26T21:38:57Z

Optimizes per-agent resident memory for the agent_scale benchmark added by the parent commit, in three independent pieces:

Router owns driver futures directly (no `spawn_local` per driver)

route_loop now drives all thread drivers through one FuturesUnordered pool instead of spawning each as its own LocalSet task. When a driver goes idle, its future yields its thread ID and the router immediately drops both the future's memory and the worker entry (input/subscribe channels).
Previously a finished driver's JoinHandle, task allocation, and channel blocks were retained in the workers map until the thread's next message — ~13 KB per idle agent.
Panic isolation is preserved (each pooled future is still wrapped in rap_protocol::log_panic); shutdown wind-down now just drains the pool.

`InfinityMessage::SubscriptionEvent` payloads boxed

Boxed result and invocation in the rare SubscriptionEvent variant, shrinking size_of::<InfinityMessage>() from 352 → 184 bytes. Every stored history message paid for the fattest variant inline. Box is serde-transparent, so the wire/persisted format is unchanged.
Added InfinityMessage::tool_result() helper (the boxed field made cross-variant or-patterns impossible).
(Boxing only invocation was measured and rejected: it grows the enum to 200 bytes because SubscriptionEvent with an inline result becomes the largest variant again.)

Tool-call dedup derived from history instead of a durable index (BREAKING)

HistoryManager no longer maintains processed_tool_calls / pending_complete_tool_calls. Incoming tool results are deduplicated by walking the history tail just in time: scan back across trailing tool calls/results (future-proof for concurrent calls, e.g. tc tc tr tr), accept on a matching unanswered call, reject as duplicate on a matching result, and discard as stale on any other message (user text, assistant content, subscription events — all turn boundaries, since a subscription event can only be injected once pending calls are settled).
safe_spawn_point uses the same walk (tracking answered calls) instead of the set.
Durable message-ID dedup is now limited to inputs that are not naturally idempotent: user text and subscription events (a redelivered subscription event would mint a fresh injected invocation). Tool results and assistant/tool-call items no longer persist IDs.
BREAKING: StateStore::get_processed_ids now returns a single HashSet<String> and add_processed_tool_calls is removed — updated InMemoryStateStore, the daemon's PersistentStateStore, and the Lambda DynamoDbStateStore (old DynamoDB processed_tool_calls attributes are simply ignored). ThreadState keeps its processed_tool_call_ids field so old daemon snapshots still deserialize.

Benchmark & docs

agent_scale example now drains lifecycle notifications per wave (like a real embedding), so they aren't counted as per-agent memory.
Landing page: refreshed MemoryChart data from a post-optimization 50k-agent run (6.08 GB vs 7.90 GB before; 118.8 KB/agent at that point, ~107 KB/agent after the dedup changes) and updated the copy; updated the history-manager doc for the new dedup model.
Note: an extended 80k-agent benchmark run was in flight to measure the exact 8 GB crossing (~72k agents projected); MemoryChart data and copy may deserve a refresh from that run. Also left open: making the per-message dedup ID in ConversationStore::append_messages optional (user/subscription messages only), which needs a decision about the Lambda DSQL message_id NOT NULL column and daemon session-file compat.

All workspace tests pass (cargo test --workspace), clippy clean.

…sand agents in 8 GB Optimizes per-agent resident memory for the `agent_scale` benchmark added by the parent commit: 154 → 107.4 KB per idle agent (measured slope), with 80,000 agents × 20 turns fitting in 8.89 GB and the 8 GB line crossed at ~71,700 agents. Three independent optimizations: ## Router owns driver futures directly (no `spawn_local` per driver) * `route_loop` drives all thread drivers through one `FuturesUnordered` pool instead of spawning each as its own `LocalSet` task. When a driver goes idle, its future yields its thread ID and the router immediately frees the future and the worker entry (input/subscribe channels). * Previously a finished driver's `JoinHandle`, task allocation, and channel blocks were retained in the `workers` map until the thread's *next* message — ~13 KB per idle agent. * Panic isolation preserved (each pooled future is wrapped in `rap_protocol::log_panic`); shutdown wind-down drains the pool. ## `InfinityMessage::SubscriptionEvent` payloads boxed * Boxed `result` and `invocation` in the rare `SubscriptionEvent` variant, shrinking `size_of::<InfinityMessage>()` from 352 → 184 bytes; every stored history message previously paid for the fattest variant inline. `Box` is serde-transparent so the persisted format is unchanged. * Added `InfinityMessage::tool_result()` helper. (Boxing only `invocation` was measured and rejected: the enum grows to 200 bytes because `SubscriptionEvent` with an inline result becomes the largest variant.) ## Tool-call dedup derived from history instead of a durable index (BREAKING) * `HistoryManager` no longer maintains `processed_tool_calls` / `pending_complete_tool_calls`. Incoming tool results are deduplicated by walking the history tail just in time: scan back across trailing tool calls/results (future-proof for concurrent calls, e.g. `tc tc tr tr`), accept on a matching unanswered call, reject as duplicate on a matching result, discard as stale on any other message (user text, assistant content, subscription events — all turn boundaries, since a subscription event is only injected once pending calls are settled). * `safe_spawn_point` uses the same walk, tracking answered calls during the scan. * Durable message-ID dedup is limited to inputs that are not naturally idempotent: user text and subscription events (a redelivered subscription event would mint a fresh injected invocation). Tool results and assistant/tool-call items no longer persist IDs. * **BREAKING**: `StateStore::get_processed_ids` returns a single `HashSet<String>`; `add_processed_tool_calls` removed. Updated `InMemoryStateStore`, the daemon's `PersistentStateStore`, and the Lambda `DynamoDbStateStore` (old DynamoDB `processed_tool_calls` attributes are ignored). `ThreadState` keeps its `processed_tool_call_ids` field so old daemon snapshots still deserialize. ## Benchmark & docs * `agent_scale` drains lifecycle notifications per wave (like a real embedding) so they aren't counted as per-agent memory. * Landing page: `MemoryChart` regenerated from an 80,000-agent run (8.89 GB total, 108.5 KB/agent); hero, chapter title, and copy updated to "seventy thousand agents on a Raspberry Pi" (measured 8 GB crossing: ~71,700). * `history-manager.md` updated for the new dedup model. All workspace tests pass; clippy clean. Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #105

cloudflare-workers-and-pages · 2026-08-26T21:39:56Z

Deploying infinity with Cloudflare Pages

Latest commit:	`844f909`
Status:	✅ Deploy successful!
Preview URL:	https://bcbf8910.infinity-dc7.pages.dev
Branch Preview URL:	https://sandbox-c621cbab-f468-4f5c-a.infinity-dc7.pages.dev

View logs

MingweiSamuel

Nice

MingweiSamuel · 2026-08-26T23:20:20Z

+                // Only remove the exited driver's own entry: if the thread
+                // already respawned, the new entry's channel is still open.
+                if workers.get(&exited).is_some_and(|w| w.input_tx.is_closed()) {
+                    workers.remove(&exited);
+                }


ABA problem here, ideally would use a slotmap or something. but probably fine?

MingweiSamuel · 2026-08-26T23:20:37Z

@@ -249,11 +252,26 @@ async fn route_loop<C, S, H, O, F>(
 {
    let mut workers: HashMap<String, WorkerChannels<O::SubscribeRequest>> = HashMap::new();


What are the string keys here?

Thread IDs, we should probably newtype hmm.

MingweiSamuel · 2026-08-26T23:30:37Z

+                    if call.id == result_id {
+                        // Its result would have been seen before the call in
+                        // a backwards walk, so this call is unanswered.
+                        return ToolResultMatch::Unanswered;
+                    }


Nice, so the assumption is the tool call and (eventual) tool result will not ever be too far apart in history?

Yes because the only trailing tool calls / results will be for concurrent calls, of which there should not be that many.

MingweiSamuel · 2026-08-26T23:32:04Z

+    processed_ids: Arc<Mutex<HashMap<String, HashSet<String>>>>,
    metadata: Arc<Mutex<HashMap<String, serde_json::Value>>>,
    subscriptions: Arc<Mutex<HashMap<String, HashSet<String>>>>,
    pending_user_choices: Arc<Mutex<HashMap<String, Vec<UserChoice>>>>,


we gotta get rid of these bare strings

…sand agents in 8 GB Optimizes per-agent resident memory for the `agent_scale` benchmark added by the parent commit: 154 → 107.4 KB per idle agent (measured slope), with 80,000 agents × 20 turns fitting in 8.89 GB and the 8 GB line crossed at ~71,700 agents. Three independent optimizations: ## Router owns driver futures directly (no `spawn_local` per driver) * `route_loop` drives all thread drivers through one `FuturesUnordered` pool instead of spawning each as its own `LocalSet` task. When a driver goes idle, its future yields its thread ID and the router immediately frees the future and the worker entry (input/subscribe channels). * Previously a finished driver's `JoinHandle`, task allocation, and channel blocks were retained in the `workers` map until the thread's *next* message — ~13 KB per idle agent. * Panic isolation preserved (each pooled future is wrapped in `rap_protocol::log_panic`); shutdown wind-down drains the pool. ## `InfinityMessage::SubscriptionEvent` payloads boxed * Boxed `result` and `invocation` in the rare `SubscriptionEvent` variant, shrinking `size_of::<InfinityMessage>()` from 352 → 184 bytes; every stored history message previously paid for the fattest variant inline. `Box` is serde-transparent so the persisted format is unchanged. * Added `InfinityMessage::tool_result()` helper. (Boxing only `invocation` was measured and rejected: the enum grows to 200 bytes because `SubscriptionEvent` with an inline result becomes the largest variant.) ## Tool-call dedup derived from history instead of a durable index (BREAKING) * `HistoryManager` no longer maintains `processed_tool_calls` / `pending_complete_tool_calls`. Incoming tool results are deduplicated by walking the history tail just in time: scan back across trailing tool calls/results (future-proof for concurrent calls, e.g. `tc tc tr tr`), accept on a matching unanswered call, reject as duplicate on a matching result, discard as stale on any other message (user text, assistant content, subscription events — all turn boundaries, since a subscription event is only injected once pending calls are settled). * `safe_spawn_point` uses the same walk, tracking answered calls during the scan. * Durable message-ID dedup is limited to inputs that are not naturally idempotent: user text and subscription events (a redelivered subscription event would mint a fresh injected invocation). Tool results and assistant/tool-call items no longer persist IDs. * **BREAKING**: `StateStore::get_processed_ids` returns a single `HashSet<String>`; `add_processed_tool_calls` removed. Updated `InMemoryStateStore`, the daemon's `PersistentStateStore`, and the Lambda `DynamoDbStateStore` (old DynamoDB `processed_tool_calls` attributes are ignored). `ThreadState` drops its `processed_tool_call_ids` field entirely — serde ignores unknown fields by default, so snapshots from older versions still deserialize. ## Benchmark & docs * `agent_scale` drains lifecycle notifications per wave (like a real embedding) so they aren't counted as per-agent memory. * Landing page: `MemoryChart` regenerated from an 80,000-agent run (8.89 GB total, 108.5 KB/agent); hero, chapter title, and copy updated to "seventy thousand agents on a Raspberry Pi" (measured 8 GB crossing: ~71,700). * `history-manager.md` updated for the new dedup model. All workspace tests pass; clippy clean. BREAKING CHANGE: `StateStore::get_processed_ids` now returns `HashSet<String>` (message IDs only) instead of a two-set tuple. BREAKING CHANGE: `StateStore::add_processed_tool_calls` is removed; tool results are deduplicated against conversation history. BREAKING CHANGE: `ThreadState::processed_tool_call_ids` is removed (old serialized snapshots still deserialize; the field is ignored). Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #105

…sand agents in 8 GB Optimizes per-agent resident memory for the `agent_scale` benchmark added by the parent commit: 154 → 107.4 KB per idle agent (measured slope), with 80,000 agents × 20 turns fitting in 8.89 GB and the 8 GB line crossed at ~71,700 agents. Three independent optimizations: ## Router owns driver futures directly (no `spawn_local` per driver) * `route_loop` drives all thread drivers through one `FuturesUnordered` pool instead of spawning each as its own `LocalSet` task. When a driver goes idle, its future yields its thread ID and the router immediately frees the future and the worker entry (input/subscribe channels). * Previously a finished driver's `JoinHandle`, task allocation, and channel blocks were retained in the `workers` map until the thread's *next* message — ~13 KB per idle agent. * Panic isolation preserved (each pooled future is wrapped in `rap_protocol::log_panic`); shutdown wind-down drains the pool. ## `InfinityMessage::SubscriptionEvent` payloads boxed * Boxed `result` and `invocation` in the rare `SubscriptionEvent` variant, shrinking `size_of::<InfinityMessage>()` from 352 → 184 bytes; every stored history message previously paid for the fattest variant inline. `Box` is serde-transparent so the persisted format is unchanged. * Added `InfinityMessage::tool_result()` helper. (Boxing only `invocation` was measured and rejected: the enum grows to 200 bytes because `SubscriptionEvent` with an inline result becomes the largest variant.) ## Tool-call dedup derived from history instead of a durable index (BREAKING) * `HistoryManager` no longer maintains `processed_tool_calls` / `pending_complete_tool_calls`. Incoming tool results are deduplicated by walking the history tail just in time: scan back across trailing tool calls/results (future-proof for concurrent calls, e.g. `tc tc tr tr`), accept on a matching unanswered call, reject as duplicate on a matching result, discard as stale on any other message (user text, assistant content, subscription events — all turn boundaries, since a subscription event is only injected once pending calls are settled). * `safe_spawn_point` uses the same walk, tracking answered calls during the scan. * Durable message-ID dedup is limited to inputs that are not naturally idempotent: user text and subscription events (a redelivered subscription event would mint a fresh injected invocation). Tool results and assistant/tool-call items no longer persist IDs. * **BREAKING**: `StateStore::get_processed_ids` returns a single `HashSet<String>`; `add_processed_tool_calls` removed. Updated `InMemoryStateStore`, the daemon's `PersistentStateStore`, and the Lambda `DynamoDbStateStore` (old DynamoDB `processed_tool_calls` attributes are ignored). `ThreadState` drops its `processed_tool_call_ids` field entirely — serde ignores unknown fields by default, so snapshots from older versions still deserialize. ## Benchmark & docs * `agent_scale` drains lifecycle notifications per wave (like a real embedding) so they aren't counted as per-agent memory. * Landing page: `MemoryChart` regenerated from an 80,000-agent run (8.89 GB total, 108.5 KB/agent); hero, chapter title, and copy updated to "seventy thousand agents on a Raspberry Pi" (measured 8 GB crossing: ~71,700). * `history-manager.md` updated for the new dedup model. All workspace tests pass; clippy clean. BREAKING CHANGE: `StateStore::get_processed_ids` now returns `HashSet<String>` (message IDs only) instead of a two-set tuple. BREAKING CHANGE: `StateStore::add_processed_tool_calls` is removed; tool results are deduplicated against conversation history. BREAKING CHANGE: `ThreadState::processed_tool_call_ids` is removed (old serialized snapshots still deserialize; the field is ignored). Co-authored-by: Infinity 🤖 <infinity@hydro.run> PR: #105 Co-authored-by: Mingwei Samuel <mingwes@amazon.com> PR: #105

shadaj force-pushed the sandbox-c621cbab-f468-4f5c-ae56-a75a42f439df branch from 0c5f87c to 9fc0e58 Compare August 26, 2026 21:39

shadaj changed the base branch from main to sandbox-c6b6c253-5de3-4526-ba0f-e98fe288bc7e August 26, 2026 21:40

MingweiSamuel approved these changes Aug 26, 2026

View reviewed changes

shadaj force-pushed the sandbox-c621cbab-f468-4f5c-ae56-a75a42f439df branch from 9fc0e58 to f5ec9ea Compare August 27, 2026 00:00

shadaj force-pushed the sandbox-c621cbab-f468-4f5c-ae56-a75a42f439df branch from f5ec9ea to d5eaaa8 Compare August 27, 2026 00:01

shadaj force-pushed the sandbox-c6b6c253-5de3-4526-ba0f-e98fe288bc7e branch 5 times, most recently from d6c4d7c to fab57ef Compare August 27, 2026 18:04

Base automatically changed from sandbox-c6b6c253-5de3-4526-ba0f-e98fe288bc7e to main August 27, 2026 19:29

shadaj force-pushed the sandbox-c621cbab-f468-4f5c-ae56-a75a42f439df branch from d5eaaa8 to 581eec7 Compare August 27, 2026 19:41

shadaj marked this pull request as ready for review August 27, 2026 19:41

shadaj requested a review from a team August 27, 2026 19:41

shadaj force-pushed the sandbox-c621cbab-f468-4f5c-ae56-a75a42f439df branch from e36365b to e3776c8 Compare August 27, 2026 19:45

shadaj force-pushed the sandbox-c621cbab-f468-4f5c-ae56-a75a42f439df branch from e3776c8 to 844f909 Compare August 27, 2026 20:29

shadaj merged commit 9c07368 into main Aug 28, 2026
8 checks passed

shadaj deleted the sandbox-c621cbab-f468-4f5c-ae56-a75a42f439df branch August 28, 2026 01:14

Navigation Menu

Uh oh!

perf(infinity-agent-core)!: cut idle-agent memory ~30% (154 → ~107 KB/agent) - #105

perf(infinity-agent-core)!: cut idle-agent memory ~30% (154 → ~107 KB/agent)#105
shadaj merged 1 commit into
mainfrom
sandbox-c621cbab-f468-4f5c-ae56-a75a42f439df

shadaj commented Aug 26, 2026

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 26, 2026 •

edited

Loading

Uh oh!

MingweiSamuel left a comment

Uh oh!

Uh oh!

MingweiSamuel Aug 26, 2026

Uh oh!

MingweiSamuel Aug 26, 2026

Uh oh!

shadaj Aug 26, 2026

Uh oh!

MingweiSamuel Aug 26, 2026

Uh oh!

shadaj Aug 26, 2026

Uh oh!

MingweiSamuel Aug 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -249,11 +252,26 @@ async fn route_loop<C, S, H, O, F>(
		{
		let mut workers: HashMap<String, WorkerChannels<O::SubscribeRequest>> = HashMap::new();

Uh oh!

Conversation

shadaj commented Aug 26, 2026

Router owns driver futures directly (no spawn_local per driver)

InfinityMessage::SubscriptionEvent payloads boxed

Tool-call dedup derived from history instead of a durable index (BREAKING)

Benchmark & docs

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying infinity with Cloudflare Pages

Uh oh!

MingweiSamuel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MingweiSamuel Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

MingweiSamuel Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

shadaj Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

MingweiSamuel Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

shadaj Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

MingweiSamuel Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Router owns driver futures directly (no `spawn_local` per driver)

`InfinityMessage::SubscriptionEvent` payloads boxed

cloudflare-workers-and-pages Bot commented Aug 26, 2026 •

edited

Loading