Skip to content

Commit a43e3bb

Browse files
Detect disconnected MCP servers and automatically reconnect them (#1081)
* Reconnect dead MCP transports with backoff and single-dial retry * Treat reconnecting MCP servers as noise in runtime notices * Show reconnecting MCP servers in the TUI command surface * Keep retained tool count on reconnecting MCP rows * Retry failed MCP servers with a single dial * Add tier-2 reconnect wait to tool_search misses * Extend tool_search only for reconnecting servers that could match * Cover the tool_search reconnect extension and its needs-auth bound * Cover retained tool counts on reconnecting MCP rows * Cover the reconnecting MCP row label and retry * Document MCP reconnecting rows, fail-fast tools, and retry * Fix openAuthURL dep in the reconnecting surface test * Keep reconnecting MCP rows up across transient redial failures Transient redial outcomes stay on the reconnecting row with the next attempt instead of flashing failed; only auth and deterministic misconfiguration stop the loop. The transport close hook and the reconnect consumer land together so death detection actually fires. * fix(tui): drop non-null assertion in MCP reconnect surface test
1 parent ab3afcd commit a43e3bb

14 files changed

Lines changed: 1155 additions & 31 deletions

‎docs/MCP.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,33 @@ file for that HTTP endpoint. Built-in Exa is disable-only — Alt+R notifies and
124124
does not open the confirm. While a local `.corbits/settings.json` `mcpServers`
125125
list is in effect, these actions edit that local file; add remains blocked.
126126

127+
## Dropped Transports Reconnect in the Background
128+
129+
When a connected server's transport dies unexpectedly, the row goes
130+
**reconnecting** instead of failed: the redial attempt and the retained tool
131+
count stay on the row (for example, `acme — reconnecting · attempt 3 · 2
132+
tools`), and backoff redials behind the scenes with a jittered delay that
133+
grows per attempt and caps at 30 seconds. While reconnecting, the server's
134+
tools stay mounted but fail fast — calling one returns "MCP server is
135+
reconnecting; retry the call once it reports connected." rather than hanging
136+
or dispatching against a dead transport.
137+
138+
Pressing Enter on a failed or reconnecting row retries now with a single dial
139+
instead of waiting for backoff, reusing the saved settings row — it never
140+
adds a second server. A failed retry leaves the row failed with no backoff
141+
until the next manual retry. Redials that hit an authorization wall or a
142+
terminal failure stop on their own and drop the tools; only a manual retry
143+
brings those rows back.
144+
145+
This contrasts with deliberate teardown: **Alt+D** (disable) and **Alt+R**
146+
(remove) drop the server's tools at once and stop any redial loop, while a
147+
transport death keeps the tools mounted as fail-fast stubs and keeps
148+
redialing. A `tool_search` miss waits up to 1 second for in-flight handshakes
149+
and looks again; when a reconnecting server holds tools that could match the
150+
query, the search takes one extra 500 ms extension for the redial to remount
151+
them. Servers still waiting on authorization never earn that extension — they
152+
settle only when authorization completes out of band.
153+
127154
## Server Kinds
128155

129156
A server is reached one of two ways:

‎src/agent/tool-search.test.ts‎

Lines changed: 117 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { describe, test, expect } from "bun:test";
1+
import { describe, test, expect, jest } from "bun:test";
22
import type { ToolDefinition } from "@intx/types/runtime";
33
import {
44
createToolIndex,
@@ -9,6 +9,8 @@ import {
99
coreToolNamesForSessionMode,
1010
CORE_TOOL_NAMES,
1111
CATALOG_TOOL_NAMES,
12+
TOOL_SEARCH_PENDING_WAIT_MS,
13+
TOOL_SEARCH_RECONNECT_WAIT_MS,
1214
type ToolAvailability,
1315
} from "./tool-search.js";
1416

@@ -243,6 +245,15 @@ function call(
243245
return tool.handler(args, new AbortController().signal);
244246
}
245247

248+
async function flushMicrotasks(rounds = 100): Promise<void> {
249+
for (let i = 0; i < rounds; i++) await Promise.resolve();
250+
}
251+
252+
async function advanceAndFlush(ms: number): Promise<void> {
253+
jest.advanceTimersByTime(ms);
254+
await flushMicrotasks();
255+
}
256+
246257
describe("createToolSearchTool", () => {
247258
test("promotes matches and lists them back to the model", async () => {
248259
const promoted: string[] = [];
@@ -368,6 +379,111 @@ describe("createToolSearchTool", () => {
368379
/still connecting|still starting up|retry shortly/i,
369380
);
370381
});
382+
383+
test("a genuine miss with nothing pending never consults the reconnect predicate", async () => {
384+
let matchChecks = 0;
385+
const tool = createToolSearchTool({
386+
search: () => [],
387+
lookup: () => undefined,
388+
promote: () => undefined,
389+
awaitPendingConnections: async () => 0,
390+
hasReconnectingMatch: () => {
391+
matchChecks += 1;
392+
return true;
393+
},
394+
});
395+
const out = await call(tool, { query: "nonsense" });
396+
expect(out).toContain("different keywords");
397+
expect(matchChecks).toBe(0);
398+
});
399+
400+
test("a reconnecting match earns one short extension and finds the remounted tool", async () => {
401+
jest.useFakeTimers();
402+
try {
403+
const live: ToolDefinition[] = [];
404+
const timeouts: (number | undefined)[] = [];
405+
const matchQueries: string[] = [];
406+
const promoted: string[] = [];
407+
let searches = 0;
408+
const tool = createToolSearchTool({
409+
search: (query) => {
410+
searches += 1;
411+
return createToolIndex(() => live).search(query);
412+
},
413+
lookup: (name) => live.find((def) => def.name === name),
414+
promote: (names) => promoted.push(...names),
415+
awaitPendingConnections: async (timeoutMs?: number) => {
416+
timeouts.push(timeoutMs);
417+
await new Promise((resolve) => setTimeout(resolve, timeoutMs ?? 0));
418+
return live.length === 0 ? 1 : 0;
419+
},
420+
hasReconnectingMatch: (query) => {
421+
matchQueries.push(query);
422+
return true;
423+
},
424+
});
425+
const pending = call(tool, { query: "linear tracker" });
426+
await advanceAndFlush(TOOL_SEARCH_PENDING_WAIT_MS);
427+
// The redial remounts mid-extension: dropped stubs at tier 1, live set
428+
// back before the extension elapses.
429+
live.push({
430+
name: "mcp__linear__create_issue",
431+
description: "Create an issue in the tracker",
432+
inputSchema: { type: "object", properties: {}, required: [] },
433+
});
434+
await advanceAndFlush(TOOL_SEARCH_RECONNECT_WAIT_MS);
435+
const out = await pending;
436+
expect(out).toContain("mcp__linear__create_issue");
437+
expect(promoted).toContain("mcp__linear__create_issue");
438+
expect(timeouts).toEqual([
439+
TOOL_SEARCH_PENDING_WAIT_MS,
440+
TOOL_SEARCH_RECONNECT_WAIT_MS,
441+
]);
442+
expect(searches).toBe(3);
443+
expect(matchQueries).toEqual(["linear tracker"]);
444+
} finally {
445+
jest.useRealTimers();
446+
}
447+
});
448+
449+
test("a needs-auth miss never earns the extension and stays within the tier-1 bound", async () => {
450+
jest.useFakeTimers();
451+
try {
452+
const timeouts: (number | undefined)[] = [];
453+
let searches = 0;
454+
let matchChecks = 0;
455+
const tool = createToolSearchTool({
456+
search: () => {
457+
searches += 1;
458+
return [];
459+
},
460+
lookup: () => undefined,
461+
promote: () => undefined,
462+
awaitPendingConnections: async (timeoutMs?: number) => {
463+
timeouts.push(timeoutMs);
464+
await new Promise((resolve) => setTimeout(resolve, timeoutMs ?? 0));
465+
return 1;
466+
},
467+
// A needs-auth server never populates the reconnect map, so the
468+
// toolset predicate stays false for it.
469+
hasReconnectingMatch: () => {
470+
matchChecks += 1;
471+
return false;
472+
},
473+
});
474+
const pending = call(tool, { query: "notion" });
475+
await advanceAndFlush(TOOL_SEARCH_PENDING_WAIT_MS);
476+
// Far past any extension: no second wait may be outstanding.
477+
await advanceAndFlush(10_000);
478+
const out = await pending;
479+
expect(out).toMatch(/retry.*shortly/i);
480+
expect(timeouts).toEqual([TOOL_SEARCH_PENDING_WAIT_MS]);
481+
expect(searches).toBe(2);
482+
expect(matchChecks).toBe(1);
483+
} finally {
484+
jest.useRealTimers();
485+
}
486+
});
371487
});
372488

373489
describe("advertisedTools", () => {

‎src/agent/tool-search.ts‎

Lines changed: 37 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -285,13 +285,26 @@ export interface ToolSearchDeps {
285285
// below so even a stuck dependency can never hang the call. Omitted callers
286286
// (tests, ad-hoc indexes) have no pending handshakes to wait for.
287287
awaitPendingConnections?: (timeoutMs?: number) => Promise<number>;
288+
// True when a reconnecting MCP server holds retained tools that could match
289+
// `query`, justifying one short extra wait for the redial to remount them.
290+
// Only transport-death reconnects qualify — a needs-auth server settles
291+
// solely via out-of-band authorization, so extending the wait for one can
292+
// never help and this must stay false for them.
293+
hasReconnectingMatch?: (query: string) => boolean;
288294
}
289295

290296
// Brief bound a tool_search miss waits for in-flight MCP handshakes before
291297
// answering. A hung authorization must never hang the call, so both the
292298
// toolset wait and the handler race below are capped by this.
293299
export const TOOL_SEARCH_PENDING_WAIT_MS = 1_000;
294300

301+
// Short extension past the tier-1 wait, taken at most once and only when a
302+
// reconnecting server holds tools that could match the query. Covers the
303+
// redial window where the stubs are dropped and the live set is not yet
304+
// remounted. Never taken for needs-auth: those servers settle solely via
305+
// out-of-band authorization.
306+
export const TOOL_SEARCH_RECONNECT_WAIT_MS = 500;
307+
295308
const ToolSearchArgs = type({ query: "string" });
296309

297310
// Render one discovered tool as name, description, and pretty-printed input
@@ -312,21 +325,19 @@ function indent(text: string, pad: string): string {
312325
.join("\n");
313326
}
314327

315-
// Race the dependency's pending-count wait against the same bound, so a
328+
// Race the dependency's pending-count wait against a bound, so a
316329
// stuck dependency (hung OAuth that never settles) cannot hang the call.
317330
// Resolves undefined when this race itself times out.
318331
async function racePendingCount(
319332
awaitPending: (timeoutMs?: number) => Promise<number>,
333+
timeoutMs: number,
320334
): Promise<number | undefined> {
321335
let timer: ReturnType<typeof setTimeout> | undefined;
322336
try {
323337
return await Promise.race([
324-
awaitPending(TOOL_SEARCH_PENDING_WAIT_MS),
338+
awaitPending(timeoutMs),
325339
new Promise<undefined>((resolve) => {
326-
timer = setTimeout(
327-
() => resolve(undefined),
328-
TOOL_SEARCH_PENDING_WAIT_MS,
329-
);
340+
timer = setTimeout(() => resolve(undefined), timeoutMs);
330341
}),
331342
]);
332343
} finally {
@@ -347,13 +358,29 @@ export function createToolSearchTool(deps: ToolSearchDeps): AgentTool {
347358
return "Error: tool_search requires a non-empty query.";
348359
let names = deps.search(query);
349360
if (names.length === 0 && deps.awaitPendingConnections !== undefined) {
350-
// Miss while connectors start up: wait briefly, then re-search so
351-
// late-mounting tools land. The race bounds even a stuck dependency
352-
// (hung OAuth) — undefined means the wait itself timed out.
353-
const stillPending = await racePendingCount(
361+
// Tier 1 — miss while connectors start up: wait briefly, then
362+
// re-search so late-mounting tools land. The race bounds even a stuck
363+
// dependency (hung OAuth) — undefined means the wait itself timed out.
364+
let stillPending = await racePendingCount(
354365
deps.awaitPendingConnections,
366+
TOOL_SEARCH_PENDING_WAIT_MS,
355367
);
356368
names = deps.search(query);
369+
if (
370+
names.length === 0 &&
371+
(stillPending ?? 1) > 0 &&
372+
deps.hasReconnectingMatch?.(query) === true
373+
) {
374+
// Tier 2 — a reconnecting server holds tools that could match: one
375+
// short extension for the redial to remount them, then a final
376+
// re-search. Needs-auth never qualifies (see the dep contract), so
377+
// a hung authorization still answers within the tier-1 bound.
378+
stillPending = await racePendingCount(
379+
deps.awaitPendingConnections,
380+
TOOL_SEARCH_RECONNECT_WAIT_MS,
381+
);
382+
names = deps.search(query);
383+
}
357384
if (names.length === 0 && (stillPending ?? 1) > 0) {
358385
const detail =
359386
stillPending === undefined

0 commit comments

Comments
 (0)