From 7069e1682eabb9deb33ba491dd7cf433c670bc25 Mon Sep 17 00:00:00 2001 From: thankyou <> Date: Thu, 1 Oct 2026 11:01:20 -0500 Subject: [PATCH 1/2] feat(runtime): warn when a topology timeout pin is below the default it overrides (#675) A node's timeout resolves to its own pin, then its group's pin, then its model profile's timeout_seconds, then run.default_timeout_seconds, so a pin wins even when it is the shorter window. Raising a default silently did not reach pinned nodes. validate, plan and run now report a TOPOLOGY_TIMEOUT_SHADOWS_DEFAULT warning for each such pin, naming the default it overrides and the nodes it applies to. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + docs/reference/configuration.md | 6 ++ packages/runtime/src/plan-run.ts | 5 +- packages/runtime/src/timeout-shadowing.ts | 70 ++++++++++++++ packages/runtime/src/validate.ts | 8 +- packages/runtime/test/runtime.test.ts | 108 ++++++++++++++++++++++ 6 files changed, 195 insertions(+), 3 deletions(-) create mode 100644 packages/runtime/src/timeout-shadowing.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index f7ee7d34a..ae6953acf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,7 @@ ### Other changes +- **[runtime] [docs]** `validate`, `plan` and `run` now report a `TOPOLOGY_TIMEOUT_SHADOWS_DEFAULT` warning for each topology `timeout_seconds` pin, on a node or in a group's defaults, that is below the model profile `timeout_seconds` or `run.default_timeout_seconds` it overrides (#675). A pin wins even when it is the shorter window, so raising a default silently did not reach a pinned node: the packaged `goals`, `strategies`, `specialists` and `review` groups pin 7,200 seconds, and goal nodes kept timing out at two hours after a longer default was configured. The warning names the pin, the default it overrides and the nodes it applies to. Before, only a CI test on the packaged topologies checked this, so a project's own topology got no check. - **[config] [runtime] [prompts] [docs]** Adds `run.friction_log_enabled` (default `false`). When enabled, agent tasks are told to record Ultrafuzz, tooling, or instruction roadblocks with [Frog](https://github.com/wevm/frog) (`frog@1.1.0`, now a pinned dependency of `@ultrafuzz/runtime`) through a run-local `/friction-bin/ultrafuzz-friction-log` command, which runs the Frog of the Ultrafuzz install that rendered the workflow. The command accepts only `log` and `list` with a few local options, refuses the built-in flags of incur, Frog's CLI framework, such as `--mcp`, where an option value belongs, points Frog at `/friction` with `GIT_DIR` set to a path that does not exist, so entries land in `/friction/.agents/friction-log/`, unsets Frog's GitHub and Postgres store variables and incur's `COMPLETE`, points `GH_CONFIG_DIR` at a path that does not exist, sets `NO_UPDATE_NOTIFIER=1`, and gives Frog `/dev/null` as stdin. It guards against misuse by mistake and enforces nothing: agents run unsandboxed (see `docs/security.md`), so `/friction` is only where the command writes, and it removes no GitHub credential, because Frog falls back to `gh auth token`, which still finds a token kept in the system keyring. Every task preparation creates the directory and rewrites the command, best-effort: a friction log that cannot be prepared never fails the task. Frog is installed with the runtime even when the friction log is disabled, but runs leave it out of their trusted CLI closure and execution snapshot. Its `postgres` dependency moves the install paths of the Smithers packages that reach drizzle-orm (see the `resume` change above); the engine still runs with `SMITHERS_BACKEND=sqlite`. A disabled run renders byte-identical prompts. Entries are free-form agent text that the artifact secret scan does not cover, so check them for credentials, such as RPC URLs with API keys, before publishing anything. Read them as Markdown, or with the Frog pinned in the Ultrafuzz checkout and `GIT_DIR` set to a path that does not exist (see `docs/reference/configuration.md`), never a bare `npx frog`, and delete a malformed entry's directory, since Frog refuses every `log` and `list` while one is malformed (#172). - **[runtime] [docs]** A failed `ClaudeAgent` or `DeepSeekAgent` attempt now records the failure Claude Code states in its result, such as a contended OAuth refresh or a rejected DeepSeek API key, in the node's `last_error` and the attempt ledger's `failure_message`, for example `Claude run failed See https://smithers.sh/reference/errors (agent stated: Failed to refresh OAuth token: …)`. Before, the record was only the generic `Claude run failed` message, and the cause survived only in the Claude Code session transcript. `ultrafuzz inspect --json` shows it as `data.state.nodes..last_error`, the dashboard shows it as the node's latest error, and a public Modal eval worker's `public-eval-diagnostics.json` carries it as the failed node's `failure_message`; `ultrafuzz status` and `ultrafuzz why` do not show it, because they relay the workflow engine's own summaries, which carry the error message but not its details. The thrown error Smithers classifies is unchanged, so quota parking, the auth disable and the retry behaviour from #1171 are unaffected; the statement travels as `details.agentStatedFailure` and gets the same secret redaction and length cap as `failure_message`. `ultrafuzz run` refuses a project whose stock `.smithers/agents` files predate this release (`CONTROLLER_SOURCE_UNTRUSTED`), so existing projects must re-run `ultrafuzz init`, which keeps an existing `ultrafuzz.toml`, topology and prompts, before their next run. A run launched by an earlier release records the statement only after `ultrafuzz resume --refresh-controller`, which continues it with this release's workflow and adapters (#1084). - **[modal]** Moves `@grpc/grpc-js` to 1.14.5, past the High advisory GHSA-m9gg-hp2v-232j (`getAuthContext` could return unauthorized certificates as authorized in certain configurations). diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 17c898152..ee76d4739 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -317,6 +317,12 @@ topology node or group default. The effective agent timeout uses this precedence: a topology node or group timeout, then the model profile timeout, then `[run].default_timeout_seconds`. +A topology pin wins even when it is shorter, so raising the profile or run +default does not reach a pinned node. `validate`, `plan` and `run` report a +`TOPOLOGY_TIMEOUT_SHADOWS_DEFAULT` warning for each node or group pin below the +default it overrides. The packaged `goals`, `strategies`, `specialists` and `review` +groups pin 7,200 seconds, so a `default_timeout_seconds` above that warns for +them. Generated defaults may include `[models] synthesized_default = true` when the default profile was synthesized by the scaffold. diff --git a/packages/runtime/src/plan-run.ts b/packages/runtime/src/plan-run.ts index 930bdb281..0a734d9c4 100644 --- a/packages/runtime/src/plan-run.ts +++ b/packages/runtime/src/plan-run.ts @@ -86,6 +86,7 @@ import { } from "./data-governance.js"; import { forgeGuardMetadata } from "./forge-guard.js"; import { assertExpandedGraphRetryChains } from "./retry-chain.js"; +import { timeoutShadowingDiagnostics } from "./timeout-shadowing.js"; import { projectArtifactSchemaDir } from "./init.js"; import { assertLaunchCheckoutRevision, @@ -194,6 +195,7 @@ export async function planRun(input: PlanRunInput, hooks: PlanRunHooks = {}) { let expandedGraph: ExpandedGraph; let catalog: PromptCatalog; + let timeoutDiagnostics: RuntimeDiagnostic[]; try { const topology = transformTopologyForRun( loadTopology(projectRoot, { @@ -215,6 +217,7 @@ export async function planRun(input: PlanRunInput, hooks: PlanRunHooks = {}) { configFingerprint: redactedConfigFingerprint }); assertExpandedGraphRetryChains(resolved.config, expandedGraph); + timeoutDiagnostics = timeoutShadowingDiagnostics(topology, expandedGraph, resolved.config); } catch (error) { return runtimeFailure([diagnosticFromError(error, "runtime", "RUN_PLAN_INVALID")]); } @@ -568,7 +571,7 @@ export async function planRun(input: PlanRunInput, hooks: PlanRunHooks = {}) { } }) }, - preMaterializeDiagnostics + [...timeoutDiagnostics, ...preMaterializeDiagnostics] ); } diff --git a/packages/runtime/src/timeout-shadowing.ts b/packages/runtime/src/timeout-shadowing.ts new file mode 100644 index 000000000..6743442af --- /dev/null +++ b/packages/runtime/src/timeout-shadowing.ts @@ -0,0 +1,70 @@ +import type { ResolvedConfig } from "@ultrafuzz/config"; +import type { ExpandedGraph, ProjectTopology } from "@ultrafuzz/topology"; + +import type { RuntimeDiagnostic } from "./types.js"; + +/** + * Warn about each topology `timeout_seconds` pin below the default it overrides (#675). + * + * A node's timeout resolves to its own pin, then its group's pin, then its model profile's + * `timeout_seconds`, then `run.default_timeout_seconds`. A pin therefore wins even when it is the + * shorter window, so raising the profile or run default silently does not reach a pinned node: the + * packaged `goals` and `strategies` pins kept high-reasoning goal nodes at 2h after #645 raised the + * profile default. Each pin is reported once, with the largest default it overrides and the + * agentic nodes it applies to. + */ +export function timeoutShadowingDiagnostics( + topology: ProjectTopology, + expanded: ExpandedGraph, + config: ResolvedConfig +): RuntimeDiagnostic[] { + const topologyNodes = new Map(topology.nodes.map((node) => [node.id, node])); + const pins = new Map< + string, + { label: string; path: string; seconds: number; shadowed: { seconds: number; source: string }; nodes: Set } + >(); + for (const node of expanded.nodes) { + const pinned = node.timeoutSeconds; + const declared = topologyNodes.get(node.logicalId); + if (node.kind !== "agentic" || pinned === undefined || declared === undefined) continue; + const pin = + declared.timeout_seconds === undefined && declared.group !== undefined + ? { + key: `group:${declared.group}`, + label: `group \`${declared.group}\``, + path: `groups.${declared.group}.defaults.timeout_seconds` + } + : { + key: `node:${declared.id}`, + label: `node \`${declared.id}\``, + path: `nodes.${declared.id}.timeout_seconds` + }; + // The default task compilation would apply without the pin: the profile's own timeout, else the + // run default. Expansion folds the run default into each fan-out entry, so read the profile itself. + const profileIds = node.modelFanout.length === 0 ? [undefined] : node.modelFanout.map((m) => m.modelProfileId); + for (const profileId of profileIds) { + const profileTimeout = profileId === undefined ? undefined : config.models.profiles[profileId]?.timeoutSeconds; + const shadowed = + profileId === undefined || profileTimeout === undefined + ? { seconds: config.run.defaultTimeoutSeconds, source: "`run.default_timeout_seconds`" } + : { seconds: profileTimeout, source: `model profile \`${profileId}\` \`timeout_seconds\`` }; + if (shadowed.seconds <= pinned) continue; + const entry = pins.get(pin.key) ?? { ...pin, seconds: pinned, shadowed, nodes: new Set() }; + if (shadowed.seconds > entry.shadowed.seconds) entry.shadowed = shadowed; + entry.nodes.add(declared.id); + pins.set(pin.key, entry); + } + } + return [...pins.values()].map((pin) => { + const nodes = [...pin.nodes].sort(); + const named = nodes.slice(0, 3).join(", "); + const applies = nodes.length > 3 ? `${named} and ${String(nodes.length - 3)} more nodes` : named; + return { + code: "TOPOLOGY_TIMEOUT_SHADOWS_DEFAULT", + message: `topology ${pin.label} pins timeout_seconds=${String(pin.seconds)}, below ${pin.shadowed.source}=${String(pin.shadowed.seconds)}; the pin wins, so ${applies} time out after ${String(pin.seconds)} seconds. Raise or remove the pin to use the longer default.`, + severity: "warning", + source: "topology", + path: pin.path + }; + }); +} diff --git a/packages/runtime/src/validate.ts b/packages/runtime/src/validate.ts index fbc9f5af9..c10fa18ad 100644 --- a/packages/runtime/src/validate.ts +++ b/packages/runtime/src/validate.ts @@ -23,6 +23,7 @@ import type { ValidateProjectResult } from "./types.js"; import { effectiveAuditPolicy } from "./audit-profile-policy.js"; +import { timeoutShadowingDiagnostics } from "./timeout-shadowing.js"; import { agentRegistryRegisters, inspectAgentRegistry } from "./agent-registry.js"; import { promptTextsForCatalog, transformPromptCatalogForRun, transformTopologyForRun } from "./topology-transform.js"; import { @@ -304,6 +305,7 @@ function validateTopologySurface( modelProfiles: config ? modelProfilesForTopology(config) : undefined, defaultModelProfileId: config?.retry.agents[0] ?? config?.models.default }); + const timeoutDiagnostics = config === undefined ? [] : timeoutShadowingDiagnostics(topology, expanded, config); const selectedAgents = new Set(expanded.nodes.flatMap((node) => node.modelFanout.map((model) => model.agentRef))); for (const model of expanded.nodes.flatMap((node) => node.modelFanout)) { if (config === undefined) continue; @@ -316,8 +318,10 @@ function validateTopologySurface( return { posture: postureFromDiagnostics( "topology", - "YAML topology v1 loads, validates, and expands", - executionDiagnostics + timeoutDiagnostics.length === 0 + ? "YAML topology v1 loads, validates, and expands" + : "YAML topology v1 loads, validates, and expands, but a timeout_seconds pin is below the default it overrides", + [...executionDiagnostics, ...timeoutDiagnostics] ), selectedAgentRefs: [...selectedAgents].sort(), summary: { diff --git a/packages/runtime/test/runtime.test.ts b/packages/runtime/test/runtime.test.ts index f380f27d7..ca6cdffd0 100644 --- a/packages/runtime/test/runtime.test.ts +++ b/packages/runtime/test/runtime.test.ts @@ -10154,6 +10154,114 @@ test("force init rejects symlinked config and nested project files before overwr assert.equal(fs.readFileSync(path.join(topologyOutside, "topology.yml"), "utf8"), "outside\n"); }); +// #675: a topology `timeout_seconds` pin outranks the profile and run defaults even when it is the +// shorter window, so raising a default silently does not reach a pinned node. +const timeoutShadowingWarnings = (diagnostics: readonly T[]): T[] => + diagnostics.filter((diagnostic) => diagnostic.code === "TOPOLOGY_TIMEOUT_SHADOWS_DEFAULT"); + +function setRunDefaultTimeout(project: string, seconds: number): void { + const configPath = path.join(project, "ultrafuzz.toml"); + const config = fs.readFileSync(configPath, "utf8"); + assert.match(config, /^\[run\]$/mu); + assert.doesNotMatch(config, /^default_timeout_seconds/mu); + fs.writeFileSync( + configPath, + config.replace(/^\[run\]$/mu, `[run]\ndefault_timeout_seconds = ${String(seconds)}`), + "utf8" + ); +} + +test("validate warns when a packaged group timeout pin is below the run default it overrides", async () => { + const project = tempProject(); + initProject({ projectRoot: project, force: true }); + + const shipped = await validateProject({ projectRoot: project, env: {} }); + assert.deepEqual(timeoutShadowingWarnings(shipped.diagnostics), []); + + setRunDefaultTimeout(project, 14_400); + const raised = await validateProject({ projectRoot: project, env: {} }); + const warnings = timeoutShadowingWarnings(raised.diagnostics); + assert.deepEqual(warnings.map((warning) => warning.path).sort(), [ + "groups.goals.defaults.timeout_seconds", + "groups.review.defaults.timeout_seconds", + "groups.specialists.defaults.timeout_seconds", + "groups.strategies.defaults.timeout_seconds" + ]); + for (const warning of warnings) { + assert.equal(warning.severity, "warning"); + assert.equal(warning.source, "topology"); + assert.match(warning.message, /pins timeout_seconds=7200, below `run\.default_timeout_seconds`=14400/u); + } + assert.equal(raised.value?.policy_posture.topology.status, "warn"); +}); + +test("plan warns about group and node timeout pins below the default they override", async () => { + const project = tempProject(); + initProject({ projectRoot: project, force: true }); + const markdownOutput = ` + - path: ${GENERIC_RUNTIME_MARKDOWN_PATH} + contract: ultrafuzz/nonempty-markdown@1 + primary: true`; + fs.writeFileSync( + path.join(project, ".ultrafuzz", "topology.yml"), + `version: 2 +defaults: + strategy_loops: 1 +groups: + pinned: + label: Pinned + defaults: + timeout_seconds: 600 +nodes: + - id: __start__ + kind: meta + role: start + depends_on: [] + - id: grouped + kind: agentic + prompt: setup/runtime-fixture.md + group: pinned + depends_on: [__start__] + outputs:${markdownOutput} + - id: own-pin + kind: agentic + prompt: setup/runtime-fixture.md + timeout_seconds: 300 + depends_on: [grouped] + outputs:${markdownOutput} + - id: long-pin + kind: agentic + prompt: setup/runtime-fixture.md + timeout_seconds: 7200 + depends_on: [own-pin] + outputs:${markdownOutput} + - id: __finish__ + kind: meta + role: finish + depends_on: [long-pin] +`, + "utf8" + ); + writeNeutralRuntimeFixturePrompt(project); + + const plan = await planRun({ projectRoot: project, runId: "timeout-shadowing", env: {} }); + + assert.equal(plan.ok, true, JSON.stringify(plan.diagnostics)); + const warnings = timeoutShadowingWarnings(plan.diagnostics); + assert.deepEqual( + warnings.map((warning) => warning.path), + ["groups.pinned.defaults.timeout_seconds", "nodes.own-pin.timeout_seconds"], + JSON.stringify(plan.diagnostics) + ); + const [groupWarning, nodeWarning] = warnings; + assert.ok(groupWarning && nodeWarning); + assert.match( + groupWarning.message, + /group `pinned` pins timeout_seconds=600, below `run\.default_timeout_seconds`=3600; the pin wins, so grouped time out after 600 seconds/u + ); + assert.match(nodeWarning.message, /node `own-pin` pins timeout_seconds=300/u); +}); + test("plan creates run layout, graph fingerprint, and rendered prompt before Smithers submission", async () => { const project = tempProject(); initProject({ projectRoot: project, force: true }); From aa4a5b0fe474f4b253ff0ddb7fceef3cd148b39c Mon Sep 17 00:00:00 2001 From: thankyou <> Date: Thu, 1 Oct 2026 12:03:11 -0500 Subject: [PATCH 2/2] test(runtime): cover a timeout pin below the model profile timeout it overrides (#675) The profile's own timeout_seconds is the default a pin overrides when it has one, and task compilation prefers it to run.default_timeout_seconds, so the warning must name the profile's value even when the run default is longer. Co-Authored-By: Claude Opus 5.5 --- packages/runtime/test/runtime.test.ts | 38 +++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/packages/runtime/test/runtime.test.ts b/packages/runtime/test/runtime.test.ts index ca6cdffd0..fc164ebd0 100644 --- a/packages/runtime/test/runtime.test.ts +++ b/packages/runtime/test/runtime.test.ts @@ -10195,6 +10195,44 @@ test("validate warns when a packaged group timeout pin is below the run default assert.equal(raised.value?.policy_posture.topology.status, "warn"); }); +// The default a pin overrides is the model profile's own `timeout_seconds` when it has one, which +// task compilation prefers to the run default, so the warning names the profile's value. +test("validate warns when a packaged group timeout pin is below the model profile timeout it overrides", async () => { + const project = tempProject(); + initProject({ projectRoot: project, force: true }); + const configPath = path.join(project, "ultrafuzz.toml"); + const config = fs.readFileSync(configPath, "utf8"); + assert.match(config, /^\[models\.default\]$/mu); + fs.writeFileSync( + configPath, + config.replace(/^\[models\.default\]$/mu, "[models.default]\ntimeout_seconds = 10800"), + "utf8" + ); + + const raised = await validateProject({ projectRoot: project, env: {} }); + const warnings = timeoutShadowingWarnings(raised.diagnostics); + assert.deepEqual(warnings.map((warning) => warning.path).sort(), [ + "groups.goals.defaults.timeout_seconds", + "groups.review.defaults.timeout_seconds", + "groups.specialists.defaults.timeout_seconds", + "groups.strategies.defaults.timeout_seconds" + ]); + for (const warning of warnings) { + assert.match(warning.message, /pins timeout_seconds=7200, below model profile `default` `timeout_seconds`=10800/u); + } + + // A longer run default does not apply to a profile with its own timeout, so the warning keeps + // naming the profile. + setRunDefaultTimeout(project, 14_400); + const both = await validateProject({ projectRoot: project, env: {} }); + const bothWarnings = timeoutShadowingWarnings(both.diagnostics); + assert.equal(bothWarnings.length, 4, JSON.stringify(both.diagnostics)); + for (const warning of bothWarnings) { + assert.match(warning.message, /below model profile `default` `timeout_seconds`=10800/u); + assert.doesNotMatch(warning.message, /14400/u); + } +}); + test("plan warns about group and node timeout pins below the default they override", async () => { const project = tempProject(); initProject({ projectRoot: project, force: true });