diff --git a/CHANGELOG.md b/CHANGELOG.md index 880ffdab7..82d89f954 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,7 +30,7 @@ ### Other changes - **[runtime] [docs]** `ultrafuzz run --run-id ` now fails with `RUN_ALREADY_EXISTS` before planning when the workflow engine already records a run with that ID, as it does after `ultrafuzz clean ` (#1258). Before, `run` rebuilt the plan and the execution snapshot, about 6 minutes and 1 GB for Damn Vulnerable DeFi, then failed at submission with `WORKFLOW_SUBMISSION_FAILED` / `DETACHED_ADMISSION_FAILED` / `RUN_EXISTS`, and left a partial run directory behind. The check asks the engine only once the project root has the engine's database, and any answer other than "this run exists" lets the launch go ahead as before. `clean` still leaves the engine record behind (#1257), so the ID stays taken. -- **[runtime] [docs]** `validate`, `plan` and `run` now report a `TOPOLOGY_TIMEOUT_SHADOWS_DEFAULT` warning for each topology `timeout_seconds` pin, on a node or in a group's defaults, that is below the model profile `timeout_seconds` or `run.default_timeout_seconds` it overrides (#675). A pin wins even when it is the shorter window, so raising a default silently did not reach a pinned node: the packaged `goals`, `strategies`, `specialists` and `review` groups pin 7,200 seconds, and goal nodes kept timing out at two hours after a longer default was configured. The warning names the pin, the default it overrides and the nodes it applies to. Before, only a CI test on the packaged topologies checked this, so a project's own topology got no check. +- **[runtime] [docs]** `validate`, `doctor` and `run` now report a `TOPOLOGY_TIMEOUT_SHADOWS_DEFAULT` warning for each topology `timeout_seconds` pin, on a node or in a group's defaults, that is below the model profile `timeout_seconds` or `run.default_timeout_seconds` it overrides (#675). A pin wins even when it is the shorter window, so raising a default silently did not reach a pinned node: the packaged `goals`, `strategies`, `specialists` and `review` groups pin 7,200 seconds, so a default above that warns for them. The warning names the pin, the default it overrides and the nodes it applies to, and, for a node pin inside a group that also pins, says that removing the node pin falls back to the group pin. Before, only a CI test on the packaged topologies checked this, so a project's own topology got no check. - **[runtime] [modal] [docs]** `resume --retry-failed` no longer reruns a failed task in a `failure_policy: continue` group, such as a property lens or an optional strategy, once a task that treats it as optional has started without it (#1231). The rerun could not reach the outputs that task had already produced, yet once it succeeded every planned task read as succeeded, so the final report said COMPLETE and a `--require-complete` run could end `succeeded`. The task now stays failed, the report stays PARTIAL, `resume` reports a `WORKFLOW_RETRY_SKIPPED` warning that names it and the tasks that ran without it, and other failed tasks are retried as before. The Modal benchmark worker no longer resumes a finished run whose only failures are such tasks, which it did over every failed task and then waited for a change that would not come. - **[config] [runtime] [prompts] [docs]** Adds `run.friction_log_enabled` (default `false`). When enabled, agent tasks are told to record Ultrafuzz, tooling, or instruction roadblocks with [Frog](https://github.com/wevm/frog) (`frog@1.1.0`, now a pinned dependency of `@ultrafuzz/runtime`) through a run-local `/friction-bin/ultrafuzz-friction-log` command, which runs the Frog of the Ultrafuzz install that rendered the workflow. The command accepts only `log` and `list` with a few local options, refuses the built-in flags of incur, Frog's CLI framework, such as `--mcp`, where an option value belongs, points Frog at `/friction` with `GIT_DIR` set to a path that does not exist, so entries land in `/friction/.agents/friction-log/`, unsets Frog's GitHub and Postgres store variables and incur's `COMPLETE`, points `GH_CONFIG_DIR` at a path that does not exist, sets `NO_UPDATE_NOTIFIER=1`, and gives Frog `/dev/null` as stdin. It guards against misuse by mistake and enforces nothing: agents run unsandboxed (see `docs/security.md`), so `/friction` is only where the command writes, and it removes no GitHub credential, because Frog falls back to `gh auth token`, which still finds a token kept in the system keyring. Every task preparation creates the directory and rewrites the command, best-effort: a friction log that cannot be prepared never fails the task. Frog is installed with the runtime even when the friction log is disabled, but runs leave it out of their trusted CLI closure and execution snapshot. Its `postgres` dependency moves the install paths of the Smithers packages that reach drizzle-orm (see the `resume` change above); the engine still runs with `SMITHERS_BACKEND=sqlite`. A disabled run renders byte-identical prompts. Entries are free-form agent text that the artifact secret scan does not cover, so check them for credentials, such as RPC URLs with API keys, before publishing anything. Read them as Markdown, or with the Frog pinned in the Ultrafuzz checkout and `GIT_DIR` set to a path that does not exist (see `docs/reference/configuration.md`), never a bare `npx frog`, and delete a malformed entry's directory, since Frog refuses every `log` and `list` while one is malformed (#172). - **[runtime] [docs]** A failed `ClaudeAgent` or `DeepSeekAgent` attempt now records the failure Claude Code states in its result, such as a contended OAuth refresh or a rejected DeepSeek API key, in the node's `last_error` and the attempt ledger's `failure_message`, for example `Claude run failed See https://smithers.sh/reference/errors (agent stated: Failed to refresh OAuth token: …)`. Before, the record was only the generic `Claude run failed` message, and the cause survived only in the Claude Code session transcript. `ultrafuzz inspect --json` shows it as `data.state.nodes..last_error`, the dashboard shows it as the node's latest error, and a public Modal eval worker's `public-eval-diagnostics.json` carries it as the failed node's `failure_message`; `ultrafuzz status` and `ultrafuzz why` do not show it, because they relay the workflow engine's own summaries, which carry the error message but not its details. The thrown error Smithers classifies is unchanged, so quota parking, the auth disable and the retry behaviour from #1171 are unaffected; the statement travels as `details.agentStatedFailure` and gets the same secret redaction and length cap as `failure_message`. `ultrafuzz run` refuses a project whose stock `.smithers/agents` files predate this release (`CONTROLLER_SOURCE_UNTRUSTED`), so existing projects must re-run `ultrafuzz init`, which keeps an existing `ultrafuzz.toml`, topology and prompts, before their next run. A run launched by an earlier release records the statement only after `ultrafuzz resume --refresh-controller`, which continues it with this release's workflow and adapters (#1084). diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index ee76d4739..007af5308 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -318,7 +318,7 @@ topology node or group default. The effective agent timeout uses this precedence: a topology node or group timeout, then the model profile timeout, then `[run].default_timeout_seconds`. A topology pin wins even when it is shorter, so raising the profile or run -default does not reach a pinned node. `validate`, `plan` and `run` report a +default does not reach a pinned node. `validate`, `doctor` and `run` report a `TOPOLOGY_TIMEOUT_SHADOWS_DEFAULT` warning for each node or group pin below the default it overrides. The packaged `goals`, `strategies`, `specialists` and `review` groups pin 7,200 seconds, so a `default_timeout_seconds` above that warns for diff --git a/packages/runtime/src/timeout-shadowing.ts b/packages/runtime/src/timeout-shadowing.ts index 6743442af..a672e1c24 100644 --- a/packages/runtime/src/timeout-shadowing.ts +++ b/packages/runtime/src/timeout-shadowing.ts @@ -8,10 +8,10 @@ import type { RuntimeDiagnostic } from "./types.js"; * * A node's timeout resolves to its own pin, then its group's pin, then its model profile's * `timeout_seconds`, then `run.default_timeout_seconds`. A pin therefore wins even when it is the - * shorter window, so raising the profile or run default silently does not reach a pinned node: the - * packaged `goals` and `strategies` pins kept high-reasoning goal nodes at 2h after #645 raised the - * profile default. Each pin is reported once, with the largest default it overrides and the - * agentic nodes it applies to. + * shorter window, so raising the profile or run default silently does not reach a pinned node, as + * when #645 raised the profile default and the packaged `goals` and `strategies` group pins kept + * those nodes at 7200 seconds. Each pin is reported once, with the largest default it overrides and + * the agentic nodes it applies to. */ export function timeoutShadowingDiagnostics( topology: ProjectTopology, @@ -21,23 +21,38 @@ export function timeoutShadowingDiagnostics( const topologyNodes = new Map(topology.nodes.map((node) => [node.id, node])); const pins = new Map< string, - { label: string; path: string; seconds: number; shadowed: { seconds: number; source: string }; nodes: Set } + { + label: string; + path: string; + fallback: string | undefined; + seconds: number; + shadowed: { seconds: number; source: string }; + nodes: Set; + } >(); for (const node of expanded.nodes) { const pinned = node.timeoutSeconds; const declared = topologyNodes.get(node.logicalId); if (node.kind !== "agentic" || pinned === undefined || declared === undefined) continue; + const groupPin = + declared.group === undefined ? undefined : topology.groups?.[declared.group]?.defaults?.timeout_seconds; const pin = declared.timeout_seconds === undefined && declared.group !== undefined ? { key: `group:${declared.group}`, label: `group \`${declared.group}\``, - path: `groups.${declared.group}.defaults.timeout_seconds` + path: `groups.${declared.group}.defaults.timeout_seconds`, + fallback: undefined } : { key: `node:${declared.id}`, label: `node \`${declared.id}\``, - path: `nodes.${declared.id}.timeout_seconds` + path: `nodes.${declared.id}.timeout_seconds`, + // Removing a node pin inside a group that pins falls back to the group's pin, not the default. + fallback: + declared.group === undefined || groupPin === undefined + ? undefined + : `\`groups.${declared.group}.defaults.timeout_seconds\`=${String(groupPin)}` }; // The default task compilation would apply without the pin: the profile's own timeout, else the // run default. Expansion folds the run default into each fan-out entry, so read the profile itself. @@ -59,9 +74,13 @@ export function timeoutShadowingDiagnostics( const nodes = [...pin.nodes].sort(); const named = nodes.slice(0, 3).join(", "); const applies = nodes.length > 3 ? `${named} and ${String(nodes.length - 3)} more nodes` : named; + const remedy = + pin.fallback === undefined + ? "Raise or remove the pin to use the longer default." + : `Raise the pin, or remove it to fall back to ${pin.fallback}.`; return { code: "TOPOLOGY_TIMEOUT_SHADOWS_DEFAULT", - message: `topology ${pin.label} pins timeout_seconds=${String(pin.seconds)}, below ${pin.shadowed.source}=${String(pin.shadowed.seconds)}; the pin wins, so ${applies} time out after ${String(pin.seconds)} seconds. Raise or remove the pin to use the longer default.`, + message: `topology ${pin.label} pins timeout_seconds=${String(pin.seconds)}, below ${pin.shadowed.source}=${String(pin.shadowed.seconds)}; the pin wins, so ${applies} time out after ${String(pin.seconds)} seconds. ${remedy}`, severity: "warning", source: "topology", path: pin.path diff --git a/packages/runtime/test/runtime.test.ts b/packages/runtime/test/runtime.test.ts index 788317adc..e51f21028 100644 --- a/packages/runtime/test/runtime.test.ts +++ b/packages/runtime/test/runtime.test.ts @@ -10196,6 +10196,15 @@ test("validate warns when a packaged group timeout pin is below the run default assert.equal(raised.value?.policy_posture.topology.status, "warn"); }); +test("validate stays silent when a timeout pin equals the default it overrides", async () => { + const project = tempProject(); + initProject({ projectRoot: project, force: true }); + setRunDefaultTimeout(project, 7_200); + + const equal = await validateProject({ projectRoot: project, env: {} }); + assert.deepEqual(timeoutShadowingWarnings(equal.diagnostics), []); +}); + // The default a pin overrides is the model profile's own `timeout_seconds` when it has one, which // task compilation prefers to the run default, so the warning names the profile's value. test("validate warns when a packaged group timeout pin is below the model profile timeout it overrides", async () => { @@ -10234,6 +10243,66 @@ test("validate warns when a packaged group timeout pin is below the model profil } }); +// A node fanned out to several model profiles is reported once, against the longest timeout of +// its profiles. +test("validate reports a fanned-out node timeout pin against its longest model profile timeout", async () => { + const project = tempProject(); + initProject({ projectRoot: project, force: true }); + const configPath = path.join(project, "ultrafuzz.toml"); + const config = fs.readFileSync(configPath, "utf8"); + assert.match(config, /^\[models\.claude\]$/mu); + assert.match(config, /^\[models\.deepseek\]$/mu); + fs.writeFileSync( + configPath, + config + .replace(/^\[models\.claude\]$/mu, "[models.claude]\ntimeout_seconds = 5400") + .replace(/^\[models\.deepseek\]$/mu, "[models.deepseek]\ntimeout_seconds = 10800"), + "utf8" + ); + fs.writeFileSync( + path.join(project, ".ultrafuzz", "topology.yml"), + `version: 2 +defaults: + strategy_loops: 1 +nodes: + - id: __start__ + kind: meta + role: start + depends_on: [] + - id: fanned + kind: agentic + prompt: setup/runtime-fixture.md + model_profiles: [claude, deepseek, default] + timeout_seconds: 1800 + depends_on: [__start__] + outputs: + - path: ${GENERIC_RUNTIME_MARKDOWN_PATH} + contract: ultrafuzz/nonempty-markdown@1 + primary: true + - id: __finish__ + kind: meta + role: finish + depends_on: [fanned] +`, + "utf8" + ); + writeNeutralRuntimeFixturePrompt(project); + + const result = await validateProject({ projectRoot: project, env: {} }); + const warnings = timeoutShadowingWarnings(result.diagnostics); + assert.deepEqual( + warnings.map((warning) => warning.path), + ["nodes.fanned.timeout_seconds"], + JSON.stringify(result.diagnostics) + ); + const [warning] = warnings; + assert.ok(warning); + assert.match( + warning.message, + /node `fanned` pins timeout_seconds=1800, below model profile `deepseek` `timeout_seconds`=10800; the pin wins, so fanned time out after 1800 seconds/u + ); +}); + test("plan warns about group and node timeout pins below the default they override", async () => { const project = tempProject(); initProject({ projectRoot: project, force: true }); @@ -10251,6 +10320,10 @@ groups: label: Pinned defaults: timeout_seconds: 600 + long: + label: Long + defaults: + timeout_seconds: 7200 nodes: - id: __start__ kind: meta @@ -10274,10 +10347,17 @@ nodes: timeout_seconds: 7200 depends_on: [own-pin] outputs:${markdownOutput} + - id: grouped-own-pin + kind: agentic + prompt: setup/runtime-fixture.md + group: long + timeout_seconds: 3000 + depends_on: [long-pin] + outputs:${markdownOutput} - id: __finish__ kind: meta role: finish - depends_on: [long-pin] + depends_on: [grouped-own-pin] `, "utf8" ); @@ -10289,16 +10369,28 @@ nodes: const warnings = timeoutShadowingWarnings(plan.diagnostics); assert.deepEqual( warnings.map((warning) => warning.path), - ["groups.pinned.defaults.timeout_seconds", "nodes.own-pin.timeout_seconds"], + [ + "groups.pinned.defaults.timeout_seconds", + "nodes.own-pin.timeout_seconds", + "nodes.grouped-own-pin.timeout_seconds" + ], JSON.stringify(plan.diagnostics) ); - const [groupWarning, nodeWarning] = warnings; - assert.ok(groupWarning && nodeWarning); + const [groupWarning, nodeWarning, groupedNodeWarning] = warnings; + assert.ok(groupWarning && nodeWarning && groupedNodeWarning); assert.match( groupWarning.message, - /group `pinned` pins timeout_seconds=600, below `run\.default_timeout_seconds`=3600; the pin wins, so grouped time out after 600 seconds/u + /group `pinned` pins timeout_seconds=600, below `run\.default_timeout_seconds`=3600; the pin wins, so grouped time out after 600 seconds\. Raise or remove the pin to use the longer default\.$/u + ); + assert.match( + nodeWarning.message, + /node `own-pin` pins timeout_seconds=300.* Raise or remove the pin to use the longer default\.$/u + ); + // Removing a node pin inside a group that pins falls back to the group's pin, not the default. + assert.match( + groupedNodeWarning.message, + /node `grouped-own-pin` pins timeout_seconds=3000, below `run\.default_timeout_seconds`=3600; .* Raise the pin, or remove it to fall back to `groups\.long\.defaults\.timeout_seconds`=7200\.$/u ); - assert.match(nodeWarning.message, /node `own-pin` pins timeout_seconds=300/u); }); test("plan creates run layout, graph fingerprint, and rendered prompt before Smithers submission", async () => {