Make agent tool configuration explicit

Author: rgarcia

README.md

@@ -12,30 +12,18 @@ cua "go to news.ycombinator.com and tell me the top 3 story titles"
 
 ## Why this exists
 
-Frontier models expose computer use through different tool protocols:
-
-- **OpenAI gpt-5.5**: a built-in `computer` tool that emits actions like
-  `{type:"click", x, y}`, `{type:"scroll", x, y, scroll_x, scroll_y}`,
-  `{type:"keypress", keys:[...]}`, …
-- **Anthropic claude-opus-5**: a built-in `computer_20251124` tool that
-  emits `{action:"left_click", coordinate:[x, y]}`,
-  `{action:"scroll", scroll_direction, scroll_amount}`, …
-- **Google gemini-2.5-pro / gemini-3.x**: a set of predefined
-  computer-use functions (`click_at`, `type_text_at`, `scroll_at`,
-  `navigate`, `go_back`, …) with 0-1000 normalized coordinates.
-- **Meta Muse Spark 1.1**: OpenAI-compatible Responses API function calls
-  paired with screenshot inputs and 0-1000 normalized coordinates.
-- **xAI Grok 4.5**: OpenAI-compatible Responses API function calls with
-  screenshot inputs, reasoning controls, and 0-1000 normalized coordinates.
-- **Yutori Navigator n1 / n1.5**: OpenAI-compatible `chat.completions`
-  responses with built-in browser action `tool_calls` like `left_click`,
-  `goto_url`, `type`, and `scroll` in 0-1000 normalized coordinates.
+Frontier models expose computer use through different protocols: native
+computer/browser declarations, predefined browser action sets, ordinary
+function tools, different coordinate systems, and different screenshot/result
+contracts. `@onkernel/cua-ai` represents those differences as an explicit,
+identity-keyed tool catalog. Callers choose the exact tools; provider transforms
+compose only the declarations and request fields required by those identities.
 
 All of them expect you to:
 
 1. Run a real browser somewhere (locally is annoying, on a server is hard).
 2. Translate every action into an actual SDK call against that browser.
-3. Capture a fresh screenshot after each action and feed it back to the model so it can verify what happened and plan the next step.
+3. Return policy-correct grounding: viewport images for browser writes, OS images for computer writes, requested data for reads, and no fresh image on failures.
 4. Keep doing this in a loop until the task is done.
 
 `cua` does all of this for you. The repo is structured as several focused npm packages so the per-provider plumbing is also reusable outside of this binary (e.g. by agents of your own spun up via [`kernel/cli`](https://github.com/kernel/cli) templates).
@@ -77,8 +65,8 @@ flowchart LR
 
 | Package                                 | What it ships                                                                                                                            |
 | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
-| [`@onkernel/cua-ai`](packages/ai)       | Computer-use model catalog (`getCuaModel`/`listCuaModels`), canonical CUA tool schemas, and provider adapters/runtime specs built on pi-ai. On npm. |
-| [`@onkernel/cua-agent`](packages/agent) | `CuaAgent`/`CuaAgentHarness` classes that execute cua-ai tool calls against a Kernel browser, screenshot loop included. On npm.          |
+| [`@onkernel/cua-ai`](packages/ai)       | Model catalog, explicit identity-keyed CUA tool factories/toolsets, compatibility checks, and composable provider transforms. On npm. |
+| [`@onkernel/cua-agent`](packages/agent) | `CuaAgent`/`CuaAgentHarness` composition wrappers with dynamic `setTools()` and shared Kernel-browser execution resources. On npm. |
 | [`@onkernel/cua-cli`](packages/cli)     | The `cua` binary: argv parsing, sessions, skills, JSONL output, pi-tui front-end.                                                        |
 
 ---
@@ -134,7 +122,7 @@ cua -p --model moonshotai:kimi-k3 "Same prompt"
 # Yutori Navigator
 cua -p --model n1.5-latest "Same prompt"
 
-# interactive TUI (default mode)
+# interactive TUI (default invocation)
 cua
 cua "summarize https://news.ycombinator.com"
 
@@ -156,23 +144,21 @@ cua -p -o jsonl "open example.com and tell me the heading"
 
 ## How it works
 
-1. **Model layer** — `@onkernel/cua-ai` owns the curated computer-use
-   model catalog (`getCuaModel`/`listCuaModels`), the canonical CUA
-   tool-call schemas, and per-provider adapters on top of `pi-ai` so
-   every model emits the same `CuaAction` vocabulary.
-2. **Execution layer** — `@onkernel/cua-agent` wraps `pi-agent-core`'s
-   `Agent`/`AgentHarness`. `CuaAgentHarness` runs the
-   prompt/screenshot/tool loop against a Kernel browser: it dispatches
-   canonical CUA tool calls into Kernel SDK `browsers.computer.*` calls
-   and captures a fresh screenshot back to the model on every turn.
+1. **Model layer** — `@onkernel/cua-ai` owns the curated model catalog,
+   stable tool identities, explicit tool factories/toolsets, compatibility
+   checks, and provider declarations/headers/payload transforms.
+2. **Execution layer** — `@onkernel/cua-agent` composes around
+   `pi-agent-core`'s `Agent`/`AgentHarness`. It materializes the caller's exact
+   catalog over one shared resource pool and executes canonical actions through
+   Kernel's computer API or a raw-CDP browser executor.
 3. **CLI** — `@onkernel/cua-cli` assembles a `CuaAgentHarness` from
    command-line flags, env-var-based API keys, a `JsonlSessionRepo` for
    transcripts, and pi skills; renders the result either as plain text
    (`--print`), JSONL events (`-o jsonl`), or an interactive pi-tui
    front-end.
 4. **Browser** — a fresh Kernel cloud browser session per run (or per
-   resume) with optional named profile load/save. Every screenshot the
-   model sees is a real PNG of a real browser tab.
+   resume) with optional named profile load/save. Grounding images come from
+   the live browser or VM using the selected tool's explicit result policy.
 
 See [`docs/architecture.md`](docs/architecture.md) for the full
 end-to-end flow.
@@ -241,7 +227,7 @@ cua session list           # tab-formatted: NAME, KERNEL_ID, AGE, LIVE_URL
 cua session show login     # full JSON: kernel_session_id, live_url, transcript_path, ...
 ```
 
-`-s <name>` works for ALL modes (action subcommands, `--print`, the
+`-s <name>` works for all invocation styles (action subcommands, `--print`, the
 interactive TUI). Liveness is checked before each attach: if the Kernel
 browser timed out, the call fails with a clear "session no longer
 alive" error suggesting `cua session stop <name> && cua session start

docs/agent-tool-configuration-spec.md

@@ -1,6 +1,7 @@
 # Agent Tool Configuration
 
-**Status:** Draft for later review  
+**Status:** Implemented
+
 **Scope:** `@onkernel/cua-agent` and the tool-building surface in `@onkernel/cua-ai`  
 **Compatibility:** Not a goal; these packages are alpha and may make breaking API changes.
 
@@ -17,7 +18,7 @@ The array may contain:
 
 Provider-recommended tools are intentionally distinct from CUA-authored tools. The former reproduce the basic tools or schemas a model provider recommends in its computer-use examples; the latter are additional capabilities designed and maintained by CUA. A caller should be able to combine either category with custom application tools while seeing exactly what the model receives.
 
-The current `extraTools`, `mode`, `nativeTool`, and `playwright` constructor options should be removed. `activeToolNames`, `setActiveTools()`, `setMode()`, `getMode()`, `computer_use_extra`, and CUA-generated default system prompts should also be removed. No global or derived mode should replace them.
+The former `extraTools`, `mode`, `nativeTool`, and `playwright` constructor options are removed. `activeToolNames`, `setActiveTools()`, `setMode()`, `getMode()`, `computer_use_extra`, and CUA-generated default system prompts are also removed. No global or derived mode replaces them.
 
 Each CUA tool specification must contain enough information to build, expose, execute, and describe that tool independently. Convenience toolsets may return arrays of tool specifications, but they must not establish hidden runtime state or add undeclared tools.
 
@@ -79,7 +80,7 @@ This terminology must also be used consistently in the architecture document, pa
 - Silently replacing incompatible tools when the model changes
 - Preserving CUA's current default system prompts
 
-## Proposed public namespace
+## Public namespace
 
 Tool factories and toolsets should be exported through one discoverable namespace rather than as a collection of global functions.
 
@@ -118,7 +119,7 @@ The distinction is deliberate:
 
 The exact property names may be refined, but the final exports must remain namespaced, autocomplete-friendly, and free of a large flat list of package-level tool factory functions.
 
-## Proposed constructor API
+## Constructor API
 
 Both constructors accept one required top-level `tools` array:
 
@@ -331,7 +332,7 @@ A tool specification has a stable identity and a preferred model-facing name. Th
 
 Composition sees the complete requested list and must detect name collisions before the first request. It must never silently shadow a tool.
 
-The naming policy is a design blocker that must be resolved before implementation. Candidate behavior is:
+The implemented naming policy is:
 
 1. Keep preferred provider-recommended names when unique.
 2. Reject collisions by default with an error naming both tool identities.
@@ -341,7 +342,7 @@ The naming policy is a design blocker that must be resolved before implementatio
 
 A toolset factory should not need hidden global state. The central composer sees all expanded tool specs and applies the collision policy. A toolset may expose explicit naming or namespace options, but automatic context-sensitive aliasing must not make the resulting catalog unpredictable.
 
-This policy must be prototyped with provider-recommended computer tools plus CUA browser tools before implementation begins.
+Catalog tests cover provider-recommended computer tools composed with CUA browser and caller tools.
 
 ## Tools and actions
 
@@ -365,7 +366,7 @@ Examples:
 
 An operation selected through a tool's arguments.
 
-Current or proposed action-bearing tools include:
+Current action-bearing tools include:
 
 - provider-native computer and browser tools, which use an `action` discriminator
 - `computer_batch`, which accepts an ordered `actions` array
@@ -392,15 +393,15 @@ A CUA or provider-recommended toolset may choose and document a default batch co
 
 ### Browser batch
 
-CUA should offer a browser-plane equivalent that does not dispatch OS computer-use input:
+CUA offers a browser-plane equivalent that does not dispatch OS computer-use input:
 
 ```ts
 cua.tools.browser.batch({
   actions: ["snapshot", "click", "fill", "wait_for", "text"],
 })
 ```
 
-The browser batch would execute browser/CDP operations and return their ordered read results. Its action schema, ref lifetime behavior, failure short-circuiting, and result grounding must be specified explicitly.
+The browser batch executes browser/CDP operations sequentially over one shared ref table and returns ordered read results. It short-circuits on the first failed or unsatisfied boundary, reports the failed index and skipped count, and follows the browser result-grounding policy.
 
 ### Browser batch versus browser act
 
@@ -409,7 +410,7 @@ The browser batch would execute browser/CDP operations and return their ordered
 - `browser_batch` is a mechanical ordered container for explicitly selected browser actions and read results.
 - `browser_act` is a dependent plan with per-step and final semantic expectations, causal outcomes, deadlines, stop reasons, and stable successor feedback.
 
-The overlap still needs a design review before implementation. In particular, the design must decide whether ref-producing reads can feed later actions inside one batch and whether a simpler batch should instead be a restricted form of the action-plan tool.
+The implemented batch is intentionally not a restricted action-plan tool. Ref-producing reads update the shared ref table before later actions, but the input has no interpolation, saved-value, branch, or workflow syntax. `browser_act` remains the semantic planning surface.
 
 ### Native action restrictions
 
@@ -522,11 +523,11 @@ A provider capability description may include:
 
 Coordinate uncertainty in one computer tool must not disable coordinate-free browser tools such as snapshots, refs, semantic waits, or action plans.
 
-Tzafon and Yutori require adapter changes before arbitrary composition is safe because their current payload hooks replace or suppress tools by name. That limitation should be reported against the affected requested tools, not represented as a blanket rejection of browser or mixed configurations.
+Tzafon and Yutori adapters compose by selected identity: Tzafon replaces only its native computer placeholder, while Yutori removes only selected native placeholders and preserves unrelated function tools.
 
 ## Removal of `computer_use_extra`
 
-`computer_use_extra` should be deleted entirely: definition, executor, implicit installation, exports, tests, and documentation.
+`computer_use_extra` is deleted entirely: definition, executor, implicit installation, exports, tests, and documentation.
 
 No replacement navigation helper is added automatically or under a new hidden name. A caller who needs navigation chooses an explicit capability, such as:
 
@@ -564,7 +565,7 @@ CUA must not silently drop tools, substitute a different toolset, append fallbac
 
 ## Removal of current API
 
-The following constructor options should be removed rather than deprecated:
+The following constructor options are removed rather than deprecated:
 
 ```ts
 extraTools
@@ -574,15 +575,15 @@ playwright
 activeToolNames
 ```
 
-The following methods should be removed from the CUA-facing API:
+The following methods are removed from the CUA-facing API:
 
 ```ts
 setMode()
 getMode()
 setActiveTools()
 ```
 
-`computer_use_extra` and CUA-generated default system prompts should be removed with them.
+`computer_use_extra` and CUA-generated default system prompts are removed with them.
 
 Their replacements are direct tool-list entries:
 
@@ -598,7 +599,7 @@ Their replacements are direct tool-list entries:
 
 ## Documentation requirements
 
-The future implementation must update:
+The implementation updates:
 
 - `docs/architecture.md` with the tool-spec composition and provider-adapter ownership boundaries
 - package READMEs with exact constructor examples and no legacy mode terminology
@@ -607,19 +608,15 @@ The future implementation must update:
 
 Provider-recommended toolsets must link to or name the provider guidance they mirror. CUA-authored additions must be described as CUA capabilities rather than provider defaults.
 
-## Design blockers before implementation
-
-No implementation should begin until these questions have concrete prototypes or decisions:
-
-1. **Name composition:** reject versus explicitly alias collisions, especially when provider-native names are fixed.
-2. **Payload transforms:** compose native and function tools without classifying them by ambiguous model-facing names.
-3. **Grounding ownership:** define post-action browser viewport versus OS display capture per tool, including mixed tool lists.
-4. **Batch overlap:** settle the relationship among `computer_batch`, `browser_batch`, and `browser_act`, including intra-batch ref flow.
-5. **Dynamic loading:** map `setTools()` onto pi's additive deferred-loading protocol while preserving CUA executors and session history.
-6. **Shared resources:** share translators, CDP state, and refs without deriving a mode or allowing one tool to mutate another's public contract.
-7. **Provider-recommended exports:** decide exactly which official or example tool shapes each provider namespace promises to mirror and how those promises are tested.
+## Implemented design resolutions
 
-These are architecture questions, not implementation details. The spec should be revisited after focused spikes for naming, Tzafon/Yutori payload composition, mixed grounding, and cache-preserving dynamic loading.
+1. **Name composition:** exact and provider-normalized collisions reject; caller aliases/namespaces are explicit; native names are fixed.
+2. **Payload transforms:** transforms consume stable identities, declare static write claims, and compose in a fixed phase order.
+3. **Grounding ownership:** each tool carries browser, computer, request-grounded, read, or failure behavior as data.
+4. **Batch overlap:** batches are mechanical; `browser_act` remains semantic; browser batches share ref state without a workflow DSL.
+5. **Dynamic loading:** `setTools()` uses pi 0.80.10 additive markers only for final, cache-preserving in-tool additions; other changes are eager.
+6. **Shared resources:** one resource pool survives tool/model changes and owns the translator and lazy CDP executor.
+7. **Provider-recommended exports:** namespaced Anthropic, Google, Meta, xAI, Moonshot, Tzafon, and Yutori surfaces are tested against their declared contracts.
 
 ## Decisions recorded
 
@@ -632,7 +629,7 @@ These are architecture questions, not implementation details. The spec should be
 - `browser_act` remains outside `cua.toolsets.browser()` until it has broader production evidence.
 - Naming, payload-transform composition, grounding, and batch overlap must be resolved before code is written.
 
-## Acceptance criteria for a future implementation
+## Acceptance criteria
 
 - Both constructors have one required tool-selection source of truth and accept `tools: []`.
 - The current tool-related constructor options, active-tool option, and mode methods are removed.