diff --git a/README.md b/README.md index b8ae3da0..909f4274 100644 --- a/README.md +++ b/README.md @@ -65,6 +65,7 @@ Driving an interactive TUI is the same loop, with key chords and a wait for the agent-tty run "$SID" 'nvim --clean' --no-wait --json agent-tty wait "$SID" --screen-stable-ms 1000 --json agent-tty send-keys "$SID" Down Down Enter --json +agent-tty mouse "$SID" press --button left --row 8 --col 20 --json agent-tty screenshot "$SID" --json agent-tty record export "$SID" --format webm --json ``` @@ -115,13 +116,13 @@ A colleague then used `agent-tty` to build an experimental TUI for Coder agents Every user-facing command takes `--json` and returns a stable, machine-readable envelope, and exits with a stable code (`0` success, `2` usage error, `3` session not found, `11` wait timeout, …) so scripts can branch without parsing output. -| Group | Commands | -| ----------------------- | ------------------------------------------------------------------------ | -| Session lifecycle | `create`, `list`, `inspect`, `destroy`, `gc` | -| Input and control | `run`, `type`, `paste`, `send-keys`, `batch`, `resize`, `signal`, `mark` | -| Observation and capture | `wait`, `snapshot`, `screenshot`, `record export` | -| Live view | `dashboard` | -| Environment | `version`, `doctor`, `skills` | +| Group | Commands | +| ----------------------- | --------------------------------------------------------------------------------- | +| Session lifecycle | `create`, `list`, `inspect`, `destroy`, `gc` | +| Input and control | `run`, `type`, `paste`, `send-keys`, `mouse`, `batch`, `resize`, `signal`, `mark` | +| Observation and capture | `wait`, `snapshot`, `screenshot`, `record export` | +| Live view | `dashboard` | +| Environment | `version`, `doctor`, `skills` | The CLI documents itself: `agent-tty --help` lists every command, and `agent-tty --help` shows its flags. The full reference, including the exit-code table, is in [`docs/USAGE.md`](./docs/USAGE.md); renderer and environment issues are in [`docs/TROUBLESHOOTING.md`](./docs/TROUBLESHOOTING.md). diff --git a/docs/USAGE.md b/docs/USAGE.md index 58e7a7b8..a2b9a772 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -50,6 +50,7 @@ agent-tty --home run 'command here' --json agent-tty --home type 'literal text' --json agent-tty --home paste 'multiline payload' --json agent-tty --home send-keys Enter Ctrl+C --json +agent-tty --home mouse press --button left --row 8 --col 20 --json agent-tty --home resize --cols 100 --rows 30 --json agent-tty --home signal SIGTERM --json @@ -83,6 +84,42 @@ Important flags: Use `type` when the target application needs literal interactive typing, `paste` when the target should receive a literal pasted payload, and `send-keys` for discrete control keys such as `Enter`, `Escape`, or `Ctrl+C`. `run` is not structured output capture and does not report the child command's exit status. +## `mouse` + +`mouse` sends a cell-addressed event through the terminal modes selected by +the child. The `libghostty-vt` renderer is required because static escape +sequences cannot account for tracking mode, wire format, pixel mode, viewport +clamping, or motion deduplication. + +```bash +agent-tty mouse press --button left --row 8 --col 20 --json +agent-tty mouse move --row 9 --col 24 --json +agent-tty mouse release --button left --row 9 --col 24 --json +agent-tty mouse press --button wheel-down --row 9 --col 24 --json +``` + +Rows and columns are zero-based. `press` and `release` require `--button`; +`move` uses the host's held-button state so press, move, and release form a real +drag. Buttons are `left`, `middle`, `right`, `wheel-up`, `wheel-down`, +`wheel-left`, and `wheel-right`. `--shift`, `--alt`, and `--ctrl` add modifiers. +An event suppressed by the child's tracking mode succeeds with +`reported: false` and `bytesWritten: 0`. Missing renderer support exits `12` +with `CAPABILITY_UNAVAILABLE`. + +The default surface uses virtual one-pixel cells. For SGR-pixels with a known +surface, pass `--cell-width` and `--cell-height` in pixels; the requested cell +maps to its top-left pixel. For example, row 2, column 3 with a 10x20 cell maps +to pixel 30,40. Screen dimensions follow the session's current rows and columns. +The same optional `cellWidth` and `cellHeight` fields are accepted in batch steps. + +The host writes the native buffer directly to the PTY, including non-UTF-8 X10 +bytes. Each `input_mouse` event stores those exact bytes as `dataBase64` alongside +the action, cell position, modifiers, renderer, and any explicit cell metrics. +Suppressed events still receive a sequence number and record an empty payload. +Older native packages remain usable for other operations; see the +[native mouse development setup](mouse-native-development.md) until a +mouse-capable package is published. + ## `wait` Use `wait` to synchronize on terminal state: @@ -153,6 +190,7 @@ Steps are a JSON array; each step is exactly one verb. The shape mirrors the res { "run": "nvim --clean", "noWait": true }, { "wait": { "screenStableMs": 1000 } }, { "sendKeys": ["i"] }, + { "mouse": { "action": "press", "button": "left", "row": 8, "col": 20 } }, { "type": "hello" }, { "sendKeys": ["Escape"] }, { "type": ":wq" }, @@ -163,6 +201,7 @@ Steps are a JSON array; each step is exactly one verb. The shape mirrors the res - `type` / `paste`: a string of literal text. - `sendKeys`: a non-empty array of key names — individual named keys or single characters (e.g. `["Enter"]`, `["Ctrl+C"]`, `["Escape", "Enter"]`). Multi-character literal text such as `:wq` is not a key name; send it with a `type` step. +- `mouse`: the same `action`, `button`, `row`, `col`, and optional `modifiers` fields as the `mouse` command. It may also carry `rendererName` for an explicit backend override. - `run`: a command string, with optional `noWait` (fire-and-forget) and `timeout` (ms). A `run` step is a waited run by default. - `wait`: the same conditions as the `wait` command — `text`, `regex`, `scope`, `screenStableMs`, `cursorRow`, `cursorCol`, and `timeout` (ms). @@ -297,6 +336,7 @@ Every command exits with a stable code, so scripts can branch without parsing ou | `9` | Protocol or RPC error. | | `10` | Replay failed. | | `11` | A standalone `wait` timed out, or a `wait` step inside a fail-fast `batch` timed out (`WAIT_TIMEOUT`; see [`wait`](#wait)). | +| `12` | A requested runtime capability is unavailable (`CAPABILITY_UNAVAILABLE`). | A fail-fast `batch` exits with the failed step's code (for example `11` for a wait timeout); `--keep-going` exits `1` if any step failed. diff --git a/docs/mouse-native-development.md b/docs/mouse-native-development.md new file mode 100644 index 00000000..68fcec5f --- /dev/null +++ b/docs/mouse-native-development.md @@ -0,0 +1,39 @@ +# Testing native mouse input + +Mouse input depends on [libghostty-vt-node PR #12](https://github.com/coder/libghostty-vt-node/pull/12). +The existing optional dependency remains compatible for semantic operations; +its published version does not yet encode mouse input. The adapter checks the +terminal method at runtime and reports `CAPABILITY_UNAVAILABLE` when absent. + +To test the binding before publication, build and pack its exact revision with +the pinned Zig 0.15.2 toolchain, Node, and node-gyp's platform prerequisites: + +```bash +git clone https://github.com/pmarreck/libghostty-vt-node.git +cd libghostty-vt-node +git checkout 8318d03cc4c5d417d988548f920a784abfde33af +npm ci +npm run build:libghostty +npm run build +npm run build:prebuild +npm run verify +npm pack --ignore-scripts +``` + +From the `agent-tty` checkout, install that tarball locally without changing the +published dependency or lockfile: + +```bash +npm install --no-save --package-lock=false ../libghostty-vt-node/coder-libghostty-vt-node-0.1.0-beta.1.tgz +AGENT_TTY_REQUIRE_MOUSE_NATIVE=1 npm test -- test/integration/mouse.test.ts +``` + +The integration tests use isolated session homes and a raw-mode PTY child to +check the bytes it actually receives, including non-UTF-8 X10 bytes. They also +check event-log preservation and replay, disabled reporting, and unsupported +renderers. They run automatically when mouse support is installed; the +environment flag makes missing support a failure instead of a skip. + +After the upstream package is released with this API, update the optional +dependency and lockfile together. A Git source dependency alone is insufficient: +the binding needs built TypeScript and a platform-specific native addon. diff --git a/skill-data/agent-tty/SKILL.md b/skill-data/agent-tty/SKILL.md index dd2a3c52..15f3a0c2 100644 --- a/skill-data/agent-tty/SKILL.md +++ b/skill-data/agent-tty/SKILL.md @@ -49,6 +49,7 @@ agent-tty --home run 'command here' --json agent-tty --home type 'literal text' --json agent-tty --home paste 'multiline payload' --json agent-tty --home send-keys Enter Ctrl+C --json +agent-tty --home mouse press --button left --row 8 --col 20 --json agent-tty --home batch '[{"run":"htop","noWait":true},{"wait":{"screenStableMs":1000}}]' --json # Observation and proof @@ -77,6 +78,13 @@ agent-tty --home "$AGENT_HOME" snapshot "$SESSION_ID" --format text --json Use `batch` to run an ordered sequence of input-and-`wait` steps in one call instead of separate `run`/`wait`/`send-keys` invocations. Each `wait` step is anchored to a Wait Baseline — it only observes screen state produced _after_ the preceding input step, so the sequence cannot race ahead and match a stale screen. A batch stops at the first failed step by default (`--keep-going` attempts every step). +Use `mouse` for TUI hit-testing, scrolling, and drag behavior. Coordinates are +zero-based cells. A drag is three actions: `press --button left`, one or more +button-free `move` actions, then `release --button left`. The child must enable +mouse tracking; a suppressed event returns `reported: false`. Mouse input +requires the `libghostty-vt` capability and must never be replaced with a +hard-coded SGR sequence. + ```bash AGENT_HOME="$(mktemp -d)" SESSION_ID=$(agent-tty --home "$AGENT_HOME" create --json -- /bin/bash | jq -r '.result.sessionId') diff --git a/src/batch/executor.ts b/src/batch/executor.ts index 1a009f40..500fa68f 100644 --- a/src/batch/executor.ts +++ b/src/batch/executor.ts @@ -170,6 +170,16 @@ async function runStep( seq, }; } + case 'mouse': { + const seq = await driver.mouse(step.input); + return { + index, + durationMs: Date.now() - startedAt, + kind: 'mouse', + status: 'completed', + seq, + }; + } case 'run': return runRunStep(step, index, driver, startedAt); case 'wait': @@ -324,6 +334,7 @@ function failedRecord( case 'type': case 'paste': case 'sendKeys': + case 'mouse': return { ...base, kind: step.kind, error: stepError }; default: return unreachable(step, `batch failed-step kind at index ${index}`); diff --git a/src/batch/plan.ts b/src/batch/plan.ts index 45ed6002..32da1796 100644 --- a/src/batch/plan.ts +++ b/src/batch/plan.ts @@ -1,9 +1,11 @@ import { z } from 'zod'; import type { PreparedRenderWaitCondition } from '../renderWait/matcher.js'; +import type { MouseParams } from '../protocol/messages.js'; import { assertValidKeyName } from '../pty/keyEncoder.js'; import { ERROR_CODES, makeCliError } from '../protocol/errors.js'; +import { MouseParamsSchema } from '../protocol/messages.js'; import { prepareRenderWaitCondition } from '../renderWait/matcher.js'; import { invariant, unreachable } from '../util/assert.js'; @@ -11,6 +13,7 @@ export type BatchStep = | { kind: 'type'; text: string } | { kind: 'paste'; text: string } | { kind: 'sendKeys'; keys: string[] } + | { kind: 'mouse'; input: MouseParams } | { kind: 'run'; command: string; @@ -27,7 +30,14 @@ export interface BatchPlan { steps: BatchStep[]; } -const VERB_KEYS = ['type', 'paste', 'sendKeys', 'run', 'wait'] as const; +const VERB_KEYS = [ + 'type', + 'paste', + 'sendKeys', + 'mouse', + 'run', + 'wait', +] as const; type VerbKey = (typeof VERB_KEYS)[number]; @@ -41,6 +51,7 @@ const PasteStepSchema = z.object({ paste: z.string().min(1) }).strict(); const SendKeysStepSchema = z .object({ sendKeys: z.array(z.string().min(1)).min(1) }) .strict(); +const MouseStepSchema = z.object({ mouse: z.unknown() }).strict(); const RunStepSchema = z .object({ run: z.string().min(1), @@ -90,13 +101,13 @@ function parseStep(rawStep: unknown, index: number): BatchStep { const verbs = presentVerbKeys(rawStep); if (verbs.length === 0) { return invalidInput( - `Batch step ${String(index)} must have exactly one of type|paste|sendKeys|run|wait; found none`, + `Batch step ${String(index)} must have exactly one of type|paste|sendKeys|mouse|run|wait; found none`, index, ); } if (verbs.length > 1) { return invalidInput( - `Batch step ${String(index)} must have exactly one of type|paste|sendKeys|run|wait; found ${verbs.join(', ')}`, + `Batch step ${String(index)} must have exactly one of type|paste|sendKeys|mouse|run|wait; found ${verbs.join(', ')}`, index, ); } @@ -111,6 +122,8 @@ function parseStep(rawStep: unknown, index: number): BatchStep { return parsePasteStep(rawStep, index); case 'sendKeys': return parseSendKeysStep(rawStep, index); + case 'mouse': + return parseMouseStep(rawStep, index); case 'run': return parseRunStep(rawStep, index); case 'wait': @@ -164,6 +177,21 @@ function parseSendKeysStep( return { kind: 'sendKeys', keys }; } +function parseMouseStep( + rawStep: Record, + index: number, +): BatchStep { + const { mouse } = unwrapStep(MouseStepSchema, rawStep, index); + const result = MouseParamsSchema.safeParse(mouse); + if (!result.success) { + throw makeCliError(ERROR_CODES.INVALID_INPUT, { + message: `Batch step ${String(index)} is invalid`, + details: { stepIndex: index, issues: result.error.issues }, + }); + } + return { kind: 'mouse', input: result.data }; +} + function parseRunStep( rawStep: Record, index: number, diff --git a/src/batch/result.ts b/src/batch/result.ts index 30836e81..1ca44bee 100644 --- a/src/batch/result.ts +++ b/src/batch/result.ts @@ -30,7 +30,7 @@ export const InputStepRecordSchema = z .object({ index: NonNegativeIntSchema, durationMs: NonNegativeIntSchema, - kind: z.enum(['type', 'paste', 'sendKeys']), + kind: z.enum(['type', 'paste', 'sendKeys', 'mouse']), status: StepStatusSchema, seq: NonNegativeIntSchema.optional(), error: BatchStepErrorSchema.optional(), @@ -105,6 +105,7 @@ export function unreachedStepRecord( case 'type': case 'paste': case 'sendKeys': + case 'mouse': return { ...base, kind: step.kind }; default: return unreachable(step, `batch unreached-step kind at index ${index}`); diff --git a/src/batch/stepDriver.ts b/src/batch/stepDriver.ts index 49c3a86c..875cdab2 100644 --- a/src/batch/stepDriver.ts +++ b/src/batch/stepDriver.ts @@ -1,11 +1,16 @@ import type { z } from 'zod'; -import type { RunResult, WaitForRenderResult } from '../protocol/messages.js'; +import type { + MouseParams, + RunResult, + WaitForRenderResult, +} from '../protocol/messages.js'; import type { PreparedRenderWaitCondition } from '../renderWait/matcher.js'; import { sendRpc } from '../host/rpcClient.js'; import { PasteResultSchema, + MouseResultSchema, RunResultSchema, SendKeysResultSchema, TypeResultSchema, @@ -23,6 +28,7 @@ export interface StepDriver { type(text: string): Promise; paste(text: string): Promise; sendKeys(keys: string[]): Promise; + mouse(input: MouseParams): Promise; run( command: string, noWait: boolean, @@ -107,6 +113,17 @@ export function createRpcStepDriver( return parseOrThrow(SendKeysResultSchema, raw, 'sendKeys').seq; }, + async mouse(input: MouseParams): Promise { + const params = { + ...input, + ...(input.rendererName === undefined && rendererName !== undefined + ? { rendererName } + : {}), + }; + const raw = await sendRpc(socketPath, 'mouse', params); + return parseOrThrow(MouseResultSchema, raw, 'mouse').seq; + }, + async run( command: string, noWait: boolean, diff --git a/src/cli/commands/batch.ts b/src/cli/commands/batch.ts index 97e67279..8ce2d299 100644 --- a/src/cli/commands/batch.ts +++ b/src/cli/commands/batch.ts @@ -177,6 +177,7 @@ function stepOutcome(record: BatchStepRecord): string { case 'type': case 'paste': case 'sendKeys': + case 'mouse': return 'completed'; } } diff --git a/src/cli/commands/mouse.ts b/src/cli/commands/mouse.ts new file mode 100644 index 00000000..3f05f316 --- /dev/null +++ b/src/cli/commands/mouse.ts @@ -0,0 +1,83 @@ +import type { CommandContext } from '../context.js'; +import type { MouseResult } from '../../protocol/messages.js'; + +import { resolveCommandTarget } from '../commandTarget.js'; +import { emitSuccess } from '../output.js'; +import { sendRpc } from '../../host/rpcClient.js'; +import { + MouseParamsSchema, + MouseResultSchema, +} from '../../protocol/messages.js'; +import { ERROR_CODES, makeCliError } from '../../protocol/errors.js'; + +interface CommandOptions { + context: CommandContext; + json: boolean; + sessionId: string; + action: string; + button?: string; + row: number; + col: number; + cellWidth?: number; + cellHeight?: number; + shift: boolean; + alt: boolean; + ctrl: boolean; +} + +/** Validate one cell-addressed action before asking the host to encode it. */ +export async function runMouseCommand(options: CommandOptions): Promise { + const paramsResult = MouseParamsSchema.safeParse({ + action: options.action, + ...(options.button === undefined ? {} : { button: options.button }), + row: options.row, + col: options.col, + ...(options.cellWidth === undefined + ? {} + : { cellWidth: options.cellWidth }), + ...(options.cellHeight === undefined + ? {} + : { cellHeight: options.cellHeight }), + modifiers: { + shift: options.shift, + alt: options.alt, + ctrl: options.ctrl, + }, + rendererName: options.context.rendererDefault, + }); + if (!paramsResult.success) { + throw makeCliError(ERROR_CODES.INVALID_INPUT, { + message: 'Invalid mouse input.', + details: { issues: paramsResult.error.issues }, + }); + } + + const target = await resolveCommandTarget({ + home: options.context.home, + sessionId: options.sessionId, + }); + const rawResult: unknown = await sendRpc( + target.socketPath, + 'mouse', + paramsResult.data, + ); + const result = MouseResultSchema.safeParse(rawResult); + if (!result.success) { + throw makeCliError(ERROR_CODES.PROTOCOL_ERROR, { + message: 'Unexpected response from host', + details: { issues: result.error.issues }, + }); + } + + const mouseResult: MouseResult = result.data; + emitSuccess({ + command: 'mouse', + json: options.json, + result: mouseResult, + lines: [ + mouseResult.reported + ? `Sent ${String(mouseResult.bytesWritten)} mouse byte(s), seq ${String(mouseResult.seq)}.` + : `Mouse tracking suppressed the input, seq ${String(mouseResult.seq)}.`, + ], + }); +} diff --git a/src/cli/commands/version.ts b/src/cli/commands/version.ts index 30d0e68e..4dc3b7e2 100644 --- a/src/cli/commands/version.ts +++ b/src/cli/commands/version.ts @@ -8,7 +8,7 @@ import { RendererNameSchema } from '../../renderer/names.js'; import { loadPackageMetadata } from '../../util/packageMetadata.js'; const COMMAND_NAME = 'version'; -const PROTOCOL_VERSION = '0.1.0'; +const PROTOCOL_VERSION = '0.2.0'; export interface VersionResult { cliVersion: string; diff --git a/src/cli/exitCodes.ts b/src/cli/exitCodes.ts index de68c7f5..f4e93287 100644 --- a/src/cli/exitCodes.ts +++ b/src/cli/exitCodes.ts @@ -22,6 +22,7 @@ const EXIT_CODE_BY_ERROR_CODE: Readonly> = Object.freeze( [ERROR_CODES.RPC_ERROR]: 9, [ERROR_CODES.REPLAY_ERROR]: 10, [ERROR_CODES.WAIT_TIMEOUT]: 11, + [ERROR_CODES.CAPABILITY_UNAVAILABLE]: 12, [ERROR_CODES.INTERNAL_ERROR]: 1, }, ); diff --git a/src/cli/main.ts b/src/cli/main.ts index 2070567f..694c4918 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -17,6 +17,7 @@ import { runHomeListCommand } from './commands/home/list.js'; import { runInspectCommand } from './commands/inspect.js'; import { runListCommand } from './commands/list.js'; import { runMarkCommand } from './commands/mark.js'; +import { runMouseCommand } from './commands/mouse.js'; import { runPasteCommand } from './commands/paste.js'; import { runRunCommand } from './commands/run.js'; import { runRecordDiffCommand } from './commands/record-diff.js'; @@ -60,6 +61,17 @@ function parseNumberOption(value: string): number { return Number(value); } +function parseNonNegativeIntegerOption(value: string): number { + const parsed = Number(value); + if (!Number.isInteger(parsed) || parsed < 0) { + throw makeCliError(ERROR_CODES.INVALID_INPUT, { + message: 'Mouse row and column values must be non-negative integers.', + details: { value }, + }); + } + return parsed; +} + // Strict integer-token parser: empty, whitespace-only, fractional, partially // numeric (e.g. "", "1.5", "2junk"), or unsafe-magnitude tokens yield NaN so // command validation rejects them instead of silently truncating or rounding. @@ -642,6 +654,75 @@ async function main(): Promise { ), ); + program + .command('mouse ') + .description('Send mode-aware mouse input to a terminal cell') + .requiredOption( + '--row ', + 'Zero-based terminal row', + parseNonNegativeIntegerOption, + ) + .requiredOption( + '--col ', + 'Zero-based terminal column', + parseNonNegativeIntegerOption, + ) + .option('--button ', 'left, middle, right, or a wheel direction') + .option( + '--cell-width ', + 'Cell width in pixels (default: virtual unit cell)', + parseNumberOption, + ) + .option( + '--cell-height ', + 'Cell height in pixels (default: virtual unit cell)', + parseNumberOption, + ) + .option('--shift', 'Hold Shift for this mouse action', false) + .option('--alt', 'Hold Alt for this mouse action', false) + .option('--ctrl', 'Hold Ctrl for this mouse action', false) + .option('--json', 'Emit a JSON command envelope', false) + .action( + wrapAction( + 'mouse', + async ( + sessionId: string, + action: string, + options: { + row: number; + col: number; + cellWidth?: number; + cellHeight?: number; + button?: string; + shift: boolean; + alt: boolean; + ctrl: boolean; + json: boolean; + }, + context: CommandContext, + ) => { + await runMouseCommand({ + context, + json: options.json, + sessionId, + action, + ...(options.button === undefined ? {} : { button: options.button }), + row: options.row, + col: options.col, + ...(options.cellWidth === undefined + ? {} + : { cellWidth: options.cellWidth }), + ...(options.cellHeight === undefined + ? {} + : { cellHeight: options.cellHeight }), + shift: options.shift, + alt: options.alt, + ctrl: options.ctrl, + }); + }, + ), + ); + program .command('resize ') .description('Resize a session') diff --git a/src/export/asciicast.ts b/src/export/asciicast.ts index 7d918cf2..8c9bca6f 100644 --- a/src/export/asciicast.ts +++ b/src/export/asciicast.ts @@ -154,6 +154,7 @@ export function generateAsciicast( case 'input_text': case 'input_paste': case 'input_keys': + case 'input_mouse': case 'input_run': case 'run_complete': case 'signal': diff --git a/src/host/eventLog.ts b/src/host/eventLog.ts index df85f6b8..7867cb0d 100644 --- a/src/host/eventLog.ts +++ b/src/host/eventLog.ts @@ -7,10 +7,12 @@ import { z } from 'zod'; import { EventRecordSchema, + InputMouseEventPayloadSchema, InputRunEventPayloadSchema, MarkerEventPayloadSchema, RunCompleteEventPayloadSchema, type EventRecord, + type InputMouseEventPayload, type InputRunEventPayload, type MarkerEventPayload, type RunCompleteEventPayload, @@ -85,6 +87,7 @@ type EventLogEventType = | 'input_text' | 'input_paste' | 'input_keys' + | 'input_mouse' | 'input_run' | 'run_complete' | 'resize' @@ -96,6 +99,7 @@ type EventLogPayload = | InputTextEventPayload | InputPasteEventPayload | InputKeysEventPayload + | InputMouseEventPayload | InputRunEventPayload | RunCompleteEventPayload | ResizeEventPayload @@ -138,6 +142,11 @@ function validatePayload( invariant(result.success, 'input_keys payload must match schema'); return result.data; } + case 'input_mouse': { + const result = InputMouseEventPayloadSchema.safeParse(payload); + invariant(result.success, 'input_mouse payload must match schema'); + return result.data; + } case 'input_run': { const result = InputRunEventPayloadSchema.safeParse(payload); invariant(result.success, 'input_run payload must match schema'); @@ -300,6 +309,10 @@ export class EventLog { type: 'input_keys', payload: InputKeysEventPayload, ): Promise; + async append( + type: 'input_mouse', + payload: InputMouseEventPayload, + ): Promise; async append( type: 'input_run', payload: InputRunEventPayload, diff --git a/src/host/hostMain.ts b/src/host/hostMain.ts index 8a318976..c7cfc78e 100644 --- a/src/host/hostMain.ts +++ b/src/host/hostMain.ts @@ -21,6 +21,7 @@ import { ERROR_CODES, makeCliError } from '../protocol/errors.js'; import { isCommandableSessionStatus } from '../protocol/sessionStatusPolicy.js'; import type { MarkParams, + MouseParams, PasteParams, ResizeParams, RunParams, @@ -46,6 +47,8 @@ import { } from '../renderer/names.js'; import { resolveProfile } from '../renderer/profiles.js'; import { createRendererBackend } from '../renderer/registry.js'; +import { supportsMouseEncoding } from '../renderer/backend.js'; +import { prepareHeldMouseButtons, type HeldMouseButton } from './mouseInput.js'; import { captureScreenshotResult } from '../screenshot/capture.js'; import { captureSnapshotResult } from '../snapshot/capture.js'; import { resolveHome } from '../storage/home.js'; @@ -217,6 +220,7 @@ export async function runHost(sessionId: string): Promise { appendRunComplete: (payload) => eventLog.append('run_complete', payload), }); let ptyIngestionQueue: Promise = Promise.resolve(); + let heldMouseButtons: ReadonlySet = new Set(); // Per-client wait-exit callbacks, cleaned up individually via ResourceScope. // Using ptyExitPromise.then() would permanently attach to the shared promise. @@ -721,6 +725,118 @@ export async function runHost(sessionId: string): Promise { seq, }; }, + mouse: async (params: unknown) => { + const { + action, + button, + row, + col, + cellWidth, + cellHeight, + modifiers, + rendererName: requestedRendererName, + } = params as MouseParams; + + assertSessionCommandable(state); + await ptyIngestionQueue; + + const normalizedModifiers = { + shift: modifiers.shift ?? false, + alt: modifiers.alt ?? false, + ctrl: modifiers.ctrl ?? false, + }; + const rendererName = resolveHostRendererName(requestedRendererName); + if (rendererName !== 'libghostty-vt') { + throw makeCliError(ERROR_CODES.CAPABILITY_UNAVAILABLE, { + message: `Renderer ${rendererName} cannot encode mouse input. Use libghostty-vt.`, + }); + } + const profile = resolveProfile(DEFAULT_RENDER_PROFILE_NAME); + const replayInput = loadReplayInput(); + + let encoded: Buffer; + let rendererBackend: string; + try { + ({ encoded, rendererBackend } = await rendererManager.withBackend( + rendererName, + profile, + replayInput, + (backend) => { + if (!supportsMouseEncoding(backend)) { + throw makeCliError(ERROR_CODES.CAPABILITY_UNAVAILABLE, { + message: `Renderer ${backend.rendererBackend} cannot encode mouse input. Use libghostty-vt.`, + details: { rendererBackend: backend.rendererBackend }, + }); + } + // The renderer lock also serializes held-button transitions. A + // second request can arrive while the first backend is booting. + const transition = prepareHeldMouseButtons(heldMouseButtons, { + action, + ...(button === undefined ? {} : { button }), + }); + const encoded = backend.encodeMouse({ + action, + ...(button === undefined + ? action === 'move' && transition.dragButton !== undefined + ? { button: transition.dragButton } + : {} + : { button }), + row, + col, + ...(cellWidth === undefined ? {} : { cellWidth }), + ...(cellHeight === undefined ? {} : { cellHeight }), + modifiers: normalizedModifiers, + anyButtonPressed: transition.anyButtonPressed, + }); + assertSessionCommandable(state); + if (encoded.length > 0) { + pty.write(encoded); + } + heldMouseButtons = transition.nextHeld; + return { + encoded, + rendererBackend: backend.rendererBackend, + }; + }, + )); + } catch (error) { + if (error instanceof RangeError || error instanceof TypeError) { + throw makeCliError(ERROR_CODES.INVALID_INPUT, { + message: error.message, + cause: error, + }); + } + if ( + error instanceof Error && + error.message.includes('terminal.encodeMouse is required') + ) { + throw makeCliError(ERROR_CODES.CAPABILITY_UNAVAILABLE, { + message: error.message, + details: { rendererBackend: rendererName }, + cause: error, + }); + } + throw error; + } + + lastActivityAt = Date.now(); + const seq = await eventLog.append('input_mouse', { + action, + ...(button === undefined ? {} : { button }), + row, + col, + modifiers: normalizedModifiers, + rendererBackend, + ...(cellWidth === undefined ? {} : { cellWidth }), + ...(cellHeight === undefined ? {} : { cellHeight }), + dataBase64: encoded.toString('base64'), + }); + return { + reported: encoded.length > 0, + bytesWritten: encoded.length, + seq, + }; + }, resize: async (params: unknown) => { const { cols, rows } = params as ResizeParams; diff --git a/src/host/mouseInput.ts b/src/host/mouseInput.ts new file mode 100644 index 00000000..178e74e4 --- /dev/null +++ b/src/host/mouseInput.ts @@ -0,0 +1,83 @@ +import type { MouseAction, MouseButton } from '../protocol/schemas.js'; +import { invariant, unreachable } from '../util/assert.js'; + +export type HeldMouseButton = 'left' | 'middle' | 'right'; + +interface MouseButtonTransitionInput { + action: MouseAction; + button?: MouseButton; +} + +interface HeldMouseButtonTransition { + nextHeld: ReadonlySet; + anyButtonPressed: boolean; + dragButton?: HeldMouseButton; +} + +const HOLDABLE_BUTTONS = new Set(['left', 'middle', 'right']); +const WHEEL_BUTTONS = new Set([ + 'wheel-up', + 'wheel-down', + 'wheel-left', + 'wheel-right', +]); + +function isHeldMouseButton(button: MouseButton): button is HeldMouseButton { + return HOLDABLE_BUTTONS.has(button); +} + +/** + * Stage held-button state without mutating the committed host state. The host + * adopts `nextHeld` only after mode-aware encoding succeeds. + */ +export function prepareHeldMouseButtons( + currentHeld: ReadonlySet, + input: MouseButtonTransitionInput, +): HeldMouseButtonTransition { + invariant(currentHeld instanceof Set, 'currentHeld must be a Set'); + invariant( + input.action === 'press' || + input.action === 'release' || + input.action === 'move', + 'mouse action must be press, release, or move', + ); + invariant( + input.button === undefined || + HOLDABLE_BUTTONS.has(input.button) || + WHEEL_BUTTONS.has(input.button), + 'mouse button is invalid', + ); + + const nextHeld = new Set(currentHeld); + switch (input.action) { + case 'press': + invariant(input.button !== undefined, 'press requires a mouse button'); + if (isHeldMouseButton(input.button)) { + // Reinsert so Set iteration order retains the most recently pressed + // button for native drag reports when more than one is held. + nextHeld.delete(input.button); + nextHeld.add(input.button); + } + break; + case 'release': + invariant(input.button !== undefined, 'release requires a mouse button'); + invariant( + isHeldMouseButton(input.button), + 'wheel buttons do not have release transitions', + ); + nextHeld.delete(input.button); + break; + case 'move': + invariant(input.button === undefined, 'move must not specify a button'); + break; + default: + unreachable(input.action, 'unsupported mouse action'); + } + + const dragButton = [...nextHeld].at(-1); + return { + nextHeld, + anyButtonPressed: nextHeld.size > 0, + ...(dragButton === undefined ? {} : { dragButton }), + }; +} diff --git a/src/protocol/errors.ts b/src/protocol/errors.ts index a3b73b56..f86d33c3 100644 --- a/src/protocol/errors.ts +++ b/src/protocol/errors.ts @@ -14,6 +14,7 @@ export const ERROR_CODES = { INVALID_KEYS: 'INVALID_KEYS', INVALID_DURATION: 'INVALID_DURATION', INVALID_INPUT: 'INVALID_INPUT', + CAPABILITY_UNAVAILABLE: 'CAPABILITY_UNAVAILABLE', STORAGE_READ_ERROR: 'STORAGE_READ_ERROR', STORAGE_WRITE_ERROR: 'STORAGE_WRITE_ERROR', MANIFEST_VALIDATION_ERROR: 'MANIFEST_VALIDATION_ERROR', @@ -40,6 +41,8 @@ export const DEFAULT_ERROR_MESSAGES: Record = { [ERROR_CODES.INVALID_KEYS]: 'Key sequence is invalid.', [ERROR_CODES.INVALID_DURATION]: 'Duration value is invalid.', [ERROR_CODES.INVALID_INPUT]: 'Invalid input provided.', + [ERROR_CODES.CAPABILITY_UNAVAILABLE]: + 'The requested capability is unavailable.', [ERROR_CODES.STORAGE_READ_ERROR]: 'Failed to read session storage.', [ERROR_CODES.STORAGE_WRITE_ERROR]: 'Failed to write session storage.', [ERROR_CODES.MANIFEST_VALIDATION_ERROR]: 'Session manifest is invalid.', diff --git a/src/protocol/messages.ts b/src/protocol/messages.ts index 0434a153..e2d3ac1a 100644 --- a/src/protocol/messages.ts +++ b/src/protocol/messages.ts @@ -20,7 +20,10 @@ import { WaitForRenderParamsSchema, WaitForRenderResultSchema, WaitResultSchema, + MouseActionSchema, + MouseButtonSchema, } from './schemas.js'; +import { RendererNameSchema } from '../renderer/names.js'; export { RecordDiffLineSchema, @@ -300,6 +303,64 @@ export const SendKeysResultSchema = z .strict(); export type SendKeysResult = z.infer; +const MouseInputModifiersSchema = z + .object({ + shift: z.boolean().optional(), + alt: z.boolean().optional(), + ctrl: z.boolean().optional(), + }) + .strict(); + +export const MouseParamsSchema = z + .object({ + action: MouseActionSchema, + button: MouseButtonSchema.optional(), + row: z.number().int().nonnegative(), + col: z.number().int().nonnegative(), + cellWidth: z.number().int().positive().max(0xffffffff).optional(), + cellHeight: z.number().int().positive().max(0xffffffff).optional(), + modifiers: MouseInputModifiersSchema.optional().default({}), + rendererName: RendererNameSchema.optional(), + }) + .strict() + .superRefine((event, ctx) => { + if (event.action === 'move' && event.button !== undefined) { + ctx.addIssue({ + code: 'custom', + message: 'button must not be set for move actions', + path: ['button'], + }); + } + if (event.action !== 'move' && event.button === undefined) { + ctx.addIssue({ + code: 'custom', + message: 'button is required for press and release actions', + path: ['button'], + }); + } + if ( + event.action === 'release' && + event.button !== undefined && + event.button.startsWith('wheel-') + ) { + ctx.addIssue({ + code: 'custom', + message: 'wheel buttons are instantaneous press actions', + path: ['button'], + }); + } + }); +export type MouseParams = z.infer; + +export const MouseResultSchema = z + .object({ + reported: z.boolean(), + bytesWritten: z.number().int().nonnegative(), + seq: z.number().int().nonnegative(), + }) + .strict(); +export type MouseResult = z.infer; + export const ResizeParamsSchema = z .object({ cols: z.number().int().positive(), @@ -365,6 +426,7 @@ const RPC_METHODS = [ 'run', 'mark', 'sendKeys', + 'mouse', 'resize', 'signal', 'wait', @@ -408,6 +470,10 @@ export const RpcMethodSchemas = { params: SendKeysParamsSchema, result: SendKeysResultSchema, }, + mouse: { + params: MouseParamsSchema, + result: MouseResultSchema, + }, resize: { params: ResizeParamsSchema, result: ResizeResultSchema, diff --git a/src/protocol/schemas.ts b/src/protocol/schemas.ts index 4c3c1be6..2a75a9ec 100644 --- a/src/protocol/schemas.ts +++ b/src/protocol/schemas.ts @@ -110,6 +110,66 @@ export const InputKeysEventPayloadSchema = z .strict(); export type InputKeysEventPayload = z.infer; +export const MouseActionSchema = z.enum(['press', 'release', 'move']); +export type MouseAction = z.infer; + +export const MouseButtonSchema = z.enum([ + 'left', + 'middle', + 'right', + 'wheel-up', + 'wheel-down', + 'wheel-left', + 'wheel-right', +]); +export type MouseButton = z.infer; + +export const MouseModifiersSchema = z + .object({ + shift: z.boolean(), + alt: z.boolean(), + ctrl: z.boolean(), + }) + .strict(); +export type MouseModifiers = z.infer; + +export const InputMouseEventPayloadSchema = z + .object({ + action: MouseActionSchema, + button: MouseButtonSchema.optional(), + row: NonNegativeIntSchema, + col: NonNegativeIntSchema, + cellWidth: PositiveIntSchema.optional(), + cellHeight: PositiveIntSchema.optional(), + modifiers: MouseModifiersSchema, + rendererBackend: NonEmptyStringSchema, + dataBase64: z + .string() + .regex( + /^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/u, + ), + }) + .strict() + .superRefine((event, ctx) => { + if (event.action === 'move' && event.button !== undefined) { + ctx.addIssue({ + code: 'custom', + message: 'button must not be set for move events', + path: ['button'], + }); + } + if (event.action !== 'move' && event.button === undefined) { + ctx.addIssue({ + code: 'custom', + message: 'button is required for press and release events', + path: ['button'], + }); + } + }); +export type InputMouseEventPayload = z.infer< + typeof InputMouseEventPayloadSchema +>; + export const InputRunEventPayloadSchema = z .object({ command: z.string().min(1), @@ -183,6 +243,7 @@ export const EventTypeSchema = z.enum([ 'input_text', 'input_paste', 'input_keys', + 'input_mouse', 'input_run', 'run_complete', 'resize', @@ -229,6 +290,14 @@ export const InputKeysEventRecordSchema = z }) .strict(); +export const InputMouseEventRecordSchema = z + .object({ + ...EventRecordBaseShape, + type: z.literal('input_mouse'), + payload: InputMouseEventPayloadSchema, + }) + .strict(); + export const InputRunEventRecordSchema = z .object({ ...EventRecordBaseShape, @@ -282,6 +351,7 @@ export const EventRecordSchema = z.discriminatedUnion('type', [ InputTextEventRecordSchema, InputPasteEventRecordSchema, InputKeysEventRecordSchema, + InputMouseEventRecordSchema, InputRunEventRecordSchema, RunCompleteEventRecordSchema, ResizeEventRecordSchema, diff --git a/src/renderer/backend.ts b/src/renderer/backend.ts index 5ede601f..d1b7cc72 100644 --- a/src/renderer/backend.ts +++ b/src/renderer/backend.ts @@ -4,6 +4,11 @@ import type { ScreenshotResult, SemanticSnapshot, } from './types.js'; +import type { + MouseAction, + MouseButton, + MouseModifiers, +} from '../protocol/schemas.js'; export interface SnapshotOptions { includeScrollback?: boolean; @@ -43,6 +48,31 @@ export interface RendererBackend { readonly isBooted: boolean; } +export interface MouseEncodingInput { + action: MouseAction; + button?: MouseButton; + row: number; + col: number; + cellWidth?: number; + cellHeight?: number; + modifiers: MouseModifiers; + anyButtonPressed: boolean; +} + +export interface MouseEncodingBackend extends RendererBackend { + /** Encode one semantic mouse action using modes negotiated by the child. */ + encodeMouse(input: MouseEncodingInput): Buffer; +} + +export function supportsMouseEncoding( + backend: RendererBackend, +): backend is MouseEncodingBackend { + return ( + 'encodeMouse' in backend && + typeof (backend as Partial).encodeMouse === 'function' + ); +} + export interface VideoRecordingOptions { outputDir: string; size: { width: number; height: number }; diff --git a/src/renderer/capabilities.ts b/src/renderer/capabilities.ts index f5eebde3..9bbf2f55 100644 --- a/src/renderer/capabilities.ts +++ b/src/renderer/capabilities.ts @@ -14,6 +14,7 @@ import { export const CapabilityNameSchema = z.enum([ 'snapshot', 'wait', + 'mouse-input', 'screenshot', 'record-export-asciicast', 'record-export-webm', @@ -79,6 +80,7 @@ export type RendererRuntimeSummary = z.infer< const CAPABILITY_NAMES: ReadonlyArray = Object.freeze([ 'snapshot', 'wait', + 'mouse-input', 'screenshot', 'record-export-asciicast', 'record-export-webm', @@ -549,6 +551,37 @@ async function discoverDashboardCapability( return buildDashboardCapability(probe, mode); } +async function discoverMouseInputCapability( + mode: DiscoveryMode, + deps: CapabilityDiscoveryDependencies, +): Promise { + const probe = await (deps.probeLibghosttyVt ?? probeLibghosttyVt)(); + if (!probe.available) { + return { + name: 'mouse-input', + status: 'unavailable', + ...(probe.reason === undefined ? {} : { reason: probe.reason }), + ...(probe.detail === undefined ? {} : { detail: probe.detail }), + }; + } + if (probe.mouseInputAvailable === false) { + return { + name: 'mouse-input', + status: 'unavailable', + reason: 'libghostty-vt mouse encoder unavailable', + ...(probe.detail === undefined ? {} : { detail: probe.detail }), + }; + } + return mode === 'full' + ? { + name: 'mouse-input', + status: 'available', + reason: 'libghostty-vt mouse encoder available', + ...(probe.detail === undefined ? {} : { detail: probe.detail }), + } + : { name: 'mouse-input', status: 'available' }; +} + function validateDiscoveredCapabilities( capabilities: ReadonlyArray, ): CapabilityEntry[] { @@ -617,6 +650,8 @@ export async function discoverCapabilities( capabilities.push(buildBuiltinCapability(name, mode)); } + capabilities.push(await discoverMouseInputCapability(mode, cachedDeps)); + capabilities.push( await buildPlaywrightCapability('screenshot', mode, cachedDeps), ); diff --git a/src/renderer/ghosttyWeb/backend.ts b/src/renderer/ghosttyWeb/backend.ts index 7e0bde4d..dd51c366 100644 --- a/src/renderer/ghosttyWeb/backend.ts +++ b/src/renderer/ghosttyWeb/backend.ts @@ -456,6 +456,7 @@ export class GhosttyWebBackend implements VideoCapableRendererBackend { case 'input_text': case 'input_paste': case 'input_keys': + case 'input_mouse': case 'input_run': case 'run_complete': case 'signal': @@ -715,6 +716,7 @@ export class GhosttyWebBackend implements VideoCapableRendererBackend { case 'input_text': case 'input_paste': case 'input_keys': + case 'input_mouse': case 'input_run': case 'run_complete': case 'signal': diff --git a/src/renderer/libghosttyVt/backend.ts b/src/renderer/libghosttyVt/backend.ts index a33b62d1..412b6c40 100644 --- a/src/renderer/libghosttyVt/backend.ts +++ b/src/renderer/libghosttyVt/backend.ts @@ -11,6 +11,8 @@ import type { import type { RendererBackend, + MouseEncodingBackend, + MouseEncodingInput, ScreenshotOptions, SnapshotOptions, } from '../backend.js'; @@ -29,8 +31,47 @@ import { DEFAULT_COLS, DEFAULT_ROWS } from '../../config/defaults.js'; import { invariant, assertString, unreachable } from '../../util/assert.js'; import { Logger, createProcessLogger } from '../../util/logger.js'; +type NativeMouseAction = 'press' | 'release' | 'motion'; +type NativeMouseButton = + | 'left' + | 'middle' + | 'right' + | 'four' + | 'five' + | 'six' + | 'seven'; + +interface NativeMouseInputEvent { + action: NativeMouseAction; + button?: NativeMouseButton; + x: number; + y: number; + modifiers?: { shift?: boolean; alt?: boolean; ctrl?: boolean }; +} + +interface NativeMouseEncoderOptions { + geometry: { + screenWidth: number; + screenHeight: number; + cellWidth: number; + cellHeight: number; + }; + anyButtonPressed?: boolean; + trackLastCell?: boolean; +} + +type MouseAwareGhosttyVtTerminal = Pick< + GhosttyVtTerminal, + 'feed' | 'resize' | 'snapshot' | 'getVisibleText' | 'dispose' +> & { + encodeMouse?( + event: NativeMouseInputEvent, + options: NativeMouseEncoderOptions, + ): Buffer; +}; + export interface LibghosttyVtNativeModule { - createTerminal(options: CreateTerminalOptions): GhosttyVtTerminal; + createTerminal(options: CreateTerminalOptions): MouseAwareGhosttyVtTerminal; getNativeInfo?: () => NativeInfo; } @@ -345,7 +386,17 @@ function mapNativeCells( }); } -export class LibghosttyVtBackend implements RendererBackend { +const NATIVE_MOUSE_BUTTONS: Readonly> = { + left: 'left', + middle: 'middle', + right: 'right', + 'wheel-up': 'four', + 'wheel-down': 'five', + 'wheel-left': 'six', + 'wheel-right': 'seven', +}; + +export class LibghosttyVtBackend implements MouseEncodingBackend { public readonly rendererBackend = 'libghostty-vt'; public isBooted = false; @@ -370,7 +421,7 @@ export class LibghosttyVtBackend implements RendererBackend { private initialReplayRows: number | null = null; private lastAppliedSeq = -1; private latestReplayInput: ReplayInput | null = null; - private terminal: GhosttyVtTerminal | null = null; + private terminal: MouseAwareGhosttyVtTerminal | null = null; public constructor( sessionId: string, @@ -495,6 +546,7 @@ export class LibghosttyVtBackend implements RendererBackend { case 'input_text': case 'input_paste': case 'input_keys': + case 'input_mouse': case 'input_run': case 'run_complete': case 'signal': @@ -562,6 +614,62 @@ export class LibghosttyVtBackend implements RendererBackend { return SemanticSnapshotSchema.parse(semanticSnapshot); } + /** + * Encode cell-addressed mouse input using the child-selected Ghostty modes. + * Explicit cell metrics map those cells into the surface's pixel space. + */ + public encodeMouse(input: MouseEncodingInput): Buffer { + const terminal = this.requireTerminal('encodeMouse()'); + invariant( + typeof terminal.encodeMouse === 'function', + 'terminal.encodeMouse is required; update @coder/libghostty-vt-node to a mouse-capable release', + ); + assertNonNegativeInteger(input.row, 'mouse row must be non-negative'); + assertNonNegativeInteger(input.col, 'mouse col must be non-negative'); + const cellWidth = input.cellWidth ?? 1; + const cellHeight = input.cellHeight ?? 1; + assertPositiveInteger(cellWidth, 'mouse cellWidth must be positive'); + assertPositiveInteger(cellHeight, 'mouse cellHeight must be positive'); + invariant( + typeof input.anyButtonPressed === 'boolean', + 'mouse anyButtonPressed must be a boolean', + ); + + const nativeButton = + input.button === undefined + ? undefined + : NATIVE_MOUSE_BUTTONS[input.button]; + invariant( + input.button === undefined || nativeButton !== undefined, + `unsupported mouse button: ${String(input.button)}`, + ); + + const bytes = terminal.encodeMouse( + { + action: input.action === 'move' ? 'motion' : input.action, + ...(nativeButton === undefined ? {} : { button: nativeButton }), + x: input.col * cellWidth, + y: input.row * cellHeight, + modifiers: { ...input.modifiers }, + }, + { + geometry: { + screenWidth: this.currentCols * cellWidth, + screenHeight: this.currentRows * cellHeight, + cellWidth, + cellHeight, + }, + anyButtonPressed: input.anyButtonPressed, + trackLastCell: true, + }, + ); + invariant( + Buffer.isBuffer(bytes), + 'terminal.encodeMouse must return a Buffer', + ); + return bytes; + } + public async screenshot( outputPath: string, options?: ScreenshotOptions, @@ -674,7 +782,7 @@ export class LibghosttyVtBackend implements RendererBackend { ); } - private requireTerminal(methodName: string): GhosttyVtTerminal { + private requireTerminal(methodName: string): MouseAwareGhosttyVtTerminal { this.assertNotDisposed(methodName); invariant( this.isBooted && this.terminal !== null, @@ -685,12 +793,12 @@ export class LibghosttyVtBackend implements RendererBackend { private assertTerminalShape( terminal: unknown, - ): asserts terminal is GhosttyVtTerminal { + ): asserts terminal is MouseAwareGhosttyVtTerminal { invariant( terminal !== null && typeof terminal === 'object', 'libghostty-vt terminal must be an object', ); - const candidate = terminal as Partial; + const candidate = terminal as Partial; invariant( typeof candidate.feed === 'function', 'terminal.feed is required', diff --git a/src/renderer/readiness.ts b/src/renderer/readiness.ts index 5b56f3fe..daf55173 100644 --- a/src/renderer/readiness.ts +++ b/src/renderer/readiness.ts @@ -17,6 +17,7 @@ export const DASHBOARD_RENDERER_UNAVAILABLE_MESSAGE = export interface LibghosttyVtProbe { available: boolean; + mouseInputAvailable?: boolean; reason?: string; detail?: string; } @@ -35,7 +36,10 @@ export async function probeLibghosttyVt( loader: LibghosttyVtLoader = defaultLoader, ): Promise { try { - const module = (await loader()) as { createTerminal?: unknown }; + const module = (await loader()) as { + createTerminal?: unknown; + supportsMouseInput?: unknown; + }; if (typeof module.createTerminal !== 'function') { return { available: false, @@ -43,10 +47,14 @@ export async function probeLibghosttyVt( detail: `${LIBGHOSTTY_VT_PACKAGE} loaded but did not expose createTerminal()`, }; } + const mouseInputAvailable = module.supportsMouseInput === true; return { available: true, + mouseInputAvailable, reason: 'libghostty-vt native module available', - detail: `${LIBGHOSTTY_VT_PACKAGE} exposes createTerminal()`, + detail: mouseInputAvailable + ? `${LIBGHOSTTY_VT_PACKAGE} exposes createTerminal() and mode-aware mouse encoding` + : `${LIBGHOSTTY_VT_PACKAGE} exposes createTerminal() but predates the mouse-capable release`, }; } catch (error) { return { diff --git a/src/renderer/types.ts b/src/renderer/types.ts index 41f63f70..88e9f9de 100644 --- a/src/renderer/types.ts +++ b/src/renderer/types.ts @@ -2,6 +2,7 @@ import { z } from 'zod'; import { MarkerEventPayloadSchema, + InputMouseEventPayloadSchema, RichSnapshotLineSchema, RunCompleteEventPayloadSchema, Sha256HexSchema, @@ -76,6 +77,15 @@ const InputKeysReplayEventSchema = z }) .strict(); +const InputMouseReplayEventSchema = z + .object({ + seq: NonNegativeIntSchema, + ts: z.iso.datetime(), + type: z.literal('input_mouse'), + payload: InputMouseEventPayloadSchema, + }) + .strict(); + const InputRunReplayEventSchema = z .object({ seq: NonNegativeIntSchema, @@ -155,6 +165,7 @@ export const ReplayEventSchema = z.discriminatedUnion('type', [ InputTextReplayEventSchema, InputPasteReplayEventSchema, InputKeysReplayEventSchema, + InputMouseReplayEventSchema, InputRunReplayEventSchema, RunCompleteReplayEventSchema, ResizeReplayEventSchema, diff --git a/test/fixtures/apps/mouse-report/main.ts b/test/fixtures/apps/mouse-report/main.ts new file mode 100644 index 00000000..7e00113a --- /dev/null +++ b/test/fixtures/apps/mouse-report/main.ts @@ -0,0 +1,11 @@ +process.stdin.setRawMode?.(true); +process.stdin.resume(); + +// Any-event tracking plus SGR encoding exercises press, drag motion, release, +// and coordinates beyond the legacy X10 byte range. +process.stdout.write('\u001b[?1003h\u001b[?1006hMOUSE_READY\r\n'); + +process.stdin.on('data', (chunk: Buffer | string) => { + const bytes = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk); + process.stdout.write(`MOUSE_HEX:${bytes.toString('hex')}\r\n`); +}); diff --git a/test/integration/cli.test.ts b/test/integration/cli.test.ts index f8d8f43a..ddc3cb5f 100644 --- a/test/integration/cli.test.ts +++ b/test/integration/cli.test.ts @@ -210,6 +210,22 @@ describe('CLI integration', () => { expect(result.stdout).toContain('List sessions'); }); + it('documents cell-addressed mouse input and its modifiers', () => { + const result = runCli(['mouse', '--help'], testEnv()); + + expect(result.status).toBe(0); + expect(result.stderr).toBe(''); + expect(result.stdout).toContain( + 'Usage: agent-tty mouse [options] ', + ); + expect(result.stdout).toContain('--row '); + expect(result.stdout).toContain('--col '); + expect(result.stdout).toContain('--button '); + expect(result.stdout).toContain('--shift'); + expect(result.stdout).toContain('--alt'); + expect(result.stdout).toContain('--ctrl'); + }, 30_000); + it('accepts --append-newline for type', () => { const result = runCli( ['type', 'session-01', 'hello', '--append-newline', '--json'], diff --git a/test/integration/host-renderer-rpc.test.ts b/test/integration/host-renderer-rpc.test.ts index 7ec4b147..5ff1f707 100644 --- a/test/integration/host-renderer-rpc.test.ts +++ b/test/integration/host-renderer-rpc.test.ts @@ -1,6 +1,7 @@ import { mkdtemp, readFile, realpath, stat } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; +import process from 'node:process'; import { afterEach, beforeEach, describe, expect, it } from 'vitest'; @@ -32,6 +33,8 @@ const SNAPSHOT_TIMEOUT_MS = 60_000; const OUTPUT_MARKER = 'hello-structured'; const SCROLLBACK_OUTPUT_MARKER = 'line-50'; +const mouseProbe = await probeLibghosttyVt(); +const itWithMouse = mouseProbe.mouseInputAvailable === true ? it : it.skip; async function waitForOutputMarker( testHome: string, @@ -100,7 +103,7 @@ describe( sessDir = sessionDir(testHome, sessionId); rpcSocketPath = socketPath(sessDir); - }); + }, 30_000); afterEach(async () => { destroySession(testHome, sessionId); @@ -211,6 +214,63 @@ describe( ); }); + itWithMouse( + 'drives press, drag motion, and release through child-negotiated mouse modes', + async () => { + await restartSession( + [ + process.execPath, + '--import', + 'tsx', + 'test/fixtures/apps/mouse-report/main.ts', + ], + 'MOUSE_READY', + ); + + const actions = [ + ['press', '--button', 'left', '--row', '4', '--col', '9'], + ['move', '--row', '5', '--col', '10'], + ['release', '--button', 'left', '--row', '5', '--col', '10'], + ]; + const expectedData = [ + '\u001b[<0;10;5M', + '\u001b[<32;11;6M', + '\u001b[<0;11;6m', + ]; + + for (const action of actions) { + const result = runCli(['mouse', sessionId, ...action, '--json'], { + AGENT_TTY_HOME: testHome, + AGENT_TTY_RENDERER: 'libghostty-vt', + }); + expect(result.status).toBe(0); + expect(result.stderr).toBe(''); + expect(JSON.parse(result.stdout)).toMatchObject({ + ok: true, + command: 'mouse', + result: { reported: true }, + }); + } + + await waitForOutputMarker( + testHome, + sessionId, + `MOUSE_HEX:${Buffer.from(expectedData.at(-1) ?? '').toString('hex')}`, + ); + const mouseEvents = (await readEvents(testHome, sessionId)).filter( + (event) => event.type === 'input_mouse', + ); + expect(mouseEvents.map((event) => event.payload.dataBase64)).toEqual( + expectedData.map((data) => Buffer.from(data).toString('base64')), + ); + expect(mouseEvents.map((event) => event.payload.action)).toEqual([ + 'press', + 'move', + 'release', + ]); + }, + ); + it('defaults snapshot RPCs to structured format', async () => { const result = (await sendRpc( rpcSocketPath, diff --git a/test/integration/mouse.test.ts b/test/integration/mouse.test.ts new file mode 100644 index 00000000..9a9b005d --- /dev/null +++ b/test/integration/mouse.test.ts @@ -0,0 +1,248 @@ +import { mkdtemp, realpath } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, describe, expect, it } from 'vitest'; +import { sendRpc } from '../../src/host/rpcClient.js'; +import { sessionDir, socketPath } from '../../src/storage/sessionPaths.js'; + +import { + cleanupHome, + createSession, + readEvents, + runCli, + type SuccessEnvelope, +} from '../helpers.js'; + +let home = ''; +const hasNativeMouse = await import('@coder/libghostty-vt-node') + .then((native) => { + const terminal = native.createTerminal({ cols: 1, rows: 1 }); + try { + return 'encodeMouse' in terminal; + } finally { + terminal.dispose(); + } + }) + .catch(() => false); +afterEach(async () => { + await cleanupHome(home); + home = ''; +}); + +describe('mouse command', () => { + it.skipIf( + !hasNativeMouse && process.env.AGENT_TTY_REQUIRE_MOUSE_NATIVE !== '1', + )( + 'retains held buttons across concurrent requests while the native backend boots', + async () => { + home = await realpath( + await mkdtemp(join(tmpdir(), 'agent-tty-mouse-race-')), + ); + const session = createSession(home, [ + process.execPath, + '-e', + "process.stdin.setRawMode(true);process.stdin.resume();process.stdout.write('\\x1b[?1003h\\x1b[?1006hREADY');", + ]); + const ready = runCli( + [ + '--renderer', + 'ghostty-web', + 'wait', + session, + '--text', + 'READY', + '--timeout', + '10000', + '--json', + ], + { AGENT_TTY_HOME: home }, + ); + expect(ready.status, ready.stdout + ready.stderr).toBe(0); + const socket = socketPath(sessionDir(home, session)); + const common = { row: 2, col: 3, rendererName: 'libghostty-vt' }; + await Promise.all([ + sendRpc(socket, 'mouse', { + ...common, + action: 'press', + button: 'left', + }), + sendRpc(socket, 'mouse', { + ...common, + action: 'press', + button: 'right', + }), + ]); + await sendRpc(socket, 'mouse', { + ...common, + action: 'release', + button: 'right', + }); + await sendRpc(socket, 'mouse', { ...common, action: 'move', col: 4 }); + const events = (await readEvents(home, session)).filter( + (event) => event.type === 'input_mouse', + ); + expect(events.at(-1)?.payload.dataBase64).toBe( + Buffer.from('\x1b[<32;5;3M').toString('base64'), + ); + }, + 30_000, + ); + + it('documents pixel geometry and reports invalid input before resolving a session', () => { + const help = runCli(['mouse', '--help']); + expect(help.status).toBe(0); + expect(help.stdout).toContain('--cell-width'); + const invalid = runCli([ + 'mouse', + 'missing', + 'click', + '--col', + '1', + '--row', + '2', + '--cell-width', + '10', + '--cell-height', + '20', + '--json', + ]); + expect(JSON.parse(invalid.stdout)).toMatchObject({ + ok: false, + error: { code: 'INVALID_INPUT' }, + }); + }); + + it('rejects an unsupported renderer without booting a browser or writing input', async () => { + home = await realpath(await mkdtemp(join(tmpdir(), 'agent-tty-mouse-'))); + const session = createSession(home, ['/bin/sh', '-c', 'exec cat']); + const result = runCli( + [ + '--renderer', + 'ghostty-web', + 'mouse', + session, + 'press', + '--button', + 'left', + '--col', + '1', + '--row', + '2', + '--cell-width', + '10', + '--cell-height', + '20', + '--json', + ], + { AGENT_TTY_HOME: home }, + ); + expect(JSON.parse(result.stdout)).toMatchObject({ + ok: false, + error: { code: 'CAPABILITY_UNAVAILABLE' }, + }); + expect( + (await readEvents(home, session)).filter((e) => e.type === 'input_mouse'), + ).toEqual([]); + }); + + // CI with the released optional package still tests the capability error. + // Set this flag when consuming the mouse-encoder branch; missing native support must fail. + it + .skipIf( + !hasNativeMouse && process.env.AGENT_TTY_REQUIRE_MOUSE_NATIVE !== '1', + ) + .each([ + ['SGR', '\x1b[?1000h\x1b[?1006h', Buffer.from('\x1b[<0;223;3M')], + ['SGR-pixels', '\x1b[?1000h\x1b[?1016h', Buffer.from('\x1b[<0;2220;40M')], + ['X10', '\x1b[?1000h', Buffer.from([27, 91, 77, 32, 255, 35])], + [ + 'UTF-8', + '\x1b[?1000h\x1b[?1005h', + Buffer.from([27, 91, 77, 32, 0xc3, 0xbf, 35]), + ], + ['URXVT', '\x1b[?1000h\x1b[?1015h', Buffer.from('\x1b[32;223;3M')], + ['disabled', '', Buffer.alloc(0)], + ] as const)( + 'delivers %s bytes to a PTY and retains them for replay', + async (_name, mode, expected) => { + home = await realpath(await mkdtemp(join(tmpdir(), 'agent-tty-mouse-'))); + const fixture = `process.stdin.setRawMode(true); process.stdout.write(${JSON.stringify(mode + 'READY')}); process.stdin.on('data', b => { process.stdout.write('BYTES:' + b.toString('hex') + '\\n'); });`; + const session = createSession(home, [process.execPath, '-e', fixture], { + cols: 300, + }); + const env = { AGENT_TTY_HOME: home }; + const ready = runCli( + [ + '--renderer', + 'libghostty-vt', + 'wait', + session, + '--text', + 'READY', + '--timeout', + '10000', + '--json', + ], + env, + ); + expect(ready.status, ready.stdout + ready.stderr).toBe(0); + const result = runCli( + [ + '--renderer', + 'libghostty-vt', + 'mouse', + session, + + 'press', + '--button', + 'left', + '--col', + '222', + '--row', + '2', + ...(_name === 'SGR-pixels' + ? ['--cell-width', '10', '--cell-height', '20'] + : []), + '--json', + ], + env, + ); + expect(result.status, result.stdout + result.stderr).toBe(0); + const envelope = JSON.parse(result.stdout) as SuccessEnvelope<{ + seq: number; + bytesWritten: number; + }>; + expect(envelope.result.bytesWritten).toBe(expected.length); + if (expected.length > 0) { + const received = runCli( + [ + '--renderer', + 'libghostty-vt', + 'wait', + session, + '--text', + `BYTES:${expected.toString('hex')}`, + '--timeout', + '10000', + '--json', + ], + env, + ); + expect(received.status, received.stdout + received.stderr).toBe(0); + } + const event = (await readEvents(home, session)).find( + (e) => e.seq === envelope.result.seq, + ); + expect(event).toMatchObject({ + type: 'input_mouse', + payload: { dataBase64: expected.toString('base64') }, + }); + const snapshot = runCli( + ['--renderer', 'libghostty-vt', 'snapshot', session, '--json'], + env, + ); + expect(snapshot.status, snapshot.stdout + snapshot.stderr).toBe(0); + }, + 30_000, + ); +}); diff --git a/test/unit/batch/executor.test.ts b/test/unit/batch/executor.test.ts index a8fa2233..3ae1a457 100644 --- a/test/unit/batch/executor.test.ts +++ b/test/unit/batch/executor.test.ts @@ -15,6 +15,7 @@ type DriverCall = | { verb: 'type'; text: string } | { verb: 'paste'; text: string } | { verb: 'sendKeys'; keys: string[] } + | { verb: 'mouse'; action: string; row: number; col: number } | { verb: 'run'; command: string; noWait: boolean } | { verb: 'wait'; afterSeq: number | undefined }; @@ -74,6 +75,15 @@ function createFakeDriver(options: FakeDriverOptions = {}): FakeDriver { calls.push({ verb: 'sendKeys', keys }); return Promise.resolve(nextInputSeq()); }), + mouse: vi.fn((input): Promise => { + calls.push({ + verb: 'mouse', + action: input.action, + row: input.row, + col: input.col, + }); + return Promise.resolve(nextInputSeq()); + }), run: vi.fn((command: string, noWait: boolean): Promise => { calls.push({ verb: 'run', command, noWait }); const result = runResults.shift(); @@ -135,6 +145,28 @@ describe('executeBatch', () => { expect(waitCall).toEqual({ verb: 'wait', afterSeq: 7 }); }); + it('drives mouse steps and threads their sequence into the following wait', async () => { + const { driver, calls } = createFakeDriver({ inputSeqs: [17] }); + const result = await executeBatch({ + plan: plan([ + { mouse: { action: 'move', row: 4, col: 9 } }, + { wait: { text: 'hovered' } }, + ]), + driver, + keepGoing: false, + }); + + expect(calls).toEqual([ + { verb: 'mouse', action: 'move', row: 4, col: 9 }, + { verb: 'wait', afterSeq: 17 }, + ]); + expect(result.steps[0]).toMatchObject({ + kind: 'mouse', + status: 'completed', + seq: 17, + }); + }); + it('records the threaded seq as the wait step waitBaseline', async () => { const { driver } = createFakeDriver({ inputSeqs: [42] }); const result = await executeBatch({ @@ -574,6 +606,7 @@ describe('executeBatch', () => { type: () => Promise.reject(new TypeError('boom')), paste: () => Promise.resolve(1), sendKeys: () => Promise.resolve(1), + mouse: () => Promise.resolve(1), run: () => Promise.resolve({ accepted: true, seq: 1 }), wait: () => Promise.resolve({ matched: true, timedOut: false, capturedAtSeq: 1 }), @@ -603,6 +636,7 @@ describe('executeBatch', () => { ), paste: () => Promise.resolve(1), sendKeys: () => Promise.resolve(1), + mouse: () => Promise.resolve(1), run: () => Promise.resolve({ accepted: true, seq: 1 }), wait: () => Promise.resolve({ matched: true, timedOut: false, capturedAtSeq: 1 }), diff --git a/test/unit/batch/plan.test.ts b/test/unit/batch/plan.test.ts index c787c55a..3f2b3113 100644 --- a/test/unit/batch/plan.test.ts +++ b/test/unit/batch/plan.test.ts @@ -40,6 +40,38 @@ describe('parseBatchPlan', () => { ).toEqual([{ kind: 'sendKeys', keys: ['Escape', 'ctrl+c', 'Enter'] }]); }); + it('parses mouse steps with cell coordinates and modifiers', () => { + expect( + parse([ + { + mouse: { + action: 'press', + button: 'left', + row: 4, + col: 9, + modifiers: { ctrl: true }, + }, + }, + { mouse: { action: 'move', row: 5, col: 10 } }, + ]).steps, + ).toEqual([ + { + kind: 'mouse', + input: { + action: 'press', + button: 'left', + row: 4, + col: 9, + modifiers: { ctrl: true }, + }, + }, + { + kind: 'mouse', + input: { action: 'move', row: 5, col: 10, modifiers: {} }, + }, + ]); + }); + it('parses a run step with noWait defaulting to false (Waited Run)', () => { expect(parse([{ run: 'echo hi' }]).steps).toEqual([ { @@ -195,7 +227,7 @@ describe('parseBatchPlan', () => { expect.objectContaining({ code: ERROR_CODES.INVALID_INPUT, message: - 'Batch step 0 must have exactly one of type|paste|sendKeys|run|wait; found none', + 'Batch step 0 must have exactly one of type|paste|sendKeys|mouse|run|wait; found none', details: { stepIndex: 0 }, }), ); @@ -206,7 +238,7 @@ describe('parseBatchPlan', () => { expect.objectContaining({ code: ERROR_CODES.INVALID_INPUT, message: - 'Batch step 0 must have exactly one of type|paste|sendKeys|run|wait; found type, wait', + 'Batch step 0 must have exactly one of type|paste|sendKeys|mouse|run|wait; found type, wait', details: { stepIndex: 0 }, }), ); @@ -259,6 +291,37 @@ describe('parseBatchPlan', () => { }); }); + describe('mouse validation', () => { + it('classifies invalid mouse batch steps', () => { + const invalid = [ + { mouse: { action: 'press', row: 0, col: 0 } }, + { mouse: { action: 'move', button: 'left', row: 0, col: 0 } }, + { mouse: { action: 'drag', button: 'left', row: 0, col: 0 } }, + { mouse: { action: 'press', button: 'left', row: -1, col: 0 } }, + { + mouse: { + action: 'press', + button: 'left', + row: 0, + col: 0, + extra: true, + }, + }, + ]; + + expect( + invalid.map((step) => { + try { + parse([step]); + return false; + } catch { + return true; + } + }), + ).toEqual(invalid.map(() => true)); + }); + }); + describe('wait condition validation', () => { it('rejects a wait with text and regex together', () => { expect(() => parse([{ wait: { text: 'a', regex: 'b' } }])).toThrow( diff --git a/test/unit/cli/exitCodes.test.ts b/test/unit/cli/exitCodes.test.ts index 55b6b580..5331ccb5 100644 --- a/test/unit/cli/exitCodes.test.ts +++ b/test/unit/cli/exitCodes.test.ts @@ -33,6 +33,7 @@ describe('CLI exit codes', () => { expect(exitCodeForError(ERROR_CODES.PROTOCOL_ERROR)).toBe(9); expect(exitCodeForError(ERROR_CODES.RPC_ERROR)).toBe(9); expect(exitCodeForError(ERROR_CODES.REPLAY_ERROR)).toBe(10); + expect(exitCodeForError(ERROR_CODES.CAPABILITY_UNAVAILABLE)).toBe(12); }); it('maps a render-wait timeout to a distinct exit code 11', () => { diff --git a/test/unit/commands/doctor.test.ts b/test/unit/commands/doctor.test.ts index 893c9595..29fbf289 100644 --- a/test/unit/commands/doctor.test.ts +++ b/test/unit/commands/doctor.test.ts @@ -129,10 +129,11 @@ describe('doctor command', () => { expect(result.ok).toBe(true); expect(result.checks.environment).toHaveLength(9); expect(result.checks.renderer).toHaveLength(6); - expect(result.capabilities).toHaveLength(6); + expect(result.capabilities).toHaveLength(7); expect(result.capabilities.map((capability) => capability.name)).toEqual([ 'snapshot', 'wait', + 'mouse-input', 'screenshot', 'record-export-asciicast', 'record-export-webm', diff --git a/test/unit/commands/version.test.ts b/test/unit/commands/version.test.ts index 8fc3f01b..f9b1f65c 100644 --- a/test/unit/commands/version.test.ts +++ b/test/unit/commands/version.test.ts @@ -18,7 +18,7 @@ describe('version command', () => { const result = await buildVersionResult(); expect(result.cliVersion).toMatch(SEMVER_WITH_OPTIONAL_PRERELEASE); - expect(result.protocolVersion).toBe('0.1.0'); + expect(result.protocolVersion).toBe('0.2.0'); expect(result.rendererBackends).toEqual(['ghostty-web', 'libghostty-vt']); expect(result.runtime.node).toMatch(/^v\d+\.\d+\.\d+$/); expect('capabilities' in result).toBe(false); @@ -27,10 +27,11 @@ describe('version command', () => { it('builds the version result with runtime capabilities when requested', async () => { const result = await buildVersionResult({ includeCapabilities: true }); - expect(result.capabilities).toHaveLength(6); + expect(result.capabilities).toHaveLength(7); expect(result.capabilities?.map((capability) => capability.name)).toEqual([ 'snapshot', 'wait', + 'mouse-input', 'screenshot', 'record-export-asciicast', 'record-export-webm', @@ -55,7 +56,7 @@ describe('version command', () => { const result = await buildVersionResult({ includeCapabilities: true }); expect(result.cliVersion).toMatch(SEMVER_WITH_OPTIONAL_PRERELEASE); - expect(result.protocolVersion).toBe('0.1.0'); + expect(result.protocolVersion).toBe('0.2.0'); expect(result.rendererBackends).toEqual(['ghostty-web', 'libghostty-vt']); expect(result.runtime.node).toMatch(/^v\d+\.\d+\.\d+$/); expect(result.runtime.platform).toBe(process.platform); diff --git a/test/unit/host/eventLog.test.ts b/test/unit/host/eventLog.test.ts index aba578c6..f200a953 100644 --- a/test/unit/host/eventLog.test.ts +++ b/test/unit/host/eventLog.test.ts @@ -224,6 +224,54 @@ describe('EventLog', () => { } }); + it('appends input_mouse events and round-trips their semantic and wire data', async () => { + const eventLog = await EventLog.open(eventLogPath); + + try { + const seq = await eventLog.append('input_mouse', { + action: 'move', + row: 3, + col: 8, + modifiers: { shift: false, alt: false, ctrl: true }, + rendererBackend: 'libghostty-vt', + dataBase64: Buffer.from('\u001b[<48;9;4M').toString('base64'), + }); + + expect(seq).toBe(0); + expect(await eventLog.readAll()).toEqual([ + expect.objectContaining({ + seq: 0, + type: 'input_mouse', + payload: { + action: 'move', + row: 3, + col: 8, + modifiers: { shift: false, alt: false, ctrl: true }, + rendererBackend: 'libghostty-vt', + dataBase64: Buffer.from('\u001b[<48;9;4M').toString('base64'), + }, + }), + ]); + } finally { + await eventLog.close(); + } + + const reopenedEventLog = await EventLog.open(eventLogPath); + try { + expect(reopenedEventLog.getEvents()).toEqual([ + expect.objectContaining({ + seq: 0, + type: 'input_mouse', + payload: expect.objectContaining({ + dataBase64: Buffer.from('\u001b[<48;9;4M').toString('base64'), + }), + }), + ]); + } finally { + await reopenedEventLog.close(); + } + }); + it('returns buffered events without rereading the log file', async () => { const eventLog = await EventLog.open(eventLogPath); diff --git a/test/unit/host/mouseInput.test.ts b/test/unit/host/mouseInput.test.ts new file mode 100644 index 00000000..f09c3615 --- /dev/null +++ b/test/unit/host/mouseInput.test.ts @@ -0,0 +1,90 @@ +import { describe, expect, it } from 'vitest'; + +import { prepareHeldMouseButtons } from '../../../src/host/mouseInput.js'; + +describe('prepareHeldMouseButtons', () => { + it('classifies held-button transitions across press, wheel, move, and release sets', () => { + const none = new Set<'left' | 'middle' | 'right'>(); + const pressed = prepareHeldMouseButtons(none, { + action: 'press', + button: 'left', + }); + const wheelWhileHeld = prepareHeldMouseButtons(pressed.nextHeld, { + action: 'press', + button: 'wheel-up', + }); + const moved = prepareHeldMouseButtons(wheelWhileHeld.nextHeld, { + action: 'move', + }); + const released = prepareHeldMouseButtons(moved.nextHeld, { + action: 'release', + button: 'left', + }); + const wheelAlone = prepareHeldMouseButtons(released.nextHeld, { + action: 'press', + button: 'wheel-down', + }); + + expect([ + [...pressed.nextHeld], + [...wheelWhileHeld.nextHeld], + [...moved.nextHeld], + [...released.nextHeld], + [...wheelAlone.nextHeld], + ]).toEqual([['left'], ['left'], ['left'], [], []]); + expect([ + pressed.anyButtonPressed, + wheelWhileHeld.anyButtonPressed, + moved.anyButtonPressed, + released.anyButtonPressed, + wheelAlone.anyButtonPressed, + ]).toEqual([true, true, true, false, false]); + expect([ + pressed.dragButton, + wheelWhileHeld.dragButton, + moved.dragButton, + released.dragButton, + wheelAlone.dragButton, + ]).toEqual(['left', 'left', 'left', undefined, undefined]); + expect([...none]).toEqual([]); + }); + + it('uses the most recently pressed held button for multi-button drag reports', () => { + const left = prepareHeldMouseButtons(new Set(), { + action: 'press', + button: 'left', + }); + const right = prepareHeldMouseButtons(left.nextHeld, { + action: 'press', + button: 'right', + }); + const leftAgain = prepareHeldMouseButtons(right.nextHeld, { + action: 'press', + button: 'left', + }); + + expect([...leftAgain.nextHeld]).toEqual(['right', 'left']); + expect(leftAgain.dragButton).toBe('left'); + }); + + it('classifies invalid transition inputs', () => { + const invalid = [ + { action: 'press' }, + { action: 'release' }, + { action: 'move', button: 'left' }, + { action: 'drag', button: 'left' }, + { action: 'press', button: 'primary' }, + ]; + + expect( + invalid.map((input) => { + try { + prepareHeldMouseButtons(new Set(), input as never); + return false; + } catch { + return true; + } + }), + ).toEqual(invalid.map(() => true)); + }); +}); diff --git a/test/unit/protocol/messages.test.ts b/test/unit/protocol/messages.test.ts index f79dc090..e006f84a 100644 --- a/test/unit/protocol/messages.test.ts +++ b/test/unit/protocol/messages.test.ts @@ -9,6 +9,8 @@ import { HostInspectResultSchema, InspectResultSchema, MarkParamsSchema, + MouseParamsSchema, + MouseResultSchema, RendererRuntimeSummarySchema, MarkResultSchema, PasteParamsSchema, @@ -35,6 +37,7 @@ import { } from '../../../src/protocol/messages.js'; import { EventRecordSchema, + InputMouseEventRecordSchema, MarkerEventRecordSchema, RunCompleteEventPayloadSchema, RunCompleteEventRecordSchema, @@ -220,6 +223,7 @@ describe('CapabilityEntrySchema', () => { const names = [ 'snapshot', 'wait', + 'mouse-input', 'screenshot', 'record-export-asciicast', 'record-export-webm', @@ -1133,6 +1137,92 @@ describe('RPC message schemas', () => { }); }); + it('classifies valid mouse parameter sets', () => { + const validParams = [ + { action: 'press', button: 'left', row: 0, col: 0 }, + { action: 'release', button: 'right', row: 23, col: 79 }, + { action: 'move', row: 2, col: 3 }, + { + action: 'move', + row: 4, + col: 5, + modifiers: { shift: true, alt: false, ctrl: true }, + rendererName: 'libghostty-vt', + }, + { action: 'press', button: 'wheel-up', row: 6, col: 7 }, + { action: 'press', button: 'wheel-down', row: 6, col: 7 }, + { action: 'press', button: 'wheel-left', row: 6, col: 7 }, + { action: 'press', button: 'wheel-right', row: 6, col: 7 }, + ]; + + expect( + validParams.map((params) => MouseParamsSchema.safeParse(params).success), + ).toEqual(validParams.map(() => true)); + }); + + it('classifies invalid mouse parameter sets', () => { + const invalidParams = [ + { action: 'press', row: 0, col: 0 }, + { action: 'release', row: 0, col: 0 }, + { action: 'move', button: 'left', row: 0, col: 0 }, + { action: 'drag', button: 'left', row: 0, col: 0 }, + { action: 'press', button: 'primary', row: 0, col: 0 }, + { action: 'release', button: 'wheel-up', row: 0, col: 0 }, + { action: 'press', button: 'left', row: -1, col: 0 }, + { action: 'press', button: 'left', row: 0, col: -1 }, + { action: 'press', button: 'left', row: 1.5, col: 0 }, + { + action: 'press', + button: 'left', + row: 0, + col: 0, + modifiers: { meta: true }, + }, + { action: 'press', button: 'left', row: 0, col: 0, extra: true }, + ]; + + expect( + invalidParams.map( + (params) => MouseParamsSchema.safeParse(params).success, + ), + ).toEqual(invalidParams.map(() => false)); + }); + + it('accepts mouse results with reporting state, byte count, and seq', () => { + expect( + MouseResultSchema.parse({ reported: true, bytesWritten: 10, seq: 42 }), + ).toEqual({ reported: true, bytesWritten: 10, seq: 42 }); + expect( + MouseResultSchema.parse({ reported: false, bytesWritten: 0, seq: 43 }), + ).toEqual({ reported: false, bytesWritten: 0, seq: 43 }); + }); + + it('strictly validates input_mouse event records', () => { + const record = { + seq: 7, + ts: '2026-03-19T12:00:02.000Z', + type: 'input_mouse', + payload: { + action: 'press', + button: 'left', + row: 4, + col: 9, + modifiers: { shift: true, alt: false, ctrl: false }, + rendererBackend: 'libghostty-vt', + dataBase64: Buffer.from('\u001b[<0;10;5M').toString('base64'), + }, + } as const; + + expect(InputMouseEventRecordSchema.parse(record)).toEqual(record); + expect(EventRecordSchema.parse(record)).toEqual(record); + expect( + InputMouseEventRecordSchema.safeParse({ + ...record, + payload: { ...record.payload, reported: true }, + }).success, + ).toBe(false); + }); + it('rejects empty type text', () => { const result = TypeParamsSchema.safeParse({ text: '', @@ -1210,6 +1300,7 @@ describe('RPC message schemas', () => { 'run', 'mark', 'sendKeys', + 'mouse', 'resize', 'signal', 'wait', diff --git a/test/unit/renderer/capabilities.test.ts b/test/unit/renderer/capabilities.test.ts index c63e3516..7bd5c6a8 100644 --- a/test/unit/renderer/capabilities.test.ts +++ b/test/unit/renderer/capabilities.test.ts @@ -10,7 +10,7 @@ function getCapability( } describe('discoverCapabilities', () => { - it('returns six quick capabilities without browser-launch details', async () => { + it('returns seven quick capabilities without browser-launch details', async () => { const probePlaywright = vi.fn(() => Promise.resolve({ available: true })); const probeLibghosttyVt = vi.fn(() => Promise.resolve({ available: true })); @@ -21,10 +21,11 @@ describe('discoverCapabilities', () => { expect(probePlaywright).toHaveBeenCalledTimes(1); expect(probeLibghosttyVt).toHaveBeenCalledTimes(1); - expect(capabilities).toHaveLength(6); + expect(capabilities).toHaveLength(7); expect(capabilities.map((capability) => capability.name)).toEqual([ 'snapshot', 'wait', + 'mouse-input', 'screenshot', 'record-export-asciicast', 'record-export-webm', @@ -38,6 +39,10 @@ describe('discoverCapabilities', () => { name: 'wait', status: 'available', }); + expect(getCapability(capabilities, 'mouse-input')).toEqual({ + name: 'mouse-input', + status: 'available', + }); expect(getCapability(capabilities, 'screenshot')).toEqual({ name: 'screenshot', status: 'available', @@ -73,6 +78,34 @@ describe('discoverCapabilities', () => { reason: 'libghostty-vt not installed', detail: 'Cannot find package @coder/libghostty-vt-node', }); + expect(getCapability(capabilities, 'mouse-input')).toEqual({ + name: 'mouse-input', + status: 'unavailable', + reason: 'libghostty-vt not installed', + detail: 'Cannot find package @coder/libghostty-vt-node', + }); + }); + + it('keeps native rendering available while rejecting an old binding for mouse input', async () => { + const capabilities = await discoverCapabilities('quick', { + probePlaywright: () => Promise.resolve({ available: true }), + probeLibghosttyVt: () => + Promise.resolve({ + available: true, + mouseInputAvailable: false, + reason: 'libghostty-vt native module available', + detail: '@coder/libghostty-vt-node predates mouse encoding', + }), + }); + + expect(getCapability(capabilities, 'snapshot')?.status).toBe('available'); + expect(getCapability(capabilities, 'dashboard')?.status).toBe('available'); + expect(getCapability(capabilities, 'mouse-input')).toEqual({ + name: 'mouse-input', + status: 'unavailable', + reason: 'libghostty-vt mouse encoder unavailable', + detail: '@coder/libghostty-vt-node predates mouse encoding', + }); }); it('marks quick browser-backed capabilities unavailable when playwright is missing', async () => { diff --git a/test/unit/renderer/libghosttyVtBackend.test.ts b/test/unit/renderer/libghosttyVtBackend.test.ts index f9eeee76..22efd716 100644 --- a/test/unit/renderer/libghosttyVtBackend.test.ts +++ b/test/unit/renderer/libghosttyVtBackend.test.ts @@ -104,8 +104,16 @@ function createNativeFixture(options: { visibleText?: string } = {}) { const getVisibleText = vi.fn( () => options.visibleText ?? 'hello world\nprompt>', ); + const encodeMouse = vi.fn(() => Buffer.from('\u001b[<64;9;4M')); const dispose = vi.fn(); - const terminal = { feed, resize, snapshot, getVisibleText, dispose }; + const terminal = { + feed, + resize, + encodeMouse, + snapshot, + getVisibleText, + dispose, + }; const createTerminal = vi.fn( (createOptions: { cols: number; rows: number }) => { cols = createOptions.cols; @@ -131,6 +139,7 @@ function createNativeFixture(options: { visibleText?: string } = {}) { resize, snapshot, getVisibleText, + encodeMouse, dispose, }; } @@ -195,6 +204,127 @@ describe('LibghosttyVtBackend', () => { }); }); + it('scales cell positions using explicit pixel metrics', async () => { + const fixture = createNativeFixture(); + const backend = createBackend(fixture); + try { + await backend.boot(); + await backend.replayTo(createReplayInput()); + backend.encodeMouse({ + action: 'press', + button: 'left', + row: 3, + col: 8, + cellWidth: 10, + cellHeight: 20, + modifiers: { shift: false, alt: false, ctrl: false }, + anyButtonPressed: true, + }); + expect(fixture.encodeMouse).toHaveBeenCalledWith( + expect.objectContaining({ x: 80, y: 60 }), + expect.objectContaining({ + geometry: { + screenWidth: 120, + screenHeight: 100, + cellWidth: 10, + cellHeight: 20, + }, + }), + ); + } finally { + await backend.dispose(); + } + }); + + it('maps semantic mouse input to native cell geometry and button names', async () => { + const fixture = createNativeFixture(); + const backend = createBackend(fixture); + + await backend.boot(); + await backend.replayTo(createReplayInput()); + + const bytes = backend.encodeMouse({ + action: 'press', + button: 'wheel-up', + row: 3, + col: 8, + modifiers: { shift: true, alt: false, ctrl: false }, + anyButtonPressed: false, + }); + + expect(bytes).toEqual(Buffer.from('\u001b[<64;9;4M')); + expect(fixture.encodeMouse).toHaveBeenCalledWith( + { + action: 'press', + button: 'four', + x: 8, + y: 3, + modifiers: { shift: true, alt: false, ctrl: false }, + }, + { + geometry: { + screenWidth: 12, + screenHeight: 5, + cellWidth: 1, + cellHeight: 1, + }, + anyButtonPressed: false, + trackLastCell: true, + }, + ); + }); + + it('maps move input to native motion while preserving held-button state', async () => { + const fixture = createNativeFixture(); + const backend = createBackend(fixture); + + await backend.boot(); + await backend.replayTo(createReplayInput()); + backend.encodeMouse({ + action: 'move', + button: 'left', + row: 1, + col: 2, + modifiers: { shift: false, alt: true, ctrl: false }, + anyButtonPressed: true, + }); + + expect(fixture.encodeMouse).toHaveBeenCalledWith( + expect.objectContaining({ + action: 'motion', + button: 'left', + x: 2, + y: 1, + modifiers: { shift: false, alt: true, ctrl: false }, + }), + expect.objectContaining({ anyButtonPressed: true, trackLastCell: true }), + ); + }); + + it('preserves rendering but rejects mouse input when the binding predates mouse encoding', async () => { + const fixture = createNativeFixture(); + const oldTerminal = { ...fixture.terminal } as Record; + delete oldTerminal.encodeMouse; + fixture.createTerminal.mockReturnValueOnce(oldTerminal as never); + const backend = createBackend(fixture); + + await backend.boot(); + await backend.replayTo(createReplayInput()); + + expect(() => + backend.encodeMouse({ + action: 'press', + button: 'left', + row: 0, + col: 0, + modifiers: { shift: false, alt: false, ctrl: false }, + anyButtonPressed: true, + }), + ).toThrow( + 'terminal.encodeMouse is required; update @coder/libghostty-vt-node to a mouse-capable release', + ); + }); + it('skips run_complete events during replay', async () => { const fixture = createNativeFixture(); const backend = createBackend(fixture); diff --git a/test/unit/renderer/readiness.test.ts b/test/unit/renderer/readiness.test.ts index 705afb53..4e6c70e6 100644 --- a/test/unit/renderer/readiness.test.ts +++ b/test/unit/renderer/readiness.test.ts @@ -10,11 +10,25 @@ import { describe('probeLibghosttyVt', () => { it('reports available when the native module exposes createTerminal', async () => { + const probe = await probeLibghosttyVt(() => + Promise.resolve({ + createTerminal: () => ({}), + supportsMouseInput: true, + }), + ); + + expect(probe.available).toBe(true); + expect(probe.mouseInputAvailable).toBe(true); + }); + + it('reports an old renderer as available without mouse input', async () => { const probe = await probeLibghosttyVt(() => Promise.resolve({ createTerminal: () => ({}) }), ); expect(probe.available).toBe(true); + expect(probe.mouseInputAvailable).toBe(false); + expect(probe.detail).toContain('mouse-capable release'); }); it('reports unavailable when the loaded module is missing createTerminal', async () => {