From a0306e5ad696659d13e4a8bb79900ecc64db04a5 Mon Sep 17 00:00:00 2001 From: Taras Mankovski Date: Fri, 16 Oct 2026 02:09:08 -0400 Subject: [PATCH 1/4] =?UTF-8?q?=E2=9C=A8=20Retain=20a=20REPL=20entry's=20F?= =?UTF-8?q?ile=20and=20Dir=20work=20in=20an=20ambient=20checkout=20(#181,?= =?UTF-8?q?=201/N)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Start `xmd repl` inside a Git checkout and what an entry does to files there is retained: the `` reads and writes, the `` deletions and the directories `` ensures. Leave an entry at a question, come back to the printed location, and the work it already did is not done again — the question is offered again, the binding a completed read produced is still the text it read, and the write after the question happens for the first time when you answer. A *new* entry reading that same path reads what is there now. Only live new work touches the environment. Answering a completed operation performs no path canonicalization, no ambient discovery, no lease acquisition, no permission lookup and no filesystem call, so a continuation works in a checkout somebody has since changed or removed. Nothing is inventoried, predicted or compared: the bounded comparison beside this change shows the cost unchanged between a checkout holding nothing else and one holding a thousand unrelated files. Rebased onto the landed U1–U5, which moved three things this feature used to carry itself. Core owns ``, `` and ``, so recording is ordinary ancestor-max `Files` middleware over the one document filesystem and a generated fragment's admitted operations are retained exactly as an authored one's are — there is no separate captured-access door any more. `Execution.init` owns lifetime, so the recording is established inside the scope the invocation already owns, once per executing entry, and a passive view initializes nothing. Generated selection is the landed names-and-forms one, untouched here. Under the accepted C1 composition the host's own selection is one CLI-private Context whose default is the provider-neutral Git Plugin. Deno and the compiled binary bind a configured native variant before any command is dispatched; the shared command path reads the value and imports no adapter; Node and Bun leave the default and retain nothing. The native variant delegates the neutral Plugin's whole install and adds ordinary initialization for command `run`, while the selected REPL host registers the recorder for an entry. A command captures that inert value into its child-host settings before any document code runs, so a test root — which activates no bundled Plugin at all — still hands its `host="run"` child the configured one, and a document that replaces the Context afterwards does not get to choose its children's host. The repository installer arguments, `repl-repositories.ts` and the installation-array transform are gone. A record's name is the digest of the record version, the operation kind and the complete request as the document wrote it. Equal work in two entries is one name, which is allowed: each entry executes against its own stream segment, and that segment rather than a label inside the record is what decides whose history an entry may continue. The history admission that reads these records travels with the Git Plugin's existing contribution for the command. A shared Git directory still admits one live entry at a time through one kernel lock on a hashed sidecar outside every checkout, taken by the first new mutation and held until the entry is torn down. A completed read acquires nothing: an observation is not a new exclusive writer. --- .github/workflows/ci.yml | 9 + architecture.md | 117 ++ packages/cli/src/bun.ts | 2 - packages/cli/src/cli.ts | 64 +- packages/cli/src/compiled-repl.ts | 8 + packages/cli/src/compiled.ts | 7 +- packages/cli/src/deno-repl.ts | 8 + packages/cli/src/deno-repositories.ts | 112 +- packages/cli/src/deno.ts | 7 +- packages/cli/src/node.ts | 2 - packages/cli/src/repl-profile.ts | 18 +- packages/cli/src/run-profile.ts | 59 +- packages/cli/src/run-repositories.ts | 37 +- packages/cli/src/testing-host.ts | 29 +- packages/cli/tests/compiled-upgrade.test.ts | 10 +- packages/cli/tests/repl-boundaries.test.ts | 2 - .../cli/tests/repl-repositories-pty.test.ts | 45 + packages/cli/tests/repl-repositories.test.ts | 568 +++++++++ packages/cli/tests/run-composition.test.ts | 3 +- packages/cli/tests/run-deadline.test.ts | 2 - packages/cli/tests/run-profile.test.ts | 81 +- packages/cli/tests/stdin-cli.test.ts | 2 - packages/cli/tests/support/pty.ts | 468 ++++++++ .../cli/tests/support/repl-files-journey.ts | 328 ++++++ .../cli/tests/support/run-markdown-tier.ts | 4 +- packages/cli/tests/targets-cli.test.ts | 3 - .../cli/tests/testing-execution-host.test.ts | 200 +++- packages/git/deno.ts | 2 + .../git/src/deno/retained-repository/files.ts | 417 +++++++ .../src/deno/retained-repository/leases.ts | 64 + .../git/src/deno/retained-repository/local.ts | 288 +++++ .../src/deno/retained-repository/provider.ts | 140 +++ .../git/src/deno/run-composition/ambient.ts | 42 +- packages/git/src/deno/run-composition/lazy.ts | 58 + .../git/src/deno/run-composition/placement.ts | 22 + .../git/src/deno/run-composition/provider.ts | 57 +- packages/git/src/plugin.ts | 33 +- .../git/src/retained-repository/admission.ts | 161 +++ .../git/src/retained-repository/records.ts | 430 +++++++ packages/git/tests/module-partition.test.ts | 7 + packages/git/tests/public-entrypoint.test.ts | 8 +- .../retained-repository-lifecycle.test.ts | 703 +++++++++++ .../tests/retained-repository-local.test.ts | 1027 +++++++++++++++++ .../tests/retained-repository-records.test.ts | 376 ++++++ packages/workflow/src/deno/advisory-lock.ts | 25 +- scripts/bench-repl-file-dir.ts | 432 +++++++ scripts/runtime-test-exclusions.ts | 24 + .../tests/repl-repositories-compiled.test.ts | 40 + scripts/tests/runtime-exclusions.test.ts | 1 + specs/executable-mdx-spec.md | 31 + specs/repl-spec.md | 82 +- specs/workflow-workspace-spec.md | 35 +- 52 files changed, 6458 insertions(+), 242 deletions(-) create mode 100644 packages/cli/tests/repl-repositories-pty.test.ts create mode 100644 packages/cli/tests/repl-repositories.test.ts create mode 100644 packages/cli/tests/support/pty.ts create mode 100644 packages/cli/tests/support/repl-files-journey.ts create mode 100644 packages/git/src/deno/retained-repository/files.ts create mode 100644 packages/git/src/deno/retained-repository/leases.ts create mode 100644 packages/git/src/deno/retained-repository/local.ts create mode 100644 packages/git/src/deno/retained-repository/provider.ts create mode 100644 packages/git/src/deno/run-composition/lazy.ts create mode 100644 packages/git/src/retained-repository/admission.ts create mode 100644 packages/git/src/retained-repository/records.ts create mode 100644 packages/git/tests/retained-repository-lifecycle.test.ts create mode 100644 packages/git/tests/retained-repository-local.test.ts create mode 100644 packages/git/tests/retained-repository-records.test.ts create mode 100644 scripts/bench-repl-file-dir.ts create mode 100644 scripts/tests/repl-repositories-compiled.test.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 49a15f114..20546f855 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -228,6 +228,15 @@ jobs: - name: The Component the compiled binary carries run: deno test --allow-all --frozen scripts/tests/plan-component-compiled.test.ts + # The entry host a REPL entry's file work is retained by is assembled by + # `compiled.ts`, which is a different entrypoint from the one the source + # row runs. A binary built without it starts a REPL, performs the work and + # retains none of it, so the defect would appear at a person's first + # reopened location. This drives the same journey the source row drives, + # on a real terminal, against the binary this job just built. + - name: A REPL entry's files across the compiled binary + run: deno test --allow-all --frozen scripts/tests/repl-repositories-compiled.test.ts + # Each package's `components.md` is a packaged asset on the same footing, # and the long-form documentation is the only thing that reports whether # one shipped — a binary missing them still lists every component and diff --git a/architecture.md b/architecture.md index e3fdfeb8b..99ed93a62 100644 --- a/architecture.md +++ b/architecture.md @@ -4314,6 +4314,30 @@ callbacks, documentation capture and public execution policy. Setup receives no installation array, request, entry identifier or receipt and creates no second execution owner. +Which middleware a command installs on that seam is the host's own selection, +made once by the runtime entrypoint. The bundled Plugin a run-profile command +begins with is read from one CLI-private Context whose default is the +provider-neutral value: Deno and the compiled binary bind a configured native +variant before any command is dispatched, Node and Bun leave the default, and +the shared command path reads the value and imports no adapter. What the Context +selects is inert — code and configuration, with no provider, lease, session, +recorder or identity in it — so nothing read from it is authority over +permission, durable identity or reconciliation. The native variant delegates the +neutral Plugin's whole install, declarations and admissions included, and adds +ordinary initialization middleware for the command whose documents are ordinary +runs. + +An isolated `host="run"` child is an ordinary run whatever command hosts it, so +the command captures that selected value into the child-host settings before any +document code runs — including for a test root, which activates no bundled +Plugin at all and has no operational provider of its own. The child installs it +again in its own scope, after its working directory is established, and core +initializes it there. A document that replaces the Context afterwards is +describing its own scope: the child's host was already chosen. The reserved +`--plugin git` selector still names that selected value without reaching a +loader, the value is still removed from the outer list by identity alone, and a +different Plugin using the same name still meets the duplicate-name refusal. + Initialization failure prevents policy, root import and body. Failure and cancellation join partially acquired teardown before the invocation settles. Ordinary descendants inherit initialized state; concurrent executions and REPL @@ -6417,6 +6441,98 @@ finished, refused, failed, was cancelled or reached end of input, stops the reader, removes every listener, restores the modes and writes the final reset exactly once. +### What one entry retains about the checkout it is standing in + +The REPL is foreground experimentation, and what that means for files is one +sentence: a completed operation answers again, and a new one executes live. An +entry that read a file, wrote three and stopped at a question is reopened at the +printed location, those four answer from what they produced, and the write after +the question happens for the first time. + +Four operations are retained and no others: a `` text read, a `` +write, a ``, and the directory a `` ensures, each in the +ambient checkout the entry was started in. A read is retained because a document +binds its text and a continuation has to offer the same binding; a glob, a path +check and a temporary directory bind nothing a continuation must answer again, +and are left alone. A call outside that checkout is ordinary filesystem +behavior, recorded by nothing — a record would place the call in a checkout it +is not in — so it runs again during a partial continuation. + +Only live new work needs the environment. Answering a completed operation +performs no path canonicalization, no ambient discovery, no lease acquisition, +no permission lookup and no filesystem call, which is why a continuation works +in a checkout somebody has since changed, moved or deleted: the question's +prompt still shows the text the read bound, while the file says something else +and a *new* entry reading the same path sees what is there now. Nothing is +inventoried, predicted or compared, and the person reading the screen is what +resolves a disagreement. Replay authenticates no checkout, and restoring where +an entry was working is not permission to work there. + +What does the retaining is established by core initialization, once per +executing entry, inside the scope the invocation already owns — registered by +the selected REPL host rather than by the ordinary run Plugin, because an entry +installs no managed root, credential helper or Git host. Everything it acquires +belongs to that execution: the Git session, its isolation directory, the ambient +discovery and the lease all end when the entry does. A passive view — a settled +history, a frozen position, a refused route — initializes nothing, discovers no +checkout and takes no lease. + +Recording is ordinary ancestor-max `Files` middleware over the one document +filesystem, so what is kept is the result the whole composition answered with, +and a generated fragment's admitted operations are retained exactly as an +authored one's are: they cross the same route. A provider beneath that answers +directly, or transforms the path or content and delegates, is provider behavior; +the request as it arrived stays the operation's identity. Terminal permission +applies to live work, after transformations, and a completed result is answered +before any of it. + +A record's name is the digest of the record version, the operation kind and the +complete request as the document wrote it. Nothing is canonicalized for naming, +because a path resolved through the filesystem is a fact about the host at that +moment and the question the document asked is not. Two equal operations in one +entry are one name at two ordered positions, and telling them apart is the +durable engine's own positional matching rather than a name search. Equal work +in two entries is equally one name, and nothing in a record says which entry +wrote it: each entry executes against its own segment of the one stream, so that +segment — not a label inside the record — is what decides whose history an entry +may continue. + +Two failures are kept apart deliberately. A valid composed success or ordinary +refusal is appended and returned as the same semantic result. A provider that +broke its contract, or whose answer this boundary cannot read, keeps the Files +infrastructure classification, activates the durability fence and is re-raised +with its identity intact — nothing is written down for it, because a provider +that cannot describe what it did may not have done it. An append that fails +after the work happened stays a fatal durability failure rather than an +acknowledged success. Either side of the acknowledgement is a real outcome: work +acknowledged before the interruption is answered from its record and an external +edit survives, while work performed and not acknowledged is attempted again and +may overwrite one. That window is what the REPL trades for not snapshotting +anybody's working tree. + +A shared Git directory admits one live entry at a time, and one mechanism says +so: a kernel-released exclusive lock on a hashed sidecar outside every checkout. +`flock` and `LockFileEx` both exclude *independently opened* handles, two in one +process included, so a second entry here is refused by the operating system +exactly as a second process is — the native descriptor is the whole live +resource, and there is no table of holders, no acquisition token and no state a +caller could substitute. It is taken through the execution's own lazy +single-flight acquisition, so one entry holds one handle however many mutations +ask for it, with the release registered before the descriptor is opened and the +hold lasting until the entry is torn down rather than until an element returns. +A completed read acquires nothing: an observation is not a new exclusive writer. +It is not exclusion against arbitrary Git tools, a person editing the checkout, +or any process that does not participate in this sidecar. An entry that cannot +have the repository it needs is told so in a sentence, before anything is read, +and that refusal is never saved as a completed result: a lease is busy only +while somebody else is working, and recording "refused" would make a temporary +condition the permanent answer every later continuation replays. + +There is no managed checkout here, no `` or `` selection, +no local Git operation and nothing external: those components stay declared +where they already are and refuse through their own unchanged missing-provider +defaults. + ### What the screen teaches, and what leaving it costs Presentation guidance, visual emphasis and the contextual status a person reads @@ -6791,6 +6907,7 @@ Status is measured against main. | `GitQuery.resolve()` | verifies and resolves one Git revision expression contextually [Interface and evidence](specs/workflow-spec.md#7-the-git-capability--moved). | built on main | | workflow run storage | creates or compatibly finds one run by public run ID, retains its identity, state, document executions and filtered journal, and validates immutable Workspace roots through one provider-owned connection entry [Invariant](#workflow-run-storage); [Interface and evidence](specs/workflow-spec.md#9-retained-run-storage). | built on the #365 stack; the CLI lifecycle reaches it on the #366 stack | | caller-owned storage transaction | publishes several changes, including journal events, in one transaction nothing else enlists in [Invariant](#one-database-at-a-time-and-a-transaction-a-caller-can-hold). | built on main | +| REPL entry retained file work | one executing REPL entry's own recording, established by core initialization in the scope the invocation owns and ending with it: it retains the `` reads, `` writes, `` deletions and `` ensures that land inside the ambient checkout the entry was started in, authored and admitted-generated alike, because both cross the one document filesystem. A completed operation answers from its saved result without live path canonicalization, ambient discovery, a lease, a permission lookup or a filesystem call — so only *new* work needs the environment, a continuation still binds the text its read answered with while the file says something else, and a new entry reading that path sees what is there now. A record's name is the digest of the record version, the operation kind and the complete received request, so two equal requests in one entry are one name at two ordered positions and the durable engine's positional matching tells them apart; equal work in two entries is the same name, and each entry executing against its own stream segment is what keeps those apart. Recording is ordinary ancestor-max `Files` middleware, so a provider beneath that answers directly or transforms and delegates is kept as provider behavior while the received request stays the operation's identity. A shared Git directory admits one live entry at a time, held lazily for as long as the execution lasts and refused rather than queued; a completed read acquires nothing. There is no managed checkout, no `` or `` selection, no local Git operation and nothing external here; work outside the checkout stays ordinary and unrecorded, and a passive reading initializes nothing [Reference](#what-one-entry-retains-about-the-checkout-it-is-standing-in). | built on the #181 stack | | live durable-operation coordinator | explicitly coordinates structured live execution with existing Yield publication while leaving replay and callback effects unchanged [Invariant](#live-effect-coordination). | built on the #365 stack | | Workspace coordination API | fails closed by default; replaceable context routes only a one-use provider selection, while the selected provider directly invokes an execution-owned credentialed capability for execution, publication and failure activation [Invariant](#live-effect-coordination). | built on the #365 stack; the Deno provider installs an adapter-private atomic handler | | shared Git-host reconciliation | Reconciles Git push and pull-request upsert through one request-only boundary whose private terminal alone settles the retained outcome. [Invariant](#external-effects-and-git-host-mutations); [Invariant](#issue-effects). | Built on #297; Push (#370) and PullRequest (#295) consume it. Issue (#296) has its own middleware boundary. | diff --git a/packages/cli/src/bun.ts b/packages/cli/src/bun.ts index 886c6b69a..e558e9f4d 100644 --- a/packages/cli/src/bun.ts +++ b/packages/cli/src/bun.ts @@ -17,7 +17,6 @@ import { importPluginModule } from "./host-plugin-modules.ts"; import type { UpgradeAssembly } from "./upgrade.ts"; import { unassembledMachineSessions } from "./session-coordinator.ts"; import { unsupportedWorkflowHost } from "./workflow.ts"; -import { unsupportedRepositories } from "./run-repositories.ts"; import { useBunService } from "./bun-service.ts"; const ENTRYPOINT = fileURLToPath(import.meta.url); @@ -72,7 +71,6 @@ await main(function* (args) { args, useBunService, UPGRADE, - unsupportedRepositories, () => readInputStream(process.stdin), importPluginModule, unsupportedWorkflowHost, diff --git a/packages/cli/src/cli.ts b/packages/cli/src/cli.ts index 9c8ce4a3d..1379e5383 100755 --- a/packages/cli/src/cli.ts +++ b/packages/cli/src/cli.ts @@ -146,7 +146,7 @@ import { syntaxSymbols, useCommandSyntax, } from "./syntax.ts"; -import { assembleRunProfile } from "./run-profile.ts"; +import { assembleRunProfile, BundledRunPlugin } from "./run-profile.ts"; import type { PluginModuleLoader } from "./plugin-loader.ts"; import { installPlugins, NO_PLUGINS } from "./plugin-host.ts"; import type { CommandPlugins } from "./plugin-host.ts"; @@ -155,8 +155,6 @@ import type { PluginSelection } from "./plugin-selection.ts"; import { deliverWhole } from "./stdout-delivery.ts"; import { testingExecutionHost } from "./testing-host.ts"; import type { ChildPlanDeclaration } from "./testing-host.ts"; -import { unsupportedRepositories } from "./run-repositories.ts"; -import type { RepositoryInstaller } from "./run-repositories.ts"; import { EVAL_ALIAS, EVAL_OPTION, evalGrammarError, readEvalFlags } from "./eval-source.ts"; import type { EvalFlags } from "./eval-source.ts"; import { STANDARD_INPUT_FAILURE, STANDARD_INPUT_PATH } from "./standard-input.ts"; @@ -1236,8 +1234,6 @@ function* runDocument( config: DocumentConfig, mode: DocumentMode, installService: HostServiceInstaller, - installRepositories: RepositoryInstaller, - childRepositories: RepositoryInstaller, ): Operation> { const { root, include, verbose, journal, raw, secretDetection, retainProcessOutput } = config; @@ -1368,16 +1364,6 @@ function* runDocument( // the provider for a service. yield* installService(); - // Repository selection belongs to document execution too, and it is this - // execution's own: the provider it installs holds an invocation identity, the - // leases on the checkouts this document selects, and the evidence of what it - // published. `xmd run` supplies the live one; `xmd plan` executes no document - // of a caller's and reaches this line for none. - // `xmd test` and every runtime without an operational provider supply the one - // that installs nothing, and every repository operation then reports an - // absent provider before touching anything. - yield* installRepositories(); - // What a `` in this document runs a nested execution under. Captured // before document code begins, so a child is offered exactly what this // command assembled — and never a second description of it. @@ -1395,13 +1381,18 @@ function* runDocument( pluginArgs: plugins.args, secretDetection, installService, - // The *entrypoint's* installer, not this command's. A `host="run"` child is - // an ordinary run whatever command is hosting it, so `xmd test` — which - // installs no repository provider for its own document — still gives one to - // a child that asked to be a run. Passed rather than inherited because a - // child runs in an isolated scope and needs a fresh instance: its own - // invocation identity, its own leases and its own Push evidence. - installRepositories: childRepositories, + // The *entrypoint's* bundled Plugin, not this command's active list. A + // `host="run"` child is an ordinary run whatever command is hosting it, so + // `xmd test` — which activates no bundled Plugin for its own document — + // still gives one to a child that asked to be a run. Read here, before any + // document code has run, because this is the host's own selection: a + // document that replaces the Context later is describing its own scope and + // does not get to choose what its children's repository work reaches. + // + // It is inert code and configuration. The provider itself is created by + // the child's own initialization, in the child's own scope, so no live + // selection, lease, Git session or publication evidence crosses. + bundledPlugin: yield* BundledRunPlugin.expect(), testAgentWorker: yield* readWorkerCommand(), planDeclaration, }); @@ -1528,13 +1519,9 @@ function* runScopedDocument( config: DocumentConfig, mode: DocumentMode, installService: HostServiceInstaller, - installRepositories: RepositoryInstaller, - childRepositories: RepositoryInstaller = installRepositories, ): Operation> { try { - return yield* scoped(() => - runDocument(config, mode, installService, installRepositories, childRepositories), - ); + return yield* scoped(() => runDocument(config, mode, installService)); } catch (error) { return Err(error instanceof Error ? error : new Error(String(error))); } @@ -1590,8 +1577,6 @@ function* test( config: TestConfig, args: string[], installService: HostServiceInstaller, - /** What a `` child installs. This command installs none. */ - installRepositories: RepositoryInstaller, /** What this command's Plugins installed, and the values a child reinstalls. */ plugins: CommandPlugins, ): Operation { @@ -1630,9 +1615,7 @@ function* test( // The outer `xmd test` command installs no operational repository // provider. A test that needs the production behavior exercises an // explicit `` child, which is an ordinary run and - // is handed the entrypoint's own installer below. - unsupportedRepositories, - installRepositories, + // assembles the entrypoint's own bundled Plugin for itself. ); if (!result.ok) { reportFailure(result.error); @@ -1673,8 +1656,6 @@ function* test( }, { testing: true, plugins }, installService, - unsupportedRepositories, - installRepositories, ); if (!result.ok) { reportFailure(result.error, document.relativePath); @@ -2672,7 +2653,6 @@ function* dispatch( helpRequest: { requested: boolean; args: string[] }, installService: HostServiceInstaller, upgrade: UpgradeAssembly, - installRepositories: RepositoryInstaller, readStandardInput: StandardInputReader, workflowHost: WorkflowHost | undefined, sessions: MachineSessionAssembly | undefined, @@ -2809,7 +2789,6 @@ function* dispatch( agent: runStack, }, installService, - installRepositories, ); }); if (!result.ok) { @@ -3071,7 +3050,6 @@ function* dispatch( { ...command.config, retainProcessOutput: keepsProcessOutput(command.config.journal) }, evalFlags.rest, installService, - installRepositories, plugins, ); break; @@ -3277,9 +3255,6 @@ function* dispatch( // service adapter would: installed inside the execution scope, // before the root document is imported. useWorkflowServiceDenial, - // A workflow run's repositories are the retained ones its Workspace - // attachment installs, so this path installs none of its own. - unsupportedRepositories, ), ), ); @@ -3297,12 +3272,6 @@ export function* runXmd( // command run under any other one refuses with that installation's own remedy // rather than reaching for a release, a lock or a file. upgrade: UpgradeAssembly, - // What an ordinary document execution installs for ``, - // ``, the Git operations, `` and ``. Deno and - // the compiled binary supply the live provider; Node and Bun supply the one - // that installs nothing, so those runtimes describe the same vocabulary and - // operate none of it. - installRepositories: RepositoryInstaller, // How this host reads a whole document from its own standard input, for the // one command form that asks for one. The shared CLI reaches no stdin global // of its own, and nothing a document can write reaches this: it is a value @@ -3361,7 +3330,6 @@ export function* runXmd( selection, installService, upgrade, - installRepositories, readStandardInput, loadPluginModule, installWorkflowHost, @@ -3423,7 +3391,6 @@ function* runCommand( selection: PluginSelection, installService: HostServiceInstaller, upgrade: UpgradeAssembly, - installRepositories: RepositoryInstaller, readStandardInput: StandardInputReader, loadPluginModule: PluginModuleLoader, installWorkflowHost: HostWorkflowInstaller, @@ -3489,7 +3456,6 @@ function* runCommand( helpRequest, installService, upgrade, - installRepositories, readStandardInput, workflowHost, sessions, diff --git a/packages/cli/src/compiled-repl.ts b/packages/cli/src/compiled-repl.ts index a8fa63322..8e3fff9d3 100644 --- a/packages/cli/src/compiled-repl.ts +++ b/packages/cli/src/compiled-repl.ts @@ -16,6 +16,7 @@ import process from "node:process"; import type { Operation } from "effection"; import { installReplHost, platformDataRoot } from "./repl-assembly.ts"; import { useCompiledReplTerminal } from "./compiled-repl-terminal.ts"; +import { useDenoReplRepositories } from "./deno-repositories.ts"; export function* useCompiledRepl(): Operation { yield* installReplHost({ @@ -26,5 +27,12 @@ export function* useCompiledRepl(): Operation { createExclusive: (path) => open(path, "wx").then((handle) => handle.close()), appendRecord: (path, record) => appendFile(path, record), }); + // What an entry's execution initializes: the recording of the File and Dir + // work it does in the checkout the person opened the session in. Registered + // here, in the runtime scope the selected REPL host owns, because only a + // runtime-named module may name the native adapter — and registering it + // acquires nothing, so a session that only reads history discovers no + // checkout and takes no lease. + yield* useDenoReplRepositories(); yield* useCompiledReplTerminal(); } diff --git a/packages/cli/src/compiled.ts b/packages/cli/src/compiled.ts index 5885be25c..dcccf64b9 100644 --- a/packages/cli/src/compiled.ts +++ b/packages/cli/src/compiled.ts @@ -17,6 +17,7 @@ import { compiledUpgradeAssembly } from "./compiled-upgrade.ts"; import { useMachineSessions } from "./session-coordinator.ts"; import { useDenoWorkflowHost } from "./deno-workflow.ts"; import { denoRunRepositories } from "./deno-repositories.ts"; +import { BundledRunPlugin } from "./run-profile.ts"; import { isCredentialHelperMode, runCredentialHelper } from "@executablemd/git/credential-helper"; import type { HelperAssembly } from "@executablemd/git/credential-helper"; import { useCompiledService } from "./compiled-service.ts"; @@ -84,11 +85,15 @@ if (isCredentialHelperMode(process.argv.slice(2))) { // The ordinary repository provider, on the same terms the Deno entrypoint // installs it: the binary is Deno, and the helper assembly it hands over is // the one that names this executable rather than a module path. + // The bundled Plugin this host runs with: the configured native variant, + // selected before any command is dispatched. The shared command path reads + // the value and imports no adapter, and an ordinary execution installs it + // in its own scope. + yield* BundledRunPlugin.set(denoRunRepositories(HELPER)); yield* runXmd( args, useCompiledService, UPGRADE, - denoRunRepositories(HELPER), () => readInputStream(process.stdin), importPluginModule, () => useDenoWorkflowHost(HELPER), diff --git a/packages/cli/src/deno-repl.ts b/packages/cli/src/deno-repl.ts index 40d2fe5d5..ad4f6d8a6 100644 --- a/packages/cli/src/deno-repl.ts +++ b/packages/cli/src/deno-repl.ts @@ -16,6 +16,7 @@ import process from "node:process"; import type { Operation } from "effection"; import { installReplHost, platformDataRoot } from "./repl-assembly.ts"; import { useDenoReplTerminal } from "./deno-repl-terminal.ts"; +import { useDenoReplRepositories } from "./deno-repositories.ts"; export function* useDenoRepl(): Operation { yield* installReplHost({ @@ -26,5 +27,12 @@ export function* useDenoRepl(): Operation { createExclusive: (path) => open(path, "wx").then((handle) => handle.close()), appendRecord: (path, record) => appendFile(path, record), }); + // What an entry's execution initializes: the recording of the File and Dir + // work it does in the checkout the person opened the session in. Registered + // here, in the runtime scope the selected REPL host owns, because only a + // runtime-named module may name the native adapter — and registering it + // acquires nothing, so a session that only reads history discovers no + // checkout and takes no lease. + yield* useDenoReplRepositories(); yield* useDenoReplTerminal(); } diff --git a/packages/cli/src/deno-repositories.ts b/packages/cli/src/deno-repositories.ts index 07e02e72a..10dc8f5ca 100644 --- a/packages/cli/src/deno-repositories.ts +++ b/packages/cli/src/deno-repositories.ts @@ -1,49 +1,115 @@ /** - * The live repository provider, assembled where it can be. + * The live repository behavior, assembled where it can be. * * Kept apart from `run-repositories.ts` because that module is on the shared * command path and this one names the Deno adapter, whose module graph reaches * `node:sqlite`. Bun has no such built-in, so a static import of this from * shared code would stop `xmd` loading there — not refuse a repository - * operation, but fail to start at all. Only `deno.ts` and `compiled.ts` import - * this file, and both of them are Deno. + * operation, but fail to start at all. Only `deno.ts`, `compiled.ts` and the + * two REPL hosts beside them import this file, and all of them are Deno. * * Managed checkouts live beneath `~/.xmd/repositories` and survive every * execution: what is in one is somebody's work, and nothing deletes one. There * is no environment variable naming a different root, because the only caller * that needs one is a test, and a test is handed the root directly. + * + * ## Configuration here, provider inside the execution + * + * What this module builds is a Plugin and a piece of middleware — inert values + * holding this deployment's credential-helper assembly and managed root. The + * provider itself is created by `Execution.init()`, inside the scope the + * invocation already owns, so every execution has its own: its own ambient + * discovery, its own leases, its own Push evidence, and teardown joined before + * the invocation settles. Two concurrent executions share none of it, and a + * child shares nothing with its parent. */ +import { Execution } from "@executablemd/core/api"; +import type { Plugin, PluginInstallRequest, PluginInstallation } from "@executablemd/core/api"; import type { Operation } from "effection"; import { cwd } from "@executablemd/runtime"; -import { useRunComposition } from "@executablemd/git/deno"; +import { gitPlugin } from "@executablemd/git"; +import { useReplRepositoryRecording, useRunComposition } from "@executablemd/git/deno"; import type { HelperAssembly } from "@executablemd/git/credential-helper"; import { DEFAULT_REPOSITORY_ROOT } from "./run-repositories.ts"; -import type { RepositoryInstaller } from "./run-repositories.ts"; /** - * The live provider Deno and the compiled binary install. + * The bundled Git Plugin, configured for this deployment. * - * It reads no configuration. The two GitHub variables belong to the adapter - * inside `@executablemd/git`, which reads them when an invoked GitHub-backed - * operation needs them — so a run that opens no pull request and files no issue - * reads neither, and an operator who wrote something unusable learns it from - * the operation that needed it rather than from a command that was only - * starting up. + * The same Plugin by name and vocabulary as the provider-neutral one — it + * delegates the whole of that install, declarations and admissions included — + * with one addition: for the command that runs an ordinary document, it + * registers what that execution initializes. Node and Bun select the neutral + * value instead and reach none of this. + * + * It reads no configuration of its own. The two GitHub variables belong to the + * adapter inside `@executablemd/git`, which reads them when an invoked + * GitHub-backed operation needs them — so a run that opens no pull request and + * files no issue reads neither, and an operator who wrote something unusable + * learns it from the operation that needed it rather than from a command that + * was only starting up. */ export function denoRunRepositories( helper: HelperAssembly, root: string = DEFAULT_REPOSITORY_ROOT, -): RepositoryInstaller { - return function* (): Operation { - yield* useRunComposition({ - root, - // The directory this execution starts in, which is where the ambient - // repository is discovered from. Read through the contextual Api rather - // than from the process, so a nested execution that composed its own - // working directory is discovered from that one. - cwd: yield* cwd(), - helper, - }); +): Plugin { + return { + // The bundled name, because this *is* the bundled Plugin: an operator + // selecting `--plugin git` is naming this value, and two values under one + // name is the collision `admitPlugins()` refuses. + name: gitPlugin.name, + *install(request: PluginInstallRequest): Operation { + // The neutral value's own install, whole: this variant adds to the + // bundled Plugin rather than reimplementing it, so the vocabulary, the + // commands it declares for and the admissions it contributes are the ones + // that package decides. + const install = gitPlugin.install; + const declared = install === undefined ? undefined : yield* install(request); + if (declared === undefined) { + return undefined; + } + // Only for the command whose documents are ordinary runs. `repl` records + // its entries through the REPL host below, `plan` and `syntax` describe a + // document without executing anybody's, and a workflow run's provider is + // the one its Workspace attachment installs. + if (request.command !== "run") { + return declared; + } + yield* Execution.around({ + *init(_args, next): Operation { + yield* useRunComposition({ + root, + // The directory this execution starts in, which is where the + // ambient repository is discovered from. Read at initialization + // through the contextual Api rather than from the process, so an + // isolated child that composed its own working directory is + // discovered from that one. + cwd: yield* cwd(), + helper, + }); + yield* next(); + }, + }); + return declared; + }, }; } + +/** + * What a REPL entry's execution initializes: recording, and nothing else. + * + * Registered by the selected REPL host rather than by the bundled Plugin, + * because an entry is not an ordinary run: it retains the file work it does in + * the checkout the person is standing in, and installs no managed root, + * credential helper or Git host. Each executing entry initializes its own, in + * the scope below its Agent attachment, so a passive view — a settled history, + * a frozen position — initializes nothing and takes no lease. + */ +export function* useDenoReplRepositories(): Operation { + yield* Execution.around({ + *init(_args, next): Operation { + yield* useReplRepositoryRecording(); + yield* next(); + }, + }); +} diff --git a/packages/cli/src/deno.ts b/packages/cli/src/deno.ts index a0b7ff5cb..742781099 100644 --- a/packages/cli/src/deno.ts +++ b/packages/cli/src/deno.ts @@ -20,6 +20,7 @@ import type { UpgradeAssembly } from "./upgrade.ts"; import { useMachineSessions } from "./session-coordinator.ts"; import { useDenoWorkflowHost } from "./deno-workflow.ts"; import { denoRunRepositories } from "./deno-repositories.ts"; +import { BundledRunPlugin } from "./run-profile.ts"; import { isCredentialHelperMode, runCredentialHelper } from "@executablemd/git/credential-helper"; import type { HelperAssembly } from "@executablemd/git/credential-helper"; import { useDenoService } from "./deno-service.ts"; @@ -103,11 +104,15 @@ if (isCredentialHelperMode(process.argv.slice(2))) { // parameterized by the same credential-helper assembly the workflow host // uses, because the program that is running is what knows how to re-invoke // itself as one. + // The bundled Plugin this host runs with: the configured native variant, + // selected before any command is dispatched. The shared command path reads + // the value and imports no adapter, and an ordinary execution installs it + // in its own scope. + yield* BundledRunPlugin.set(denoRunRepositories(HELPER)); yield* runXmd( args, useDenoService, UPGRADE, - denoRunRepositories(HELPER), () => readInputStream(process.stdin), importPluginModule, () => useDenoWorkflowHost(HELPER), diff --git a/packages/cli/src/node.ts b/packages/cli/src/node.ts index d72a3b860..adc545d7d 100755 --- a/packages/cli/src/node.ts +++ b/packages/cli/src/node.ts @@ -23,7 +23,6 @@ import { importPluginModule } from "./host-plugin-modules.ts"; import type { UpgradeAssembly } from "./upgrade.ts"; import { unassembledMachineSessions } from "./session-coordinator.ts"; import { unsupportedWorkflowHost } from "./workflow.ts"; -import { unsupportedRepositories } from "./run-repositories.ts"; import { useNodeService } from "./node-service.ts"; const ENTRYPOINT = fileURLToPath(import.meta.url); @@ -80,7 +79,6 @@ await main(function* (args) { args, useNodeService, UPGRADE, - unsupportedRepositories, () => readInputStream(process.stdin), importPluginModule, unsupportedWorkflowHost, diff --git a/packages/cli/src/repl-profile.ts b/packages/cli/src/repl-profile.ts index a4156294f..7ab84b20e 100644 --- a/packages/cli/src/repl-profile.ts +++ b/packages/cli/src/repl-profile.ts @@ -38,8 +38,9 @@ import type { CommandPlugins } from "./plugin-host.ts"; const REPL_INCLUDES: readonly string[] = Object.freeze(["components", "."]); /** - * The two facts a proof states and production leaves to the host. + * What the caller states and this assembler cannot work out for itself. * + * The first two are facts a proof states and production leaves to the host. * `planAgentContext` and `installAgentProviderStack` already take the first: * production states none and both get the real ACPX runtime, while a journey * states a scriptable one so it can drive this exact assembler and this exact @@ -47,8 +48,13 @@ const REPL_INCLUDES: readonly string[] = Object.freeze(["components", "."]); * keeps its conversations, which production leaves at the host default and a * proof points at a directory it created itself — a suite that used the default * would read and remove directories under the developer's own home. + * + * The third is the opposite way round: production states it and a proof usually + * does not. Only a runtime whose module graph can reach the Deno adapter has an + * entry host to offer, so the entrypoint that knows supplies it and the shared + * command path carries it as data. */ -export interface ReplProfileSeams { +export interface ReplProfileInputs { readonly acp?: AcpxProviderDependencies; readonly planWriterRoot?: string; } @@ -95,13 +101,13 @@ export interface ReplExecutionProfile { export function* assembleReplProfile( stack: AgentStack, plugins: CommandPlugins, - seams: ReplProfileSeams = {}, + inputs: ReplProfileInputs = {}, ): Operation { - yield* installAgentProviderStack(stack, seams.acp); + yield* installAgentProviderStack(stack, inputs.acp); const plan = yield* planComponentDeclaration({ surface: "component", - ...(seams.planWriterRoot === undefined ? {} : { planWriterRoot: seams.planWriterRoot }), + ...(inputs.planWriterRoot === undefined ? {} : { planWriterRoot: inputs.planWriterRoot }), includes: REPL_INCLUDES, // The command's own Plugin assembly, not a second installation of it: two // would be two catalogs claiming the same names. @@ -109,7 +115,7 @@ export function* assembleReplProfile( // The same seam `planAgentContext` already takes: production states none and // gets the real ACPX runtime, and a proof states a scriptable one so a // journey can drive this exact assembler rather than a copy of it. - context: planAgentContext(stack, seams.acp), + context: planAgentContext(stack, inputs.acp), ...(stack.sessions === undefined ? {} : { sessions: stack.sessions }), host: yield* useScope(), // Plan review is answered by the REPL's own Elicit provider, which the diff --git a/packages/cli/src/run-profile.ts b/packages/cli/src/run-profile.ts index 4be96ba08..52afe6bb5 100644 --- a/packages/cli/src/run-profile.ts +++ b/packages/cli/src/run-profile.ts @@ -27,7 +27,8 @@ * module's business, and assembling them is all it does. */ -import type { Operation } from "effection"; +import { createContext } from "effection"; +import type { Context, Operation } from "effection"; import { cwd } from "@executablemd/runtime"; import { gitPlugin, gitPluginDeclaresFor } from "@executablemd/git"; import type { Plugin } from "@executablemd/core/api"; @@ -43,9 +44,34 @@ import type { PluginSelection } from "./plugin-selection.ts"; */ export const BUNDLED_SELECTOR = "git"; -/** The Plugin every run-profile command begins with. */ +/** The Plugin every run-profile command begins with, where the host states none. */ export const BUNDLED_PLUGIN: Plugin = gitPlugin; +/** + * The bundled Plugin this host runs with. + * + * One selection, made by the runtime entrypoint before any command is + * dispatched, because only a runtime-named module may name a native adapter: + * Deno and the compiled binary bind a configured variant of the bundled Git + * Plugin, and Node and Bun leave the provider-neutral default in place. The + * shared command path reads the value and imports no adapter. + * + * What it selects is inert code and configuration — a Plugin value, whose + * `install` registers what an execution later initializes. It holds no + * provider, no lease, no Git session, no recorder and no identity, so nothing + * read from here is authority over permission, durable identity or + * reconciliation; those stay with core and with the providers an execution + * creates for itself. + * + * It is CLI-private. Nothing exports it as a package member and no selector + * names it: an operator chooses Plugins with `--plugin`, and this is the host + * saying which bundled value those are chosen beside. + */ +export const BundledRunPlugin: Context = createContext( + "executablemd.cli.bundledRunPlugin", + BUNDLED_PLUGIN, +); + /** * Whether this command line runs with the bundled profile. * @@ -77,19 +103,23 @@ export function* assembleRunProfile( load: PluginModuleLoader, ): Operation { const bundled = carriesBundledPlugin(selection); + // The host's selection, not the statically imported default: a configured + // native variant and the neutral value are the same Plugin to everything + // below, and which one this is was decided by the entrypoint. + const value = yield* BundledRunPlugin.expect(); // Resolved before anything is loaded: the reserved selector names a value // this profile holds, so it is never handed to a module loader, and a command // whose profile does not carry the bundled Plugin cannot conjure one by // writing the selector either. const named = selection.specifiers.filter((specifier) => specifier !== BUNDLED_SELECTOR); if (named.length === 0) { - return bundled ? Object.freeze([BUNDLED_PLUGIN]) : Object.freeze([]); + return bundled ? Object.freeze([value]) : Object.freeze([]); } // Captured before the first module is loaded, so a relative path and a // package specifier both resolve where the caller is standing. const directory = yield* cwd(); const loaded = yield* loadPlugins(named, directory, load); - return Object.freeze(bundled ? [BUNDLED_PLUGIN, ...loaded] : [...loaded]); + return Object.freeze(bundled ? [value, ...loaded] : [...loaded]); } /** @@ -101,13 +131,18 @@ export function* assembleRunProfile( * parent declined. * * The bundled value is dropped from the outer list **by identity**. What may be - * dropped is the one object this host statically bundled — an operator who - * wrote the reserved selector is holding exactly that value, and prefixing it - * twice would be the host duplicating itself. A *different* Plugin that merely - * uses the same name is not this value, is not dropped, and meets - * `admitPlugins()`'s duplicate-name refusal like any other collision. Comparing - * names here would have let an impostor disappear instead of collide. + * dropped is the one object this host selected — an operator who wrote the + * reserved selector is holding exactly that value, and prefixing it twice would + * be the host duplicating itself. A *different* Plugin that merely uses the + * same name is not this value, is not dropped, and meets `admitPlugins()`'s + * duplicate-name refusal like any other collision. Comparing names here would + * have let an impostor disappear instead of collide. + * + * The value is passed in rather than read here, because a child assembles in + * its own isolated scope: what it must install is the value the *host* selected + * and the parent captured before any document code ran, not whatever a + * document later put in the Context. */ -export function nestedRunProfile(outer: readonly Plugin[]): readonly Plugin[] { - return Object.freeze([BUNDLED_PLUGIN, ...outer.filter((plugin) => plugin !== BUNDLED_PLUGIN)]); +export function nestedRunProfile(outer: readonly Plugin[], bundled: Plugin): readonly Plugin[] { + return Object.freeze([bundled, ...outer.filter((plugin) => plugin !== bundled)]); } diff --git a/packages/cli/src/run-repositories.ts b/packages/cli/src/run-repositories.ts index 859a7b3a6..1b0719cf5 100644 --- a/packages/cli/src/run-repositories.ts +++ b/packages/cli/src/run-repositories.ts @@ -14,47 +14,20 @@ * no environment variable naming a different root, because the only caller that * needs one is a test, and a test is handed the root directly. * - * Node and Bun install none of this. They register the same thirteen - * declarations, so `xmd syntax` describes one language and a document resolves - * the same names everywhere, and every operation then reaches a clear + * Node and Bun configure none of this. They select the provider-neutral + * bundled Plugin, so `xmd syntax` describes one language and a document + * resolves the same names everywhere, and every operation then reaches a clear * provider-absence error before anything local or remote is touched. * * Nothing here imports the Deno adapter. This module is on the shared command * path, and the adapter's module graph reaches `node:sqlite` — a built-in Bun * does not have — so a static import of it here would stop `xmd` loading there - * at all. The live installer lives beside the entrypoints that can use it, in - * `deno-repositories.ts`. + * at all. The configured Plugin that holds this root lives beside the + * entrypoints that can use it, in `deno-repositories.ts`. */ import { homedir } from "node:os"; import { join } from "node:path"; -import type { Operation } from "effection"; /** Where managed repositories and worktrees live. */ export const DEFAULT_REPOSITORY_ROOT: string = join(homedir(), ".xmd", "repositories"); - -/** - * How one document execution obtains repository operations, or does not. - * - * A function rather than a value, because the provider is installed *inside* - * the execution scope and holds that execution's own invocation identity, - * leases and Push evidence. A nested `` calls it again - * and gets a fresh instance, which is what keeps a child's evidence and locks - * out of its parent and its siblings. - */ -export type RepositoryInstaller = () => Operation; - -/** - * The runtimes that register the vocabulary and operate none of it. - * - * Installing nothing is the whole implementation: ``, ``, - * the Git operations, `` and `` each reach their own Api's - * default, which reports an absent provider before a lock, a credential, a - * subprocess or a request exists. `` is unaffected — it needs no provider. - */ -export function unsupportedRepositories(): Operation { - return noRepositories(); -} - -// deno-lint-ignore require-yield -function* noRepositories(): Operation {} diff --git a/packages/cli/src/testing-host.ts b/packages/cli/src/testing-host.ts index d34820269..b0d768f05 100644 --- a/packages/cli/src/testing-host.ts +++ b/packages/cli/src/testing-host.ts @@ -63,7 +63,6 @@ import { nestedRunProfile } from "./run-profile.ts"; import type { Plugin } from "@executablemd/core/api"; import { useOrdinaryGeneratedSelection } from "./evaluation-profile.ts"; import type { HostServiceInstaller } from "./cli.ts"; -import type { RepositoryInstaller } from "./run-repositories.ts"; /** What one child asks the entrypoint to build its `` declaration from. */ export interface ChildPlanDeclaration { @@ -123,12 +122,19 @@ export interface TestingHostSettings { /** * How this entrypoint installs repository operations for one execution. * - * Called again for every child, so an isolated `` - * constructs a provider instance of its own: its own invocation identity, its - * own leases and its own Push evidence. Nothing it publishes authorizes its - * parent or a sibling, and nothing they published authorizes it. + * Installed again for every child, so an isolated `` + * initializes a provider instance of its own: its own invocation identity, + * its own leases and its own Push evidence. Nothing it publishes authorizes + * its parent or a sibling, and nothing they published authorizes it. + * + * It is the value the *host* selected, captured by the command before any + * document code ran. A Plugin is inert — code and configuration — so what + * crosses into the child is what to install, never a live provider: the + * child's own initialization creates that inside the child's own scope. A + * test root that activates no bundled Plugin still carries this, because a + * `host="run"` child is an ordinary run whatever command is hosting it. */ - readonly installRepositories: RepositoryInstaller; + readonly bundledPlugin: Plugin; /** * How this entrypoint re-invokes itself as the test-agent worker, or why it * cannot. @@ -302,10 +308,10 @@ function* runProfileChild( // The bundled value is prefixed here rather than carried down from the root, // because the root never had it: `xmd test` is not a run profile, and a child // does not inherit what its parent declined. - const childPlugins = yield* installPlugins(nestedRunProfile(settings.plugins), { - command: "run", - args: settings.pluginArgs, - }); + const childPlugins = yield* installPlugins( + nestedRunProfile(settings.plugins, settings.bundledPlugin), + { command: "run", args: settings.pluginArgs }, + ); const installations: ExecutionInstallation[] = [...childPlugins.installations]; // What this child can establish for a `` written inside it. A child // nobody configured establishes nothing, which is the refusal `` has @@ -379,9 +385,6 @@ function* runProfileChild( // Attaching a native service belongs only to document execution, here as in the // command that owns it. yield* settings.installService(); - // And Repository selection the same way, from the same installer the command - // used — a fresh instance for this child alone. - yield* settings.installRepositories(); // The run profile's own generated selection, offered in this child's own // scope. A `host="run"` child is an ordinary run whatever command is hosting // it, so a child of `xmd test` — whose own document is a different profile — diff --git a/packages/cli/tests/compiled-upgrade.test.ts b/packages/cli/tests/compiled-upgrade.test.ts index 28b263c53..5aad92191 100644 --- a/packages/cli/tests/compiled-upgrade.test.ts +++ b/packages/cli/tests/compiled-upgrade.test.ts @@ -47,7 +47,6 @@ import process from "node:process"; import { runXmd } from "../src/cli.ts"; import { refusedStandardInput } from "./support/standard-input.ts"; import { refusedPluginModules } from "./support/plugin-modules.ts"; -import { unsupportedRepositories } from "../src/run-repositories.ts"; import { compiledUpgradeAssembly, writeAll } from "../src/compiled-upgrade.ts"; import type { CompiledUpgradeDependencies, @@ -1536,14 +1535,7 @@ function* commandStatus(args: string[], assembly: UpgradeAssembly): Operation { [...args], function* () {}, UPGRADE, - unsupportedRepositories, refusedStandardInput, refusedPluginModules, unsupportedWorkflowHost, diff --git a/packages/cli/tests/repl-repositories-pty.test.ts b/packages/cli/tests/repl-repositories-pty.test.ts new file mode 100644 index 000000000..3559805cd --- /dev/null +++ b/packages/cli/tests/repl-repositories-pty.test.ts @@ -0,0 +1,45 @@ +/** + * The whole public behavior, on a real terminal, from source (#181 E1, E9). + * + * Every other row of this feature's evidence installs its own host, which is + * what makes those rows portable and deterministic. None of them can show the + * thing a person actually does: type an entry that writes files, walk away from + * it at a question, change those files from a shell, and reopen the printed + * location. So this launches the production entrypoint for the host runtime on + * a pty, inside a checkout it seeded, and reads the answer off the screen and + * off the filesystem. + * + * ## One suite, two behavior, by runtime + * + * Deno is where the entry host exists, so the Deno branch runs the journey. Node + * and Bun assemble no entry host at all — the Deno adapter's module graph + * reaches `node:sqlite`, which Bun has not — so their branch runs the other + * behavior that matters, which is that a REPL there still performs ordinary file + * and directory work. Neither branch silently skips its own behavior. + * + * The journey itself is `support/repl-files-journey.ts`, shared with the + * compiled row so the two cannot drift. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { cliRuntime, cliShellCommand } from "@executablemd/test-support/launch"; +import type { Operation } from "effection"; +import { replFilesJourney, replOrdinaryFileWork } from "./support/repl-files-journey.ts"; + +const JOURNEY = cliRuntime() === "deno"; + +/** How this suite invokes `xmd`: the production entrypoint for this runtime. */ +function sourceCommand(args: readonly string[]): string { + return cliShellCommand([...args]); +} + +describe("a REPL entry's files, across leaving and coming back", () => { + it( + JOURNEY + ? "E1: a completed prefix answers again, and the suffix executes on the answer" + : "E9: this runtime performs ordinary file and directory work and retains none of it", + function* (): Operation { + yield* JOURNEY ? replFilesJourney(sourceCommand) : replOrdinaryFileWork(sourceCommand); + }, + ); +}); diff --git a/packages/cli/tests/repl-repositories.test.ts b/packages/cli/tests/repl-repositories.test.ts new file mode 100644 index 000000000..4cff74de6 --- /dev/null +++ b/packages/cli/tests/repl-repositories.test.ts @@ -0,0 +1,568 @@ +/** + * How a REPL entry gets its repository behavior, and who never gets any. + * + * This setup is the one thing in this command prepared per *entry* rather than + * per command, and what follows from it are what this suite holds. An + * entry that mutates files in the checkout the person is standing in records + * them in its own segment of the history, so a later entry doing identical work + * does that work rather than answering from the earlier entry's records. A view + * that is only reading — a settled history, a frozen position — prepares + * nothing at all, so it discovers nothing and takes no lease. And a host that + * installs no setup runs ordinary file work with nothing retained. + * + * The generated half is here rather than beside the provider because a + * generated fragment's operations cross the same one Files route an authored + * one does: there is no second instance to wrap, and what proves it is a + * fragment's read, write and delete being retained exactly as an authored + * one's are. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, resource, scoped, until } from "effection"; +import type { Operation, Result } from "effection"; +import { exists, readTextFile, rm } from "@effectionx/fs"; +import { mkdtemp, realpath, writeFile } from "node:fs/promises"; +import { appendFile, open } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { API, useHostFiles } from "@executablemd/runtime"; +import { InMemoryStream } from "@executablemd/durable-streams"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import { denoRepositoryHost } from "../../git/src/deno/composition/host.ts"; +import { gitSession } from "../../git/src/deno/composition/git.ts"; +import { + parseReplLocalRecord, + REPL_REPOSITORY_LOCAL, + REPL_REPOSITORY_SELECTION, +} from "../../git/src/retained-repository/records.ts"; +import { Execution } from "@executablemd/core/api"; +import type { ExecutionInstallation } from "@executablemd/core/host"; +import { ordinaryGeneratedSelection } from "../src/evaluation-profile.ts"; +import { useDenoReplRepositories } from "../src/deno-repositories.ts"; +import { gitPlugin } from "../../git/src/plugin.ts"; +import { ReplHost } from "../src/repl/host.ts"; +import { openReplSession, submitReplEntry } from "../src/repl/session.ts"; +import type { ReplSession } from "../src/repl/session.ts"; +import type { ReplExecution } from "../src/repl/journal.ts"; + +/** + * What a command contributes to an entry's execution. + * + * Taken from the Git Plugin's own contribution for the `repl` command rather + * than written out here, because that contribution is the production path: the + * admission that reads this feature's records travels with the vocabulary, and + * a row that hand-attached its own would pass with the Plugin contributing + * nothing. + */ +function* replInstallations(): Operation { + // Installing it also registers this package's components in the calling + // scope, which is what an entry needs to resolve ``: the Plugin is one + // contribution, and taking half of it would be assembling something no + // command assembles. + const install = gitPlugin.install; + if (install === undefined) { + throw new Error("the Git Plugin installs nothing"); + } + const installed = yield* install.call(gitPlugin, { command: "repl", args: [] }); + if (installed === undefined) { + throw new Error("the Git Plugin declared nothing for the repl command"); + } + return [{ admissions: [...(installed.admissions ?? [])] }]; +} + +/** One entry that settles, for the rows about a history nobody is continuing. */ +const SETTLED_ENTRY = 'settled'; + +interface Fixture { + readonly root: string; + readonly checkout: string; +} + +function useFixture(): Operation { + return resource(function* (provide) { + const root = yield* until(realpath(yield* until(mkdtemp(join(tmpdir(), "repl-cli-"))))); + yield* ensure(() => rm(root, { recursive: true, force: true })); + const checkout = join(root, "checkout"); + const git = gitSession(denoRepositoryHost(), root); + yield* git.run(["init", "-b", "main", checkout], root); + yield* until(writeFile(join(checkout, "seed.md"), "seed\n")); + yield* provide({ root, checkout }); + }); +} + +/** The host half a runtime entrypoint owns, installed here by the suite. */ +function useNodeReplHost(name: string): Operation { + return ReplHost.around( + { + // deno-lint-ignore require-yield + *identify(): Operation { + return name; + }, + *createExclusive([path]: [string]): Operation { + const handle = yield* until(open(path, "wx")); + yield* until(handle.close()); + }, + *appendRecord([path, record]: [string, string]): Operation { + yield* until(appendFile(path, record)); + }, + }, + { at: "min" }, + ); +} + +/** Where every authored path in these entries resolves. */ +function useWorkingDirectory(directory: string): Operation { + return API.Env.around( + { + // deno-lint-ignore require-yield + *cwd(): Operation { + return directory; + }, + }, + { at: "min" }, + ); +} + +interface Counted { + /** How many executions this session initialized. */ + initialized: number; +} + +/** + * The production REPL recording, with each initialization counted. + * + * Counted by a second layer on the same seam rather than by instrumenting the + * recording: what this observes is core invoking `init`, which is the rule — + * an entry that executes initializes once, and a view that only reads a history + * initializes not at all. + */ +function* useCountedRecording(counted: Counted): Operation { + yield* Execution.around({ + *init(_args, next): Operation { + counted.initialized += 1; + yield* next(); + }, + }); + yield* useDenoReplRepositories(); +} + +function execution(events: readonly DurableEvent[] = []): ReplExecution { + return { id: "execution-1", stream: new InMemoryStream([...events]) }; +} + +function opened(result: Result): ReplSession { + if (!result.ok) { + throw result.error; + } + return result.value; +} + +/** + * Wait until this session is holding a question. + * + * Subscribed before the pending one is read, because a signal delivers from + * subscription time: checking first and subscribing after would lose a question + * that arrived in between and wait forever for the next one. + */ +function* waitForQuestion(session: ReplSession): Operation { + const changes = yield* session.elicitation.changes; + if (session.elicitation.pending !== undefined) { + return; + } + // The stream carries each question as it arrives, so the first one it + // delivers is the one this entry is holding. + yield* changes.next(); +} + +/** The text a recorded read answers with, read off its journaled payload. */ +function recordedText(event: DurableEvent | undefined): string | undefined { + if (event?.type !== "yield" || event.result.status !== "ok") { + return undefined; + } + const record = parseReplLocalRecord(event.result.value); + const value = record?.outcome.ok === true ? record.outcome.value : undefined; + return typeof value === "string" ? value : undefined; +} + +/** Every record of one type this history holds. */ +function recordsOf(events: readonly DurableEvent[], type: string): readonly DurableEvent[] { + return events.filter((event) => event.type === "yield" && event.description.type === type); +} + +/** One submitted entry, run to completion against this execution's history. */ +function* submit( + fixture: Fixture, + source: string, + run: ReplExecution, + counted?: Counted, +): Operation { + return yield* scoped(function* () { + yield* useNodeReplHost("execution-1"); + yield* useHostFiles(); + yield* useWorkingDirectory(fixture.checkout); + const installations = yield* replInstallations(); + // Absent for the row about a host that registers none: the entry then runs + // on the ordinary Files provider with nothing retained. + if (counted !== undefined) { + yield* useCountedRecording(counted); + } + const session = opened( + yield* submitReplEntry({ + execution: run, + source, + installations, + generated: ordinaryGeneratedSelection(), + }), + ); + yield* session.join(); + return yield* run.stream.readAll(); + }); +} + +describe("how a REPL entry gets its repository behavior", () => { + it("retains an admitted fragment's write and delete, and replays them", function* () { + const fixture = yield* useFixture(); + const counted: Counted = { initialized: 0 }; + const run = execution(); + // A fragment that writes, a fragment that deletes, and a question nobody + // answers — which is what leaves the entry unfinished, so reopening it is a + // continuation rather than a reading of a settled history. + const source = [ + `from a fragment\n')}} allow={["write"]} />`, + `\n')}} allow={["write"]} />`, + 'Continue?', + ].join("\n"); + + const events = yield* scoped(function* () { + yield* useNodeReplHost("execution-1"); + yield* useHostFiles(); + yield* useWorkingDirectory(fixture.checkout); + const installations = yield* replInstallations(); + yield* useCountedRecording(counted); + const session = opened( + yield* submitReplEntry({ + execution: run, + source, + installations, + generated: ordinaryGeneratedSelection(), + }), + ); + yield* waitForQuestion(session); + // Left at the question: this scope's teardown halts the entry, which is + // what leaving through `[exit]` does. + return yield* run.stream.readAll(); + }); + + // The fragment's work happened, on the one Files route an authored + // operation uses. + expect(yield* readTextFile(join(fixture.checkout, "generated.md"))).toBe("from a fragment"); + expect(yield* exists(join(fixture.checkout, "seed.md"))).toBe(false); + // And it is this entry's, recorded exactly as an authored mutation is. + expect(recordsOf(events, REPL_REPOSITORY_LOCAL).length).toBe(2); + expect(counted.initialized).toBe(1); + + // The checkout changes behind the records, and the entry is reopened at the + // live head. A completed operation is answered from its record before any + // of this execution's middleware is reached, so a provider that refuses + // everything changes nothing about what it answers. + yield* until(writeFile(join(fixture.checkout, "seed.md"), "put back\n")); + yield* until(writeFile(join(fixture.checkout, "generated.md"), "edited by somebody else")); + const again: Counted = { initialized: 0 }; + yield* scoped(function* () { + yield* useNodeReplHost("execution-1"); + yield* useHostFiles(); + yield* useWorkingDirectory(fixture.checkout); + const installations = yield* replInstallations(); + // The document filesystem now refuses every mutation. A completed + // operation is answered from its record without reaching a provider at + // all, authored and generated alike. + yield* API.Files.around( + { + // deno-lint-ignore require-yield + *writeTextFile() { + throw new Error("the provider must not be reached by a replay"); + }, + // deno-lint-ignore require-yield + *deleteFile() { + throw new Error("the provider must not be reached by a replay"); + }, + }, + // The implementation slot, which is what "the provider" means: the + // recording stands above it, so a completed operation answers from the + // journal without delegating this far. + { at: "min" }, + ); + yield* useCountedRecording(again); + const session = opened( + yield* openReplSession({ + execution: { id: "execution-1", stream: new InMemoryStream([...events]) }, + installations, + generated: ordinaryGeneratedSelection(), + }), + ); + // The unfinished entry started once, and is holding its question again. + yield* waitForQuestion(session); + expect(again.initialized).toBe(1); + }); + + // Both external changes survive, which is what says neither completed + // operation ran again: a replayed delete would have removed the file + // somebody put back, and a replayed write would have overwritten their edit. + expect(yield* readTextFile(join(fixture.checkout, "seed.md"))).toBe("put back\n"); + expect(yield* readTextFile(join(fixture.checkout, "generated.md"))).toBe( + "edited by somebody else", + ); + }); + + it("denies a generated Dir, and performs nothing when access is denied", function* () { + const fixture = yield* useFixture(); + const counted: Counted = { initialized: 0 }; + const run = execution(); + + const source = [ + `\n\n')}} allow={["write"]} />`, + `x\n')}} allow={["read"]} />`, + ].join("\n"); + + const events = yield* submit(fixture, source, run, counted); + + // The write table is unchanged: carrying an ensure on the captured access + // admits no generated ``. + expect(yield* exists(join(fixture.checkout, "made"))).toBe(false); + // And a fragment admitted for reads performs no write at all. + expect(yield* exists(join(fixture.checkout, "refused.md"))).toBe(false); + expect(recordsOf(events, REPL_REPOSITORY_LOCAL)).toEqual([]); + }); + + it("E7: keeps two entries' equal records in their own segments", function* () { + const fixture = yield* useFixture(); + const counted: Counted = { initialized: 0 }; + const run = execution(); + const source = 'same'; + + yield* submit(fixture, source, run, counted); + // Removed between the two entries, so whether the second one performed its + // write is a question the filesystem answers rather than a counter. + yield* rm(join(fixture.checkout, "same.md")); + const events = yield* submit(fixture, source, run, counted); + + // One setup call per entry, prepared inside that entry rather than cached. + expect(counted.initialized).toBe(2); + const records = recordsOf(events, REPL_REPOSITORY_LOCAL); + expect(records.length).toBe(2); + const names = records.map((event) => (event.type === "yield" ? event.description.name : "")); + // Equal work in two entries is one name, and that is allowed: nothing in a + // record says which entry wrote it. + expect(names[0]).toBe(names[1]); + // What keeps them apart is the segment. The second entry replayed nothing + // of the first entry's — it performed its own write, which is why the file + // removed between them is back. + expect(yield* exists(join(fixture.checkout, "same.md"))).toBe(true); + expect(yield* readTextFile(join(fixture.checkout, "same.md"))).toBe("same"); + }); + + it("E6: retains a generated read's text, and answers it after the file changes", function* () { + const fixture = yield* useFixture(); + const counted: Counted = { initialized: 0 }; + const run = execution(); + // A fragment reads a file in the checkout, then the entry stops at a + // question — so the read is completed and the entry is not. The fragment + // binds the text rather than interpolating it: a generated fragment may not + // read a binding, which is the ceiling's own rule and not this feature's. + const source = [ + `\n')}} allow={["read"]} />`, + 'Continue?', + ].join("\n"); + + const events = yield* scoped(function* () { + yield* useNodeReplHost("execution-1"); + yield* useHostFiles(); + yield* useWorkingDirectory(fixture.checkout); + const installations = yield* replInstallations(); + yield* useCountedRecording(counted); + const session = opened( + yield* submitReplEntry({ + execution: run, + source, + installations, + generated: ordinaryGeneratedSelection(), + }), + ); + yield* waitForQuestion(session); + return yield* run.stream.readAll(); + }); + + // One record, for the read a fragment performed on the one Files route, + // carrying the exact text it answered with. + const records = recordsOf(events, REPL_REPOSITORY_LOCAL); + expect(records.length).toBe(1); + expect(recordedText(records[0])).toBe("seed\n"); + + // The file now says something else. A completed read is answered from what + // it answered with the first time, so the binding a continuation offers is + // the recorded one rather than the current text — and this execution's own + // provider is never asked. + yield* until(writeFile(join(fixture.checkout, "seed.md"), "changed by somebody else\n")); + const again: Counted = { initialized: 0 }; + const continued: ReplExecution = { + id: "execution-1", + stream: new InMemoryStream([...events]), + }; + const output = yield* scoped(function* () { + yield* useNodeReplHost("execution-1"); + yield* useHostFiles(); + yield* useWorkingDirectory(fixture.checkout); + const installations = yield* replInstallations(); + yield* API.Files.around( + { + // deno-lint-ignore require-yield + *readTextFile() { + throw new Error("a completed read must not reach a provider"); + }, + }, + { at: "min" }, + ); + yield* useCountedRecording(again); + const session = opened( + yield* openReplSession({ + execution: continued, + installations, + generated: ordinaryGeneratedSelection(), + }), + ); + // Reaching the question again is the rule: the completed read was + // answered from its record, because a provider that was asked would have + // thrown instead. + yield* waitForQuestion(session); + return yield* continued.stream.readAll(); + }); + + expect(again.initialized).toBe(1); + // Nothing was appended for the read a second time. + expect(recordsOf(output, REPL_REPOSITORY_LOCAL).length).toBe(1); + }); + + it("E9: retains nothing for a host that installs no setup", function* () { + const fixture = yield* useFixture(); + const run = execution(); + + const events = yield* submit(fixture, 'plain', run); + + // The ordinary Files provider did the work, and nothing was recorded. + expect(yield* readTextFile(join(fixture.checkout, "plain.md"))).toBe("plain"); + expect(recordsOf(events, REPL_REPOSITORY_LOCAL)).toEqual([]); + expect(recordsOf(events, REPL_REPOSITORY_SELECTION)).toEqual([]); + }); + + it("prepares nothing for a settled history nobody is continuing", function* () { + const fixture = yield* useFixture(); + const counted: Counted = { initialized: 0 }; + const run = execution(); + const events = yield* submit(fixture, SETTLED_ENTRY, run, counted); + expect(counted.initialized).toBe(1); + + // Reopened as a reader, with every entry already settled. + const reading: Counted = { initialized: 0 }; + yield* scoped(function* () { + yield* useNodeReplHost("execution-1"); + yield* useHostFiles(); + yield* useWorkingDirectory(fixture.checkout); + const installations = yield* replInstallations(); + yield* useCountedRecording(reading); + const session = opened( + yield* openReplSession({ + execution: { id: "execution-1", stream: new InMemoryStream([...events]) }, + installations, + generated: ordinaryGeneratedSelection(), + }), + ); + // Nothing is running, so there is nothing to join and nothing was prepared. + expect(session.live).toBe(false); + }); + expect(reading.initialized).toBe(0); + }); + + it("E7: refuses a history whose own record it cannot read, before new work", function* () { + const fixture = yield* useFixture(); + const counted: Counted = { initialized: 0 }; + const run = execution(); + const source = [ + 'recorded', + 'Continue?', + 'after', + ].join("\n"); + + const events = yield* scoped(function* () { + yield* useNodeReplHost("execution-1"); + yield* useHostFiles(); + yield* useWorkingDirectory(fixture.checkout); + const installations = yield* replInstallations(); + yield* useCountedRecording(counted); + const session = opened( + yield* submitReplEntry({ + execution: run, + source, + installations, + generated: ordinaryGeneratedSelection(), + }), + ); + yield* waitForQuestion(session); + return yield* run.stream.readAll(); + }); + + // A record of this feature's type that this version never wrote, standing + // *past* where the entry stopped. Nothing replays it, so the operation-level + // reader never looks at it: what refuses it is the admission the Git Plugin + // contributes for this command, which reads the whole retained segment + // before any of it is replayed. + const tampered: readonly DurableEvent[] = [ + ...events, + { + type: "yield", + coroutineId: "root", + description: { type: REPL_REPOSITORY_LOCAL, name: "beyond-the-cursor" }, + result: { status: "ok", value: { version: 1, operation: "file-write" } }, + }, + ]; + + const again: Counted = { initialized: 0 }; + const refusal = yield* scoped(function* (): Operation { + yield* useNodeReplHost("execution-1"); + yield* useHostFiles(); + yield* useWorkingDirectory(fixture.checkout); + const installations = yield* replInstallations(); + yield* useCountedRecording(again); + try { + const session = opened( + yield* openReplSession({ + execution: { id: "execution-1", stream: new InMemoryStream([...tampered]) }, + installations, + generated: ordinaryGeneratedSelection(), + }), + ); + if (session.live) { + yield* session.join(); + } + return undefined; + } catch (error) { + return error instanceof Error ? error.message : String(error); + } + }); + + // Raised rather than answered, and in this feature's own words. + expect(refusal).toContain("is not one this host can read"); + // And refused before the work after the question: the suffix never ran. + expect(yield* exists(join(fixture.checkout, "after.md"))).toBe(false); + }); + + it("records the entry's routing once, and no operation outside the checkout", function* () { + const fixture = yield* useFixture(); + const counted: Counted = { initialized: 0 }; + const run = execution(); + + const events = yield* submit(fixture, 'first', run, counted); + expect(recordsOf(events, REPL_REPOSITORY_SELECTION).length).toBe(1); + expect(recordsOf(events, REPL_REPOSITORY_LOCAL).length).toBe(1); + }); +}); diff --git a/packages/cli/tests/run-composition.test.ts b/packages/cli/tests/run-composition.test.ts index 6642637b1..373a182ba 100644 --- a/packages/cli/tests/run-composition.test.ts +++ b/packages/cli/tests/run-composition.test.ts @@ -34,7 +34,7 @@ import { syntaxSymbols } from "../src/syntax.ts"; import { installPlugins } from "../src/plugin-host.ts"; import type { CommandPlugins } from "../src/plugin-host.ts"; import { BUNDLED_PLUGIN } from "../src/run-profile.ts"; -import { DEFAULT_REPOSITORY_ROOT, unsupportedRepositories } from "../src/run-repositories.ts"; +import { DEFAULT_REPOSITORY_ROOT } from "../src/run-repositories.ts"; /** Every element an author can write that needs a repository provider. */ const OPERATIONS: readonly { readonly name: string; readonly source: string }[] = [ @@ -96,7 +96,6 @@ function ordinaryWithoutProvider(source: string, cwd: string): Operation { } }, SOURCE_UPGRADE, - unsupportedRepositories, refusedStandardInput, refusedPluginModules, ); diff --git a/packages/cli/tests/run-profile.test.ts b/packages/cli/tests/run-profile.test.ts index 4b7d06ffa..c9d5c1b07 100644 --- a/packages/cli/tests/run-profile.test.ts +++ b/packages/cli/tests/run-profile.test.ts @@ -11,6 +11,13 @@ * carried, never loaded, and the reserved selector names that value rather than * a module — so any module load at all is the failure this suite exists to * catch, and nothing here needs a fixture on disk. + * + * What the value *is* comes from the host: a Deno or compiled entrypoint + * selects a configured native variant before any command is dispatched, and + * Node and Bun leave the provider-neutral default. These rows select a distinct + * object deliberately, and assert on identity rather than on the name, because + * a prefix hard-coded to the statically imported default would pass every + * name-shaped assertion while delivering the wrong host's Plugin. */ import { describe, it } from "@executablemd/test-support/bdd"; @@ -20,6 +27,7 @@ import { assembleRunProfile, BUNDLED_PLUGIN, BUNDLED_SELECTOR, + BundledRunPlugin, nestedRunProfile, } from "../src/run-profile.ts"; import { installPlugins } from "../src/plugin-host.ts"; @@ -39,8 +47,31 @@ function* profile(args: readonly string[]): Operation { return assembled.map((plugin) => plugin.name); } +/** The values one command line assembles, with this host's own selection made. */ +function selected(args: readonly string[]): Operation { + return scoped(function* () { + yield* BundledRunPlugin.set(CONFIGURED); + return yield* assembleRunProfile(selectPlugins(args), refusing); + }); +} + const GIT = "@executablemd/git"; +/** + * What a runtime entrypoint selects: the bundled Plugin, configured. + * + * A distinct object under the bundled name, which is what a native variant is. + * Nothing here needs it to do anything — the question these rows ask is which + * object arrives. + */ +const CONFIGURED: Plugin = Plugin({ + name: GIT, + // deno-lint-ignore require-yield + *install(): Operation { + return undefined; + }, +}); + describe("the bundled run profile", () => { it("carries the bundled Plugin for the commands that reach a document", function* () { for (const args of [["run", "doc.md"], ["plan", "doc.md"], ["syntax"], ["doc.md"]]) { @@ -70,6 +101,34 @@ describe("the bundled run profile", () => { } }); + it("assembles the value this host selected, and the default where it selects none", function* () { + // The host's selection reaches the profile as the bundled prefix. + expect(yield* selected(["run", "doc.md"])).toEqual([CONFIGURED]); + // And with nothing selected, the provider-neutral default stands: that is + // what Node and Bun run with. + expect( + yield* scoped(() => assembleRunProfile(selectPlugins(["run", "doc.md"]), refusing)), + ).toEqual([BUNDLED_PLUGIN]); + }); + + it("resolves the reserved selector to the selected value without loading anything", function* () { + // Once, five times, and beside an ineligible command: the selector names + // the value this host selected, so it is consumed rather than resolved — + // and it is that object, not the statically imported default. + expect(yield* selected(["--plugin", BUNDLED_SELECTOR, "run", "doc.md"])).toEqual([CONFIGURED]); + expect( + yield* selected([ + "--plugin", + BUNDLED_SELECTOR, + "--plugin", + BUNDLED_SELECTOR, + "--plugin=git", + "run", + "doc.md", + ]), + ).toEqual([CONFIGURED]); + }); + it("resolves the reserved selector without loading anything", function* () { // Once, five times, and beside an ineligible command: the selector names // the value this profile already holds, so it is consumed rather than @@ -92,12 +151,13 @@ describe("the bundled run profile", () => { expect(yield* profile(["--plugin", BUNDLED_SELECTOR, "test"])).toEqual([]); }); - it("gives a nested run child the bundled prefix without duplicating it", function* () { - // The outer command held the bundled value itself — an operator who wrote - // the reserved selector holds exactly this object. Prefixing it again would - // be the host duplicating itself, so it is dropped by identity. - expect(nestedRunProfile([BUNDLED_PLUGIN]).map((plugin) => plugin.name)).toEqual([GIT]); - expect(nestedRunProfile([]).map((plugin) => plugin.name)).toEqual([GIT]); + it("gives a nested run child the captured value without duplicating it", function* () { + // The value a child installs is the one its parent captured from the host, + // so these are identities rather than names: a child handed the neutral + // default where the host selected a configured variant would install the + // wrong Plugin under the right name. + expect(nestedRunProfile([CONFIGURED], CONFIGURED)).toEqual([CONFIGURED]); + expect(nestedRunProfile([], CONFIGURED)).toEqual([CONFIGURED]); // And an unrelated Plugin keeps its place behind the prefix. const other = Plugin({ @@ -107,7 +167,10 @@ describe("the bundled run profile", () => { return undefined; }, }); - expect(nestedRunProfile([other]).map((plugin) => plugin.name)).toEqual([GIT, "other"]); + expect(nestedRunProfile([other], CONFIGURED)).toEqual([CONFIGURED, other]); + // The outer list's own order is what follows the prefix. + const third = Plugin({ name: "third" }); + expect(nestedRunProfile([other, third], CONFIGURED)).toEqual([CONFIGURED, other, third]); }); it("refuses an impostor claiming the bundled name rather than dropping it", function* () { @@ -122,9 +185,9 @@ describe("the bundled run profile", () => { return undefined; }, }); - expect(impostor).not.toBe(BUNDLED_PLUGIN); + expect(impostor).not.toBe(CONFIGURED); - const assembled = nestedRunProfile([impostor]); + const assembled = nestedRunProfile([impostor], CONFIGURED); expect(assembled).toHaveLength(2); expect(assembled[1]).toBe(impostor); diff --git a/packages/cli/tests/stdin-cli.test.ts b/packages/cli/tests/stdin-cli.test.ts index 3f7faaf3c..72ff40ef0 100644 --- a/packages/cli/tests/stdin-cli.test.ts +++ b/packages/cli/tests/stdin-cli.test.ts @@ -42,7 +42,6 @@ import { STANDARD_INPUT_PATH, } from "../src/standard-input.ts"; import type { StandardInputReader } from "../src/standard-input.ts"; -import { unsupportedRepositories } from "../src/run-repositories.ts"; import { refusedPluginModules } from "./support/plugin-modules.ts"; import { SOURCE_UPGRADE } from "./support/upgrade-assembly.ts"; @@ -846,7 +845,6 @@ function* observedRun( }); }, SOURCE_UPGRADE, - unsupportedRepositories, reader.read, refusedPluginModules, ); diff --git a/packages/cli/tests/support/pty.ts b/packages/cli/tests/support/pty.ts new file mode 100644 index 000000000..36c4fb289 --- /dev/null +++ b/packages/cli/tests/support/pty.ts @@ -0,0 +1,468 @@ +/** + * A real pty, allocated by whichever allocator this host can drive. + * + * Copied rather than imported from `repl-exit-pty.test.ts`, which is where it + * was written and which still owns its own copy. Two suites need a real + * terminal for two different promises — one about leaving the screen, one about + * what an entry retained in a checkout — and neither is a reason to move the + * other's evidence. The duplication is deliberate and bounded: it is this file + * and that one, and nothing else reaches for a pty. + * + * ## Two pty allocators, because no host has both + * + * `script` is the portable one and is what the Linux runners use, but the BSD + * `script` on macOS refuses to start unless its *own* standard input is a + * terminal — which a test runner's never is. `tmux` has no such requirement and + * hands back the rendered screen directly. Either is a real pty with a real + * window size; whichever this host can drive is the one a row uses, and a host + * with neither says so rather than passing on nothing. + */ + +import { expect } from "@executablemd/test-support/expect"; +import { cliRuntime, shellQuote } from "@executablemd/test-support/launch"; +import { execFile, spawn as spawnChild } from "node:child_process"; +import type { ChildProcess } from "node:child_process"; +import { join } from "node:path"; +import { randomBytes } from "node:crypto"; +import { ensure, type Operation, scoped, sleep, withResolvers } from "effection"; + +/** The window the pty is given, which is a size this REPL draws at. */ +export const WINDOW = { columns: 160, rows: 36 }; + +/** What this row printed when it ended, so a status can be read off the screen. */ +const SENTINEL = "XMD-PTY-EXIT="; + +/** + * What running one program came to: whether it worked, and everything it said. + * + * The whole outcome rather than the output of a successful run, because the + * allocator's failures are this row's own failures and a row that discards them + * reports a product defect it has no evidence for. `tmux would not start a + * session` was all #872 had to go on; tmux had said `server exited unexpectedly` + * and nothing kept it. + */ +interface Attempt { + /** The command line, so a report names what was actually run. */ + readonly command: string; + /** Zero when the program succeeded, and none when it never ran. */ + readonly status: number | undefined; + /** Everything it wrote to either stream. */ + readonly output: string; + /** What stopped it before it could run, if anything did. */ + readonly failure: Error | undefined; + /** It ran, and it succeeded. */ + readonly ok: boolean; +} + +/** + * Run one program and keep its whole outcome. + * + * Scoped, so a cancelled probe leaves no client of its own behind: these are + * short-lived, and a short-lived process is still a process somebody owns. + */ +function* attempted(command: string, args: string[]): Operation { + return yield* scoped(function* (): Operation { + const owned = yield* ownedChild(() => + spawnChild(command, args, { stdio: ["ignore", "pipe", "pipe"] }), + ); + // The child's own closing fact, not a second pair of listeners watching for + // it: one closing per child, removed by the one cleanup that owns it. + yield* owned.closing; + const status = owned.child.exitCode; + const failure = owned.failure(); + return { + command: [command, ...args].join(" "), + status: status ?? undefined, + output: owned.written(), + failure, + ok: status === 0 && failure === undefined, + }; + }); +} + +/** + * What an attempt did, in the words of the program that made it. + * + * A spawn that never happened and a program that refused are different + * conditions, and an allocator that cannot be driven has to say which one it + * met — so both are spelled out here rather than collapsed into a failure. + */ +function described(attempt: Attempt): string { + const ended = + attempt.failure !== undefined + ? `could not be started (${attempt.failure.message})` + : `exited ${attempt.status ?? "without a status"}`; + const said = attempt.output.trim(); + return ( + `\`${attempt.command}\` ${ended} and said ` + + `${said.length === 0 ? "nothing" : JSON.stringify(said)}` + ); +} + +/** + * A child this scope owns, from before it exists until after it has closed. + * + * The lifecycle rule this file is about applies to the file itself: a test that + * proved a command joins what it owns, while leaking the process it proved it + * with, would be making the rule and breaking it in one row. + * + * So the release is registered before the child is spawned — a cancellation + * between the two must leave nothing running — and on teardown the child is + * signalled if it is still there and then **waited for**. An assigned exit status + * is not proof a process closed: `close` is, because it fires once the stdio has + * gone with it. The handlers are the ones that wait, so they are removed in a + * synchronous `finally` inside that same cleanup rather than before it. + */ +interface Listening { + // Loose in the listener's own parameters, so the real `ChildProcess` matches a + // shape it satisfies: what each event carries is checked where it arrives. + // oxlint-disable-next-line typescript/no-explicit-any + on(event: string, listener: (...args: any[]) => void): unknown; + // oxlint-disable-next-line typescript/no-explicit-any + off(event: string, listener: (...args: any[]) => void): unknown; +} + +interface ChildLike extends Listening { + // oxlint-disable-next-line typescript/no-explicit-any + kill(signal?: any): unknown; + readonly exitCode: number | null; + readonly signalCode: string | null; + readonly stdout?: Listening | null; + readonly stderr?: Listening | null; +} + +/** What an owned child offers while it runs. */ +interface Owned { + readonly child: T; + /** Everything it has written to either stream. */ + written(): string; + /** Whether it has closed, as the close event says rather than as a status does. */ + closed(): boolean; + /** + * Settles when the child has closed, and only then. + * + * Exposed rather than rebuilt by each consumer: there is one closing fact per + * child, and a second set of listeners watching for it would be a second thing + * nobody removes. + */ + readonly closing: Operation; + /** What the child reported going wrong, if it reported anything. */ + failure(): Error | undefined; +} + +function* ownedChild(start: () => T): Operation> { + let child: T | undefined; + let written = ""; + let gone = false; + let failure: Error | undefined; + const closing = withResolvers(); + const onClose = (): void => { + gone = true; + closing.resolve(); + }; + // An error is not a close. A child can report one and go on running, and its + // stdio is still open until `close` says otherwise — so this keeps the error + // and keeps waiting, because treating it as closure would be joining a process + // that is still there. + // oxlint-disable-next-line typescript/no-explicit-any + const onError = (cause: any): void => { + failure = failure ?? (cause instanceof Error ? cause : new Error(String(cause))); + }; + // oxlint-disable-next-line typescript/no-explicit-any + const collect = (chunk: any): void => { + written += String(chunk); + }; + + yield* ensure(function* (): Operation { + const open = child; + if (open === undefined) { + return; + } + child = undefined; + try { + // Decided from the close event, not from a status: a status is assigned + // when a process exits and says nothing about whether its stdio has gone, + // and this cleanup is about the child being finished rather than about it + // having a number. + if (!gone) { + open.kill("SIGKILL"); + } + // Awaited, not signalled and forgotten: until this settles there is still a + // process holding the pty this row allocated. + yield* closing.operation; + } finally { + // Synchronous, and the same handlers on the same receivers: these are what + // the wait above was waiting on, so they come off after it and not before. + open.off("close", onClose); + open.off("error", onError); + open.stdout?.off("data", collect); + open.stderr?.off("data", collect); + } + }); + + const spawned = start(); + child = spawned; + spawned.on("close", onClose); + spawned.on("error", onError); + spawned.stdout?.on("data", collect); + spawned.stderr?.on("data", collect); + return { + child: spawned, + written: () => written, + closed: () => gone, + closing: closing.operation, + failure: () => failure, + }; +} + +/** One terminal a test owns: what it shows, and how it ended. */ +interface Pty { + /** The screen, one string per row. */ + shows(): Operation; + /** Type these bytes. Nothing closes the input. */ + type(bytes: string): Operation; + /** The exit status once the command has ended, or none while it runs. */ + status(): Operation; +} + +/** Whichever allocator this host can drive. */ +function* allocate(line: string, home: string): Operation { + // One shell line, run by whichever allocator's own `sh -c`: the data root is + // isolated so this row cannot read or write the history of whoever ran it, the + // window is sized *inside* the pty because one allocated by a parent that is + // not a terminal has no window at all, and the status is printed where it can + // be read off the screen. + const inside = [ + `export HOME=${shellQuote(home)}`, + `export XDG_DATA_HOME=${shellQuote(join(home, "share"))}`, + `stty rows ${WINDOW.rows} cols ${WINDOW.columns}`, + line, + `printf '${SENTINEL}%s\\n' "$?"`, + ].join("; "); + + if ((yield* attempted("tmux", ["-V"])).ok) { + return yield* tmuxPty(inside, home); + } + const flavour = yield* attempted("script", ["--version"]); + if (flavour.ok && flavour.output.includes("util-linux")) { + return yield* scriptPty(inside); + } + throw new Error( + "this host can drive neither `tmux` nor util-linux `script`, so the pty this row " + + "needs cannot be allocated. Install tmux, or run on a host whose `script` does not " + + "require a terminal of its own.", + ); +} + +/** + * A tmux pane, which is a pty with a window and a screen that can be read. + * + * ## A server of this row's own + * + * tmux without `-S` is one server per user, shared by every session on the + * machine — the operator's own windows, the pane-worker proofs, a hand-driven + * `xmd repl`, and this same row running in another worktree. **That server + * exits with its last session**, so any one of those parties finishing takes the + * server down, and a `new-session` that happens to be in that window fails with + * `server exited unexpectedly` before the product has run. It is intermittent by + * construction: whether it happens depends on who else was finishing. + * + * So this row allocates against a socket inside the temporary home it already + * owns. Nobody else is on it, which is also why the teardown is `kill-server`: + * the server belongs to this row, so leaving it running would leak a process, + * and the socket file — which outlives the server — goes away with the directory + * holding it. `-f /dev/null` is the same isolation for configuration: a window + * size or a `default-shell` from whoever's `~/.tmux.conf` is foreign state this + * row would otherwise be reading. + */ +function* tmuxPty(inside: string, home: string): Operation { + const name = `xmd-pty-${randomBytes(6).toString("hex")}`; + const socket = join(home, "tmux.sock"); + /** Every tmux invocation for this row's own server, configured by nobody. */ + const control = (...args: string[]): string[] => ["-S", socket, "-f", "/dev/null", ...args]; + // Registered before the server exists, and it waits for the session to be + // gone rather than for the kill to be accepted: a kill that was sent is not a + // pane that has closed, and the pty belongs to the pane. + yield* ensure(function* (): Operation { + yield* attempted("tmux", control("kill-server")); + for (let attempt = 0; attempt < 100; attempt += 1) { + if (!(yield* attempted("tmux", control("has-session", "-t", name))).ok) { + return; + } + yield* sleep(50); + } + throw new Error(`the tmux session ${name} on ${socket} would not go away`); + }); + const started = yield* attempted( + "tmux", + control( + "new-session", + "-d", + "-s", + name, + "-x", + String(WINDOW.columns), + "-y", + String(WINDOW.rows), + "-c", + process.cwd(), + // Held open after the command ends, because tmux takes the pane away with + // the last process in it — and the status and the printed location are on + // that pane. The teardown above kills the server either way. + `${inside}; sleep 300`, + ), + ); + if (!started.ok) { + throw new Error( + `tmux would not start a session for this row: ${described(started)}. ` + + // The socket's length because a unix path is capped at 104 bytes and a + // longer one fails here, which is a condition of this row's own making + // rather than anything tmux can explain. + `socket=${socket} (${socket.length} of the 104 bytes a unix socket path may use), ` + + `session=${name}, window=${WINDOW.columns}x${WINDOW.rows}.`, + ); + } + /** The screen, or the reason there is no reading it — never a blank one. */ + function* captured(): Operation { + const pane = yield* attempted( + "tmux", + control("capture-pane", "-p", "-N", "-S", "-200", "-t", name), + ); + if (!pane.ok) { + // An unreadable screen used to read as an empty one, which this file then + // reported as the REPL never drawing. A pane that cannot be captured is a + // pty that has gone, and that is the fact worth printing. + throw new Error(`the tmux pane for this row could not be read: ${described(pane)}.`); + } + return pane.output; + } + return { + *shows(): Operation { + return (yield* captured()).split("\n"); + }, + *type(bytes: string): Operation { + // Literal, so what the decoder receives is what a keyboard sends. + const sent = yield* attempted("tmux", control("send-keys", "-t", name, "-l", "--", bytes)); + if (!sent.ok) { + // Typing that did not arrive is not typing: without this the row waits + // out its deadline and blames the REPL for not reacting to a keystroke + // it never received. + throw new Error( + `${JSON.stringify(bytes)} could not be typed into this row: ${described(sent)}.`, + ); + } + }, + *status(): Operation { + const found = (yield* captured()).match(new RegExp(`${SENTINEL}([0-9]+)`)); + return found === null ? undefined : Number(found[1]); + }, + }; +} + +/** A `script` pty, whose screen has to be reassembled from what it wrote. */ +function* scriptPty(inside: string): Operation { + const owned = yield* ownedChild(() => + spawnChild("script", ["-q", "-c", inside, "/dev/null"], { + stdio: ["pipe", "pipe", "pipe"], + cwd: process.cwd(), + }), + ); + return { + // deno-lint-ignore require-yield + *shows(): Operation { + return replayed(owned.written()); + }, + // deno-lint-ignore require-yield + *type(bytes: string): Operation { + owned.child.stdin?.write(bytes); + }, + // deno-lint-ignore require-yield + *status(): Operation { + const found = replayed(owned.written()) + .join("\n") + .match(new RegExp(`${SENTINEL}([0-9]+)`)); + return found === null ? undefined : Number(found[1]); + }, + }; +} + +/** + * What the terminal shows, by replaying what was written to it. + * + * A buffer rather than the bytes with escapes stripped, because this renderer + * writes *diffs*: it moves the cursor to what changed and writes only that. A + * focus marker arrives as "go to row 30, column 13, write `>`", so the text a + * person reads as `> [exit]` is never a run of bytes anybody sent. + */ +function replayed(written: string): string[] { + const rows: string[][] = []; + let row = 0; + let column = 0; + const put = (character: string): void => { + while (rows.length <= row) { + rows.push([]); + } + const line = rows[row]; + while (line.length < column) { + line.push(" "); + } + line[column] = character; + column += 1; + }; + for (let index = 0; index < written.length; index += 1) { + const character = written[index]; + if (character !== "\u001B") { + if (character === "\n") { + row += 1; + column = 0; + } else if (character === "\r") { + column = 0; + } else { + put(character); + } + continue; + } + const csi = /^\u001B\[([0-9;]*)([@-~])/.exec(written.slice(index)); + if (csi !== null) { + const parameters = csi[1].split(";").map((one) => (one === "" ? 0 : Number(one))); + if (csi[2] === "H") { + row = Math.max(0, (parameters[0] ?? 1) - 1); + column = Math.max(0, (parameters[1] ?? 1) - 1); + } else if (csi[2] === "J") { + rows.length = 0; + row = 0; + column = 0; + } + index += csi[0].length - 1; + continue; + } + const osc = /^\u001B\][^\u0007\u001B]*(?:\u0007|\u001B\\)/.exec(written.slice(index)); + if (osc !== null) { + index += osc[0].length - 1; + continue; + } + index += 1; + } + return rows.map((line) => line.join("")); +} + +/** Wait until the screen shows this text, or say what it shows instead. */ +function* showing(pty: Pty, what: string, within: number): Operation { + const deadline = Date.now() + within; + while (true) { + const screen = yield* pty.shows(); + if (screen.some((line) => line.includes(what))) { + return; + } + if (Date.now() > deadline) { + throw new Error( + `the terminal never showed ${what} within ${within}ms. screen=` + + JSON.stringify(screen.map((line) => line.trimEnd()).filter((line) => line.length > 0)), + ); + } + yield* sleep(150); + } +} + +export type { Pty }; +export { allocate, showing }; diff --git a/packages/cli/tests/support/repl-files-journey.ts b/packages/cli/tests/support/repl-files-journey.ts new file mode 100644 index 000000000..13611ced8 --- /dev/null +++ b/packages/cli/tests/support/repl-files-journey.ts @@ -0,0 +1,328 @@ +/** + * The one real-terminal journey this feature's public behavior rests on. + * + * Shared by two rows that differ in exactly one thing: which `xmd` runs. The + * Deno source row proves the entry host this repository builds; the compiled row + * proves the binary carries it. Writing the journey twice would let the two + * drift, and a compiled row asserting something slightly weaker than the source + * row is how a packaging defect survives. + * + * It is the specification's own example, typed as a person types it: a `` + * holding a write, a write and a delete, a read that binds what the write + * produced, a question that shows that binding, and a write after the question. + * Leave at the question, change the files from a shell, come back to the printed + * location, and what the entry already did must not happen again — including the + * read, whose binding the question still shows while the disk says something + * else. A *new* entry then reads the same path and sees what is there now, which + * is what says this is retention rather than a cache. + */ + +import { expect } from "@executablemd/test-support/expect"; +import { cliRuntime, shellQuote } from "@executablemd/test-support/launch"; +import { useTempDirectory } from "@executablemd/test-support/temp"; +import { type Operation, scoped, sleep, until } from "effection"; +import { exists, readTextFile, writeTextFile } from "@effectionx/fs"; +import { execFile } from "node:child_process"; +import { mkdir } from "node:fs/promises"; +import { join } from "node:path"; +import process from "node:process"; +import { allocate, showing } from "./pty.ts"; +import type { Pty } from "./pty.ts"; + +/** + * How long reaching the screen may take, and how long one interaction may take. + * + * Reaching the screen compiles or loads an entrypoint, which is slow and + * uninteresting. Everything after that is the subject. + */ +const REACH_MS = 180_000; +const ACT_MS = 30_000; + +/** How this row's `xmd` is invoked, as one shell command. */ +export type ReplCommand = (args: readonly string[]) => string; + +/** The entry this journey types, exactly as the specification shows it. */ +const ENTRY = [ + '', + 'first', + 'remove me', + '', + "", + '', + 'Recorded value: {held}. Continue this experiment?', + 'continued', +].join("\n"); + +/** A second entry, which reads the same path live rather than from a record. */ +const LIVE_READ = ['', "live is {live}"].join("\n"); + +/** + * One seeded checkout, inside the isolated home this row already owns. + * + * Seeded with a plain `git` rather than through this repository's own Deno + * adapter, so the fixture is the same on all three runtimes — the adapter is the + * subject of the Deno branch below, not of the setup both branches share. + */ +function* seedCheckout(home: string): Operation { + const checkout = join(home, "checkout"); + yield* until(mkdir(checkout, { recursive: true })); + yield* until( + new Promise((resolve, reject) => { + execFile( + "git", + ["init", "-b", "main", checkout], + { env: { ...process.env, HOME: home, GIT_CONFIG_GLOBAL: "/dev/null" } }, + (error) => (error === null ? resolve() : reject(error)), + ); + }), + ); + yield* writeTextFile(join(checkout, "seed.md"), "seed\n"); + return checkout; +} + +/** Walk the focus ring until this control is the marked one. */ +function* reach(pty: Pty, label: string): Operation { + const deadline = Date.now() + ACT_MS; + while (!(yield* pty.shows()).some((line) => line.includes(`> ${label}`))) { + if (Date.now() > deadline) { + throw new Error( + `focus never reached ${label} under ${cliRuntime()}. screen=` + + JSON.stringify((yield* pty.shows()).map((one) => one.trimEnd())), + ); + } + yield* pty.type("\t"); + yield* sleep(250); + } +} + +/** + * Activate the control holding this text with a pointer. + * + * The screen asks for SGR mouse reporting, so a press and a release at the + * cell the text starts in is what a person's click is. It is needed because the + * focus ring inside the question drawer does not stop on an enum's offered + * values: they are drawn and pointable, and a pointer is how a keyboard-only + * harness reaches one. + */ +function* click(pty: Pty, text: string): Operation { + const screen = yield* pty.shows(); + const row = screen.findIndex((line) => line.includes(text)); + if (row < 0) { + throw new Error( + `nothing on this screen holds ${JSON.stringify(text)} to click. screen=` + + JSON.stringify(screen.map((one) => one.trimEnd())), + ); + } + const column = (screen[row] ?? "").indexOf(text) + 1; + yield* pty.type(`\u001b[<0;${column};${row + 1}M`); + yield* pty.type(`\u001b[<0;${column};${row + 1}m`); + yield* sleep(300); +} + +/** Wait until this file holds this exact text. */ +function* holding(path: string, content: string): Operation { + const deadline = Date.now() + ACT_MS; + while (true) { + if ((yield* exists(path)) && (yield* readTextFile(path)) === content) { + return; + } + if (Date.now() > deadline) { + const found = (yield* exists(path)) ? yield* readTextFile(path) : ""; + throw new Error( + `${path} never held ${JSON.stringify(content)}; it holds ${JSON.stringify(found)}`, + ); + } + yield* sleep(200); + } +} + +/** + * Wait until this path is absent. + * + * The delete inside the `` happens after the write this row waited for, so + * asking once would be asking inside the window where the file still exists: + * which of two authored operations the entry has reached is not something this + * journey controls. A deadline is what makes the behavior "it was removed" rather + * than "it was removed before this line ran". + */ +function* removed(path: string): Operation { + const deadline = Date.now() + ACT_MS; + while (true) { + if (!(yield* exists(path))) { + return; + } + if (Date.now() > deadline) { + throw new Error(`${path} was never removed`); + } + yield* sleep(200); + } +} + +/** Wait for the command to end, and answer with its status. */ +function* ended(pty: Pty): Operation { + const deadline = Date.now() + ACT_MS; + while (true) { + const status = yield* pty.status(); + if (status !== undefined) { + return status; + } + if (Date.now() > deadline) { + throw new Error( + `the command never finished under ${cliRuntime()}. screen=` + + JSON.stringify((yield* pty.shows()).map((one) => one.trimEnd())), + ); + } + yield* sleep(150); + } +} + +/** The one location this command printed for a caller to come back with. */ +function printedLocation(screen: string): string { + const found = screen.match(/xmd repl '(xmd:\/\/repl\/[^']+)'/); + if (found === null || found[1] === undefined) { + throw new Error( + `this command printed no reopenable location. screen=${JSON.stringify(screen)}`, + ); + } + return found[1]; +} + +/** + * Run the whole journey, and answer with nothing. + * + * Takes the command rather than building one, so the only difference between + * the two rows that call this is which `xmd` they name. + */ +export function replFilesJourney(repl: ReplCommand): Operation { + return scoped(function* (): Operation { + const home = yield* useTempDirectory("xmd-repl-files-pty-"); + const checkout = yield* seedCheckout(home); + const inside = (args: readonly string[]): string => + `cd ${shellQuote(checkout)} && ${repl(args)}`; + + const first = yield* allocate(inside(["repl", "--deny-all"]), home); + yield* showing(first, "[exit]", REACH_MS); + + // Typed as a person types it: one paste carrying newlines, then Enter. + yield* first.type(ENTRY); + yield* sleep(500); + yield* first.type("\r"); + + // The prefix ran. Everything the entry authored before the question is on + // disk, and the write after the question is not. + yield* holding(join(checkout, "drafts", "note.md"), "first"); + yield* removed(join(checkout, "drafts", "obsolete.md")); + // And the write after the question has not happened: by now the prefix is + // through its last authored operation, so this one is absent because the + // entry is waiting rather than because it has not got there yet. + expect(yield* exists(join(checkout, "after.md"))).toBe(false); + + // The question is waiting, and the footer says so on one row. The read + // before it settled, which the transcript marks; what it *bound* is read in + // the drawer after the reopen below, because the prompt is rendered where it + // is answered rather than on the surface behind it. + yield* showing(first, "waiting for an answer", ACT_MS); + + // Left through [exit], with the question still open. + yield* reach(first, "[exit]"); + yield* first.type("\r"); + expect(yield* ended(first)).toBe(0); + const location = printedLocation((yield* first.shows()).join("\n")); + + // Changed from outside, which is exactly what this execution model allows: + // a completed result and the environment may disagree. + yield* writeTextFile(join(checkout, "drafts", "note.md"), "second"); + yield* writeTextFile(join(checkout, "drafts", "obsolete.md"), "back again"); + + // Reopened at the live head, in the same data root. + const second = yield* allocate(inside(["repl", "--deny-all", location]), home); + yield* showing(second, "[exit]", REACH_MS); + // The question is offered again, which is what says the unfinished entry + // started once rather than being replayed into a settled state. + yield* showing(second, "waiting for an answer", ACT_MS); + + // And no completed operation ran again: a replayed write would have + // overwritten the shell's edit, a replayed delete would have removed the + // file the shell put back, and a replayed read would have bound "second". + expect(yield* readTextFile(join(checkout, "drafts", "note.md"))).toBe("second"); + expect(yield* readTextFile(join(checkout, "drafts", "obsolete.md"))).toBe("back again"); + expect(yield* exists(join(checkout, "after.md"))).toBe(false); + + // Answered through the real drawer. Activating the one value the schema + // offers is the answer: a keystroke in this drawer reaches the field a + // *pointer* last selected rather than the line holding focus, so an enum is + // what a pointer alone can settle. Reported as a non-blocking observation; + // it is not this feature's behavior. + yield* reach(second, "[answer]"); + yield* second.type("\r"); + // The prompt the drawer renders carries the binding the completed read + // produced — `first`, from a record — while the file on disk says `second`. + // A replayed read would have bound the new text, and this is where that + // would show. + yield* showing(second, "Recorded value: first", ACT_MS); + yield* showing(second, "Continue this experiment?", ACT_MS); + yield* click(second, "( ) continue"); + yield* showing(second, "[ok] entry-1", ACT_MS); + + // The suffix write executed, for the first time, on the answer. + yield* holding(join(checkout, "after.md"), "continued"); + + yield* reach(second, "[exit]"); + yield* second.type("\r"); + expect(yield* ended(second)).toBe(0); + + // A separate new entry, opened at the same location: its read is new work, + // so it reads the file as it is now. What is retained is each operation, + // not a value the session remembers for the path. + // + // A third session rather than another entry typed into the second one: a + // draft is where keystrokes land when a session opens, and after answering + // a question they reach the transcript instead. That is the pre-existing + // focus defect the enum above is clicked for, and typing through it is not + // what this journey is about. + const third = yield* allocate(inside(["repl", "--deny-all", location]), home); + yield* showing(third, "[exit]", REACH_MS); + yield* third.type(LIVE_READ); + yield* sleep(500); + yield* third.type("\r"); + yield* showing(third, "live is second", ACT_MS); + + yield* reach(third, "[exit]"); + yield* third.type("\r"); + expect(yield* ended(third)).toBe(0); + }); +} + +/** + * The prefix alone, for a runtime that assembles no entry host. + * + * Node and Bun start a REPL and perform ordinary file and directory work; what + * they do not have is anything that retains it. So their behavior is that the + * document ran and the command left cleanly, and the journey above is not + * theirs. + */ +export function replOrdinaryFileWork(repl: ReplCommand): Operation { + return scoped(function* (): Operation { + const home = yield* useTempDirectory("xmd-repl-files-pty-"); + const checkout = yield* seedCheckout(home); + const pty = yield* allocate( + `cd ${shellQuote(checkout)} && ${repl(["repl", "--deny-all"])}`, + home, + ); + yield* showing(pty, "[exit]", REACH_MS); + yield* pty.type(ENTRY); + yield* sleep(500); + yield* pty.type("\r"); + + yield* holding(join(checkout, "drafts", "note.md"), "first"); + yield* removed(join(checkout, "drafts", "obsolete.md")); + // And the write after the question has not happened: by now the prefix is + // through its last authored operation, so this one is absent because the + // entry is waiting rather than because it has not got there yet. + expect(yield* exists(join(checkout, "after.md"))).toBe(false); + + yield* reach(pty, "[exit]"); + yield* pty.type("\r"); + expect(yield* ended(pty)).toBe(0); + }); +} diff --git a/packages/cli/tests/support/run-markdown-tier.ts b/packages/cli/tests/support/run-markdown-tier.ts index 26e12a968..7a9c6d07e 100644 --- a/packages/cli/tests/support/run-markdown-tier.ts +++ b/packages/cli/tests/support/run-markdown-tier.ts @@ -35,8 +35,8 @@ import { testingExecutionHost } from "../../src/testing-host.ts"; import { useBunService } from "../../src/bun-service.ts"; import { useDenoService } from "../../src/deno-service.ts"; import { useNodeService } from "../../src/node-service.ts"; +import { BUNDLED_PLUGIN } from "../../src/run-profile.ts"; -import { unsupportedRepositories } from "../../src/run-repositories.ts"; import reviewPlugin from "@executablemd/code-review-agent"; /** The native service adapter the entrypoint for this runtime installs. */ const SERVICES = { @@ -115,7 +115,7 @@ export function runMarkdownTier(document: string): Operation { }), // This harness runs Markdown tiers, not repository work: a child that // asked for a checkout is told there is no provider. - installRepositories: unsupportedRepositories, + bundledPlugin: BUNDLED_PLUGIN, }); const execution = yield* executeInstalled({ path: document, stream: new InMemoryStream() }, [ // The child declarations `xmd test` hands the harness, so a suite can diff --git a/packages/cli/tests/targets-cli.test.ts b/packages/cli/tests/targets-cli.test.ts index 74b93257f..9d62056c9 100644 --- a/packages/cli/tests/targets-cli.test.ts +++ b/packages/cli/tests/targets-cli.test.ts @@ -24,7 +24,6 @@ import { SOURCE_UPGRADE } from "./support/upgrade-assembly.ts"; import { refusedStandardInput } from "./support/standard-input.ts"; import { refusedPluginModules } from "./support/plugin-modules.ts"; -import { unsupportedRepositories } from "../src/run-repositories.ts"; function* useFixture( files: Record, body: (dir: string) => Operation, @@ -590,7 +589,6 @@ function* replacingRun( }); }, SOURCE_UPGRADE, - unsupportedRepositories, refusedStandardInput, refusedPluginModules, ); @@ -767,7 +765,6 @@ function* helpRun(args: string[], cwd: string): Operation { serviceInstalled = true; }, SOURCE_UPGRADE, - unsupportedRepositories, refusedStandardInput, refusedPluginModules, ); diff --git a/packages/cli/tests/testing-execution-host.test.ts b/packages/cli/tests/testing-execution-host.test.ts index 9359a7139..81b5896df 100644 --- a/packages/cli/tests/testing-execution-host.test.ts +++ b/packages/cli/tests/testing-execution-host.test.ts @@ -28,8 +28,18 @@ import { PLAN_WRITER_INSTRUCTIONS, DEFAULT_PLAN_WRITER_ROOT } from "../src/plan- import type { PlanWriterObservation } from "../src/plan-writer-profile.ts"; import { testingExecutionHost } from "../src/testing-host.ts"; import { planComponentDeclaration, planComponentDescription } from "../src/plan-component.ts"; +import { BUNDLED_PLUGIN, BundledRunPlugin } from "../src/run-profile.ts"; +import { denoRunRepositories } from "../src/deno-repositories.ts"; +import { Execution } from "@executablemd/core/api"; +import type { Plugin, PluginInstallation } from "@executablemd/core/api"; +import { cwd } from "@executablemd/runtime"; +import { useTempDirectory } from "@executablemd/test-support/temp"; +import { spawnSync } from "node:child_process"; +import process from "node:process"; +import { fileURLToPath } from "node:url"; +import type { HelperAssembly } from "@executablemd/git/credential-helper"; +import type { TestingHostSettings } from "../src/testing-host.ts"; -import { unsupportedRepositories } from "../src/run-repositories.ts"; import reviewPlugin from "@executablemd/code-review-agent"; function doc(...lines: string[]): string { return `${lines.join("\n")}\n`; @@ -783,7 +793,7 @@ describe("deterministic dependencies declared for a nested run", () => { secretDetection: true, // deno-lint-ignore require-yield installService: function* (): Operation {}, - installRepositories: unsupportedRepositories, + bundledPlugin: BUNDLED_PLUGIN, testAgentWorker: Ok([...cliBase(), "test-agent"]), planDeclaration: (request) => planComponentDeclaration({ @@ -1006,7 +1016,7 @@ describe("deterministic dependencies declared for a nested run", () => { secretDetection: true, // deno-lint-ignore require-yield installService: function* (): Operation {}, - installRepositories: unsupportedRepositories, + bundledPlugin: BUNDLED_PLUGIN, testAgentWorker: Err(new Error("xmd command not installed")), // The run profile's own Component is built for every child, and this case // is about the relaunch it cannot perform rather than about ``. @@ -1147,3 +1157,187 @@ describe("execution initialization in isolated run children", () => { expect(result.stdout + result.stderr).not.toContain("❌"); }); }); + +/** + * C1 — the native host an isolated run child actually initializes. + * + * A test root is not a run profile: it activates no bundled Plugin and has no + * operational repository provider of its own. Its `` + * child *is* an ordinary run, so it has to receive what this host selected — + * and what it receives is the inert Plugin value the command captured before + * any document code ran, never a live provider and never whatever a document + * put in the Context afterwards. + * + * This is asked of a configured native variant rather than the provider-neutral + * default, because the defect worth catching is a child initialized with the + * wrong host's Plugin: one hard-coded to the statically bundled value would + * satisfy every assertion about names while reaching a managed root nobody + * configured. + */ +describe("C1 — the native host a run child initializes", () => { + /** A bare remote a child may clone without reaching a network. */ + function* useBareRemote(home: string): Operation { + const parent = yield* until(realpath(yield* useTempDirectory("xmd-c1-remote-"))); + const seed = join(parent, "seed"); + git(["init", "--initial-branch=main", seed], parent, home); + git(["commit", "--allow-empty", "-m", "first"], seed, home); + const bare = join(parent, "remote.git"); + git(["clone", "--bare", "--", seed, bare], parent, home); + return bare; + } + + /** The configured native Plugin, with each execution it initializes recorded. */ + function observedNative(initialized: string[], root: string): Plugin { + const native = denoRunRepositories(HELPER, root); + return { + name: native.name, + *install(request): Operation { + const install = native.install; + const declared = install === undefined ? undefined : yield* install(request); + if (declared === undefined) { + return undefined; + } + yield* Execution.around({ + *init(_args, next): Operation { + initialized.push(`${request.command}@${yield* cwd()}`); + yield* next(); + }, + }); + return declared; + }, + }; + } + + it("E9: gives a test root's run child the captured configured Plugin, once", function* () { + const home = yield* useTempDirectory("xmd-c1-home-"); + const remote = yield* useBareRemote(home); + // This fixture's own managed root, so where a clone lands is observable and + // nothing reaches the tree a production run owns. + const managed = yield* until(realpath(yield* useTempDirectory("xmd-c1-managed-"))); + const root = join(managed, "repositories"); + // Where the child stands: a directory the parent chose, which is what a + // `` around the `` selects. + const workspace = yield* until(realpath(yield* useTempDirectory("xmd-c1-work-"))); + + const initialized: string[] = []; + const settings = hostSettings({ bundledPlugin: observedNative(initialized, root) }); + const host = testingExecutionHost(settings); + + // A document could replace the selection after the command captured it. + // What it replaces it with here would be visible if the child read it: this + // one declares for every document command and records a different word. + const replaced: string[] = []; + yield* BundledRunPlugin.set({ + name: "@executablemd/git", + *install(): Operation { + yield* Execution.around({ + *init(_args, next): Operation { + replaced.push("replaced"); + yield* next(); + }, + }); + return {}; + }, + }); + + let rendered = ""; + const settlement = yield* host.runChild({ + request: { + host: "run", + source: + `\n` + + "selected {repository}\n", + props: {}, + journal: "transient", + collectJournal: false, + configuration: [], + }, + run: undefined, + cwd: workspace, + *chunk(text: string): Operation { + rendered += text; + }, + }); + rendered += settlement.output; + + // The child ran, and the repository work an ordinary run can do worked. + expect(settlement.outcome.kind).toBe("settled"); + expect(settlement.outcome.kind === "settled" ? settlement.outcome.result.ok : false).toBe(true); + // Exactly one initialization, for command `run`, standing where the parent + // put the child rather than where this process is. + expect(initialized).toEqual([`run@${workspace}`]); + // And it was the captured value that did it: the Context the document + // replaced afterwards never reached the child. + expect(replaced).toEqual([]); + // The clone landed under the root this fixture configured, which is what + // proves the *configured* Plugin arrived rather than a default one. + expect(rendered).toContain(root); + expect(rendered).toContain("selected "); + }); +}); + +/** + * What this suite's host is, stated the way the Deno entrypoint states it. + * + * The same shape `deno.ts` builds, because what the configured Plugin holds is + * host configuration: a fixture that invented a different one would be + * measuring its own invention. + */ +const HELPER: HelperAssembly = { + runtime: "source", + platform: process.platform === "win32" ? "windows" : "unix", + execPath: process.execPath, + modulePath: fileURLToPath(new URL("../src/credential-helper-entry.ts", import.meta.url)), + launcherEnvironment: {}, +}; + +/** Git, run with configuration nobody's machine contributes to. */ +function git(args: readonly string[], cwd: string, home: string): string { + const outcome = spawnSync("git", [...args], { + cwd, + env: { + ...(process.env.PATH === undefined ? {} : { PATH: process.env.PATH }), + HOME: home, + GIT_CONFIG_GLOBAL: "/dev/null", + GIT_CONFIG_SYSTEM: "/dev/null", + GIT_CONFIG_NOSYSTEM: "1", + GIT_TERMINAL_PROMPT: "0", + LC_ALL: "C", + GIT_AUTHOR_NAME: "Fixture", + GIT_AUTHOR_EMAIL: "fixture@example.invalid", + GIT_COMMITTER_NAME: "Fixture", + GIT_COMMITTER_EMAIL: "fixture@example.invalid", + }, + encoding: "utf8", + }); + if (outcome.status !== 0) { + throw new Error(`git ${args.join(" ")} failed: ${outcome.stderr}`); + } + return outcome.stdout; +} + +/** The host settings a command assembles, with one member under test. */ +function hostSettings(chosen: { readonly bundledPlugin: Plugin }): TestingHostSettings { + return { + includes: [], + plugins: [], + pluginArgs: [], + secretDetection: false, + // deno-lint-ignore require-yield + installService: function* (): Operation {}, + bundledPlugin: chosen.bundledPlugin, + testAgentWorker: Ok([...cliBase(), "test-agent"]), + planDeclaration: (request) => + planComponentDeclaration({ + surface: "component", + includes: [], + context: request.context, + ...(request.planWriterRoot === undefined ? {} : { planWriterRoot: request.planWriterRoot }), + host: request.host, + ...(request.observePlanWriter === undefined + ? {} + : { observePlanWriter: request.observePlanWriter }), + installElicitation: request.installElicitation, + }), + }; +} diff --git a/packages/git/deno.ts b/packages/git/deno.ts index 67b2606ac..2c3bfcd86 100644 --- a/packages/git/deno.ts +++ b/packages/git/deno.ts @@ -48,3 +48,5 @@ export { gitWorkspaceAttachment } from "./src/deno/attachment.ts"; export type { GitWorkspaceOptions } from "./src/deno/attachment.ts"; export { useRunComposition } from "./src/deno/run-composition/provider.ts"; export type { RunCompositionOptions } from "./src/deno/run-composition/provider.ts"; +export { useReplRepositoryRecording } from "./src/deno/retained-repository/provider.ts"; +export type { ReplRecordingOptions } from "./src/deno/retained-repository/provider.ts"; diff --git a/packages/git/src/deno/retained-repository/files.ts b/packages/git/src/deno/retained-repository/files.ts new file mode 100644 index 000000000..4db8186bd --- /dev/null +++ b/packages/git/src/deno/retained-repository/files.ts @@ -0,0 +1,417 @@ +/** + * The file work one REPL entry records, and the work it simply performs. + * + * A document reads and writes files in two places: inside the checkout the + * entry was started in, and everywhere else. Only the first is repository work. + * A write beside the document, in a temporary directory or anywhere outside + * that checkout is ordinary filesystem behavior and is journaled by nothing + * here — giving it a record would place it in a checkout it is not in. + * + * Inside the checkout the call goes through this entry's durable boundary: the + * composition beneath performs it, and what it produced is recorded once. A + * completed read answers with the text it answered with the first time and a + * completed write answers from its receipt, each touching no file, however the + * file has changed since. + * + * ## One route, one policy + * + * `` reads and writes, `` and `` all arrive as + * ordinary `API.Files` calls, and a generated fragment's operations cross that + * same route, so the recording is ordinary middleware over the four operations + * this feature retains and nothing else. A Glob, a path check and a temporary + * directory bind nothing a continuation has to answer again, and are left + * alone. + * + * What the middleware sees is the request as it arrived and the `Result` as the + * composition answered. A middleware beneath that transforms the path or the + * content is provider behavior rather than a second authored identity, so the + * received request stays the operation's name. + */ + +import { Err, Ok } from "effection"; +import type { Operation, Result } from "effection"; +import { isAbsolute, join, relative, resolve } from "node:path"; +import { realpath } from "node:fs/promises"; +import { until } from "effection"; +import { + FilesInvariantError, + fileWriteFailure, + filesFailure, + parseFilesFailure, + parseFileWriteFailure, + parseFileWriteSuccess, +} from "@executablemd/runtime"; +import type { + FilesHandler, + FilesOperation, + FilesPhase, + FileWriteSuccess, +} from "@executablemd/runtime"; +import type { Around } from "@effectionx/context-api"; +import { + localRecordName, + recordedLocalOperation, + retainedResultAhead, + retainedSelectionAhead, +} from "./local.ts"; +import type { + ReplAmbientCheckout, + ReplLocalOperation, + ReplLocalRequest, + ReplLocalValue, +} from "../../retained-repository/records.ts"; + +/** What the entry knows about where it is working and what it may hold. */ +export interface ReplLocalRecorder { + /** + * This entry's ambient routing, restored from the journal or discovered live. + * + * An operation because discovery is lazy: the first mutation is what looks for + * a checkout, and an entry that makes none never looks. + */ + selection(): Operation; + /** + * Hold the shared Git directory this entry is about to mutate in, or refuse. + * + * Asked once per entry and shared by every later request, so the lease belongs + * to the entry rather than to the element that first needed it. + */ + hold(commonDirectory: string): Operation; +} + +/** The middleware that records this entry's authored mutations. */ +export function replFileMiddleware(recorder: ReplLocalRecorder): Partial> { + return { + *readTextFile([input], next): Operation> { + const produced = yield* routed( + recorder, + "file-read", + { cwd: input.cwd, path: input.path }, + function* () { + return yield* textOutcome(next(input)); + }, + ); + return produced.ok ? Ok(expectText(produced.value)) : produced; + }, + *writeTextFile([input], next): Operation> { + const produced = yield* routed( + recorder, + "file-write", + { cwd: input.cwd, path: input.path, content: input.content }, + function* () { + return yield* writeOutcome(next(input)); + }, + ); + return produced.ok ? Ok(expectWritten(produced.value)) : produced; + }, + *deleteFile([input], next): Operation> { + const produced = yield* routed( + recorder, + "file-delete", + { cwd: input.cwd, path: input.path }, + function* () { + return yield* unitOutcome(next(input), { operation: "delete", phase: "access" }); + }, + ); + return produced.ok ? Ok(undefined) : produced; + }, + *ensureDirectory([input], next): Operation> { + const produced = yield* routed( + recorder, + "dir-ensure", + { cwd: input.cwd, path: input.path }, + function* () { + return yield* unitOutcome(next(input), { + operation: "ensure-directory", + phase: "access", + }); + }, + ); + return produced.ok ? Ok(undefined) : produced; + }, + }; +} + +/** + * Answer this request from the journal, or route it and perform it live. + * + * The retained cursor is read before the filesystem, because resolving a path is + * work a saved result is not about: a continuation whose checkout has since been + * removed answers from what it already recorded rather than failing at a + * question nobody asked. Saved routing is restored first for the same reason — + * it stands at the cursor ahead of the operation that discovered it. + */ +function* routed( + recorder: ReplLocalRecorder, + operation: ReplLocalOperation, + request: ReplLocalRequest, + perform: () => Operation>, +): Operation> { + const name = yield* localRecordName({ operation, request }); + if (yield* retainedSelectionAhead()) { + yield* recorder.selection(); + } + if (yield* retainedResultAhead(name)) { + return yield* recordedLocalOperation({ operation, request, perform }); + } + + const checkout = yield* covering(recorder, request); + if (checkout === undefined) { + return yield* perform(); + } + yield* recorder.hold(checkout.commonDirectory); + return yield* recordedLocalOperation({ operation, request, perform }); +} + +/** The checkout this request points inside, when it points inside one. */ +function* covering( + recorder: ReplLocalRecorder, + request: ReplLocalRequest, +): Operation { + const checkout = yield* recorder.selection(); + if (checkout === undefined) { + return undefined; + } + const target = yield* targetOf(request); + return inside(checkout.checkoutRoot, target) ? checkout : undefined; +} + +/** + * Where this request points, with the directory it is relative to resolved. + * + * The working directory is resolved through the filesystem rather than used as + * written, because a host's temporary root is commonly reached through a link — + * `/var` for `/private/var` on macOS — and a checkout discovered by its real + * path would not appear to contain a file addressed through the link. The file + * itself need not exist; its directory does. + */ +function* targetOf(request: ReplLocalRequest): Operation { + return resolve(join(yield* canonical(request.cwd), request.path)); +} + +function* canonical(directory: string): Operation { + try { + return yield* until(realpath(directory)); + } catch { + return resolve(directory); + } +} + +/** + * Whether this absolute path is somewhere inside the checkout. + * + * The root itself is not: it is not a file anybody writes and not a directory + * this entry creates, so a request naming it is the ordinary one. + */ +function inside(checkoutRoot: string, target: string): boolean { + const at = relative(checkoutRoot, target); + return at !== "" && !at.startsWith("..") && !isAbsolute(at); +} + +/** + * The write outcome the composition answered with, normalized before it is kept. + * + * Canonical core validates this same value again on the way out, and it is the + * authority: `packages/core/src/files.ts` decides what a settlement, a Unit + * success and an unrecognized failure mean. The readers below are its readers, + * because what arrives here is an arbitrary runtime value rather than the + * `Result` the signature promises — a `Proxy` can refuse to be read — and a + * wrapper that normalized differently would either record an outcome core is + * about to refuse or erase the invalid data core detects it by. + * + * A write's report is the one outcome that says something about the world, so + * both a malformed success and a malformed failure are fatal: a provider that + * cannot describe what it did may not have done it. + */ +function* writeOutcome(call: Operation>): Operation> { + const produced: unknown = yield* call; + const settled = settlement(produced); + if (settled === undefined) { + throw new FilesInvariantError("protocol"); + } + if (settled) { + const carried = read(produced, "value"); + if (carried === undefined) { + throw new FilesInvariantError("protocol"); + } + const success = parseFileWriteSuccess(carried.value); + if (success === undefined) { + throw new FilesInvariantError("protocol"); + } + return Ok(success); + } + const reported = read(produced, "error"); + if (reported === undefined) { + throw new FilesInvariantError("protocol"); + } + const data = parseFileWriteFailure(reported.value); + if (data === undefined) { + throw new FilesInvariantError("protocol"); + } + return Err( + fileWriteFailure({ + phase: data.phase, + ...(data.reason === undefined ? {} : { reason: data.reason }), + ...(data.cleanup === undefined ? {} : { cleanup: data.cleanup }), + }), + ); +} + +/** + * The read outcome the composition answered with, normalized before it is kept. + * + * Core's rules for a payload-carrying non-write operation: a readable boolean + * settlement, a `value` that is really a string, and — where it failed — the + * generic ordinary refusal for failure data this version does not recognize, + * because an unrecognized reason still described a failure and the document + * carries on past it. + */ +function* textOutcome(call: Operation>): Operation> { + const produced: unknown = yield* call; + const settled = settlement(produced); + if (settled === undefined) { + throw new FilesInvariantError("protocol"); + } + if (!settled) { + const reported = read(produced, "error"); + if (reported === undefined) { + throw new FilesInvariantError("protocol"); + } + const data = parseFilesFailure(reported.value); + if (data === undefined || data.operation !== "read") { + return Err(filesFailure({ operation: "read", phase: "access", reason: "operation-failed" })); + } + return Err(filesFailure({ operation: data.operation, phase: data.phase, reason: data.reason })); + } + const carried = read(produced, "value"); + if (carried === undefined || typeof carried.value !== "string") { + throw new FilesInvariantError("protocol"); + } + return Ok(carried.value); +} + +/** + * The outcome of the two operations that answer with nothing, normalized. + * + * Core's rules, kept exactly. A success carries no payload at all, so an + * *absent* `value` is the ordinary answer; one that cannot be read, and one + * that is present but is not `undefined`, are both provider-contract failures. + * + * A failure is where the two differ from a write, and the difference matters: an + * `error` that reads fine but carries data this version does not recognize *did* + * describe a failure, just not a known one, so it becomes the generic ordinary + * refusal and the document carries on. Only an `error` that is absent or + * unreadable is fatal. + */ +function* unitOutcome( + call: Operation>, + contract: { readonly operation: FilesOperation; readonly phase: FilesPhase }, +): Operation> { + const produced: unknown = yield* call; + const settled = settlement(produced); + if (settled === undefined) { + throw new FilesInvariantError("protocol"); + } + if (!settled) { + const reported = read(produced, "error"); + if (reported === undefined) { + throw new FilesInvariantError("protocol"); + } + const data = parseFilesFailure(reported.value); + if (data === undefined || data.operation !== contract.operation) { + return Err( + filesFailure({ + operation: contract.operation, + phase: contract.phase, + reason: "operation-failed", + }), + ); + } + return Err(filesFailure({ operation: data.operation, phase: data.phase, reason: data.reason })); + } + const carried = present(produced, "value"); + if (carried === undefined) { + throw new FilesInvariantError("protocol"); + } + if (carried) { + // Present, so it has to be readable *and* be the absent payload. Asking + // only whether the value is `undefined` would let a member that refused to + // be read pass as one that read as nothing. + const value = read(produced, "value"); + if (value === undefined || value.value !== undefined) { + throw new FilesInvariantError("protocol"); + } + } + return Ok(undefined); +} + +/** + * Whether a member is there at all, or `undefined` when asking threw. + * + * Core's reader, because this boundary reads the same provider-controlled + * values it does. Absence and unreadability are different answers and only one + * of them is ever legitimate, and a `has` trap can refuse the question. + */ +function present(target: unknown, name: string): boolean | undefined { + if (typeof target !== "object" || target === null) { + return undefined; + } + try { + return Reflect.has(target, name); + } catch { + return undefined; + } +} + +/** + * A member of a value this code did not create, boxed so that an absent member + * is distinguishable from one that could not be read. + */ +function read(target: unknown, name: string): { readonly value: unknown } | undefined { + if (present(target, name) !== true) { + return undefined; + } + try { + return { value: Reflect.get(Object(target), name) }; + } catch { + return undefined; + } +} + +/** + * Whether this container said it succeeded, as a boolean it really carries. + * + * Truthiness is not the question: a provider whose `ok` is a string has not + * described a settlement, and reading it as success would record an outcome + * core is about to refuse. + */ +function settlement(result: unknown): boolean | undefined { + const ok = read(result, "ok"); + return typeof ok?.value === "boolean" ? ok.value : undefined; +} + +/** + * The write success a recorded write answers with. + * + * A record of a write carries one by construction, so an absent payload here is + * this host's own invariant rather than anything a provider did. + */ +function expectWritten(value: ReplLocalValue): FileWriteSuccess { + if (value === undefined || typeof value === "string") { + throw new FilesInvariantError("protocol"); + } + return value; +} + +/** + * The text a recorded read answers with. + * + * A record of a read carries a string by construction, so anything else here is + * this host's own invariant rather than anything a provider did. + */ +function expectText(value: ReplLocalValue): string { + if (typeof value !== "string") { + throw new FilesInvariantError("protocol"); + } + return value; +} diff --git a/packages/git/src/deno/retained-repository/leases.ts b/packages/git/src/deno/retained-repository/leases.ts new file mode 100644 index 000000000..a2bf612e1 --- /dev/null +++ b/packages/git/src/deno/retained-repository/leases.ts @@ -0,0 +1,64 @@ +/** + * The one checkout a live REPL entry is working in, held or refused. + * + * Two entries moving files in one repository is the hazard, and one mechanism + * answers it: the kernel lock on a sidecar beside the repository. `flock` and + * `LockFileEx` both exclude *independently opened* handles — including two + * opened by one process — so a second entry here is refused by the operating + * system exactly as a second process is, and the native descriptor is the only + * live resource involved. Nothing else is needed: no table of who holds what, + * no acquisition token, and no state a caller could substitute. + * + * Refusing rather than waiting is what makes the answer useful. An entry that + * cannot have the repository it needs is told so before it reads a byte of it, + * and a person with two entries open gets a sentence rather than a stall. + * + * The sidecar is left in place. Unlinking a locked path lets the next caller + * create and lock a different file at the same name while this lock is still + * held. + */ + +import type { Operation } from "effection"; +import { StaleInputError } from "@executablemd/durable-streams"; +import { useAdvisoryLock } from "@executablemd/workflow/deno"; +import { commonDirectoryLock } from "../run-composition/placement.ts"; + +/** + * A checkout another live entry or another process is already working in. + * + * A `StaleInputError`, and raised before the mutation it would have guarded. + * Two things follow from that, and both are the point. A document cannot print + * it away and carry on, because a later sibling running as though the write had + * happened is exactly what the exclusion exists to prevent. And it never + * becomes a saved local refusal: a lease is busy only for as long as somebody + * else is working, so recording "refused" would make a temporary condition the + * permanent answer every later continuation replays. + */ +export class CommonDirectoryLeaseError extends StaleInputError { + override name = "CommonDirectoryLeaseError"; + + readonly reason = "checkout-in-use"; + + constructor() { + super( + "Another XMD entry or process holds this checkout. End that work before trying again. " + + "Nothing was read and nothing was changed.", + ); + } +} + +/** + * Hold the canonical common Git directory for the calling scope, or refuse. + * + * The caller reaches this through the entry owner's lazy single-flight + * acquisition, so one entry takes one handle however many mutations ask for it, + * and the handle lasts as long as the entry rather than as long as the element + * that first needed it. The release is registered before the descriptor is + * opened, inside the helper this delegates to. + */ +export function* useCommonDirectoryLease(commonDirectory: string): Operation { + const held = yield* useAdvisoryLock(commonDirectoryLock(commonDirectory)); + if (held === undefined) { + throw new CommonDirectoryLeaseError(); + } +} diff --git a/packages/git/src/deno/retained-repository/local.ts b/packages/git/src/deno/retained-repository/local.ts new file mode 100644 index 000000000..baa3181d5 --- /dev/null +++ b/packages/git/src/deno/retained-repository/local.ts @@ -0,0 +1,288 @@ +/** + * One local file operation, performed once and answered from its result. + * + * This is the whole of the REPL's local durability: an ordinary durable + * operation whose name is this entry's own, whose value is what the ordinary + * Files call produced, and whose replay is the durable engine's own positional + * matching. A completed operation answers with its saved result and touches no + * file, however the file has changed since. The first operation with no saved + * result runs live, against whatever the environment actually holds. + * + * Nothing is inventoried, predicted or compared. The window that honesty buys + * is explicit: native work can happen and its result can fail to be appended. + * The journal then holds no completion, so a continuation takes the ordinary + * live path and may attempt it again. That is normal durable-stream behavior + * rather than an exactly-once local effect, and it is what the REPL trades for + * not snapshotting anybody's working tree. + * + * ## Why this reads the cursor before it looks at the filesystem + * + * Deciding whether a path is inside the checkout means resolving it, and + * resolving it means touching a filesystem the saved result is not about — a + * continuation whose checkout has since been deleted would fail at the question + * rather than answer from what it already knows. So the retained cursor is read + * first, without consuming anything: a position holding this operation's own + * record is answered from the record, and only an operation with nothing saved + * resolves, routes, leases and performs. + * + * The read is a hint and never an authority. `createDurableOperation` remains + * the only consumer of the cursor and the only judge of position, type and name, + * so a peek that guessed wrong costs a wasted comparison and changes no outcome. + */ + +import { Err, Ok } from "effection"; +import type { Operation, Result } from "effection"; +import { createDurableOperation, DurableContext } from "@executablemd/durable-streams"; +import type { + ActivateDurabilityFailure, + Json, + LiveDurableOperationCoordinator, + Result as ProtocolResult, +} from "@executablemd/durable-streams"; +import { StaleInputError } from "@executablemd/durable-streams"; +import { asFilesFatal, FilesInvariantError } from "@executablemd/runtime"; +import { + parseReplLocalRecord, + parseReplSelectionRecord, + replLocalFingerprint, + replLocalRecordJson, + replLocalRecordName, + replSelectionFingerprint, + replSelectionRecordJson, + replSelectionRecordName, + REPL_REPOSITORY_LOCAL, + REPL_REPOSITORY_SELECTION, + retainableRefusal, +} from "../../retained-repository/records.ts"; +import type { + ReplAmbientCheckout, + ReplLocalOperation, + ReplLocalRequest, + ReplLocalValue, +} from "../../retained-repository/records.ts"; + +/** + * A saved record standing where this operation belongs that it cannot read. + * + * Fatal: a value that does not describe this operation is not an outcome of it, + * so it does not belong in the result channel. A refusal there is the + * operation's own sentence, which `` may print and carry on past. + */ +export class ReplLocalRecordError extends StaleInputError { + override name = "ReplLocalRecordError"; +} + +/** One local mutation, as this durable boundary needs it described. */ +export interface LocalOperation { + readonly operation: ReplLocalOperation; + /** What the document asked for. Names the record; never live state. */ + readonly request: ReplLocalRequest; + /** The ordinary composed call, run only when no result is saved. */ + perform(): Operation>; +} + +/** Where this entry's ambient routing comes from when none is saved. */ +export interface AmbientSelectionSource { + /** The directory discovery starts from. */ + readonly cwd: string; + /** The live discovery, run only when no routing is saved. */ + discover(): Operation; +} + +/** The name this operation's record is journaled under. */ +export function* localRecordName(operation: { + readonly operation: ReplLocalOperation; + readonly request: ReplLocalRequest; +}): Operation { + return replLocalRecordName({ + operation: operation.operation, + fingerprint: yield* replLocalFingerprint(operation), + }); +} + +/** + * Whether the next retained position in this coroutine holds this record. + * + * Reads the description alone and consumes nothing. A `true` says a saved + * result is probably standing here, which is reason enough to answer from the + * journal instead of resolving a path; the matcher below decides whether it + * really is. + */ +export function* retainedRecordAhead(type: string, name?: string): Operation { + const context = yield* DurableContext.expect(); + const entry = context.replayIndex.peekYield(context.coroutineId); + if (entry === undefined || entry.description.type !== type) { + return false; + } + return name === undefined || entry.description.name === name; +} + +/** + * Which durable coroutine is asking. + * + * A record belongs to a position in one coroutine, so anything this host + * remembers *about* records has to be remembered per coroutine too. An entry is + * one journal with several coroutines in it, and `` is how a document gets + * more than one. + */ +export function* currentCoroutine(): Operation { + const context = yield* DurableContext.expect(); + return context.coroutineId; +} + +/** Whether this entry's saved ambient routing stands at the cursor. */ +export function retainedSelectionAhead(): Operation { + return retainedRecordAhead(REPL_REPOSITORY_SELECTION); +} + +/** Whether this operation's own saved result stands at the cursor. */ +export function retainedResultAhead(name: string): Operation { + return retainedRecordAhead(REPL_REPOSITORY_LOCAL, name); +} + +/** + * This entry's ambient routing, restored from the journal or discovered live. + * + * Routing facts only: the request, the checkout root and the common Git + * directory, or their absence. Restoring them authenticates no checkout and + * acquires no lease — a continuation that reads them has learned where this + * entry was working, which is not permission to work there. + */ +export function selectedCheckout( + source: AmbientSelectionSource, +): Operation { + return selecting(source); +} + +function* selecting(source: AmbientSelectionSource): Operation { + const fingerprint = yield* replSelectionFingerprint({ cwd: source.cwd }); + const name = replSelectionRecordName({ fingerprint }); + + const stored: unknown = yield createDurableOperation( + { type: REPL_REPOSITORY_SELECTION, name }, + function* () { + return replSelectionRecordJson({ cwd: source.cwd, checkout: yield* source.discover() }); + }, + ); + + const parsed = parseReplSelectionRecord(stored); + if (parsed === undefined || parsed.cwd !== source.cwd) { + throw new ReplLocalRecordError( + "the saved routing for this entry is not one this host can read, so where it was working " + + "is unknown.", + ); + } + return parsed.checkout ?? undefined; +} + +/** + * Perform this operation, or answer with what performing it produced before. + * + * The result is the operation's own — the same value a live call produces and a + * replay restores — so a caller cannot tell from it which of the two happened, + * which is the point. + */ +export function recordedLocalOperation( + operation: LocalOperation, +): Operation> { + return recording(operation); +} + +function* recording(operation: LocalOperation): Operation> { + const fingerprint = yield* replLocalFingerprint({ + operation: operation.operation, + request: operation.request, + }); + const name = replLocalRecordName({ operation: operation.operation, fingerprint }); + + // Whatever comes back is journal data on a replay, so it is handed on as the + // unknown it is and parsed below rather than trusted as this code's own. + const stored: unknown = yield createDurableOperation( + { type: REPL_REPOSITORY_LOCAL, name }, + function* () { + const outcome = yield* operation.perform(); + if (!outcome.ok && !retainableRefusal(outcome.error)) { + // Not a refusal in the ordinary Files vocabulary, so not an outcome + // this host may write down as one. The coordinator below classifies it. + throw outcome.error; + } + return replLocalRecordJson({ + operation: operation.operation, + fingerprint, + request: operation.request, + outcome, + }); + }, + { coordinator: localOperationCoordinator }, + ); + + const parsed = parseReplLocalRecord(stored); + if (parsed === undefined) { + throw new ReplLocalRecordError( + `the saved result for "${operation.operation}" is not one this host can read, so what it ` + + "says about work already done is unknown.", + ); + } + if (parsed.operation !== operation.operation || parsed.fingerprint !== fingerprint) { + throw new ReplLocalRecordError( + `the saved result standing where "${operation.operation}" belongs describes another ` + + "operation, so this history does not say what happened here.", + ); + } + return parsed.outcome.ok ? Ok(parsed.outcome.value) : Err(parsed.outcome.error); +} + +/** + * The live boundary one local operation publishes through. + * + * The default coordinator serializes whatever was thrown and publishes it as + * this operation's outcome, which is the one thing that must not happen here: a + * provider that broke its contract may or may not have changed the file, so + * recording a completion for it would let a continuation skip work that never + * happened — and serializing it would lose the Files classification the engine + * and `` both read. + * + * So a throw is classified, activates the durability fence and is rethrown with + * its identity intact, publishing nothing. An already active failure wins over + * the new one: the first failure is the one that describes what went wrong. + * + * A valid outcome — success or an ordinary refusal — is published once and + * returned. A failing append is the engine's own persistence fence and is not + * caught here. + */ +const localOperationCoordinator: LiveDurableOperationCoordinator = { + *run( + execute: () => Operation, + publish: (result: ProtocolResult) => Operation, + activateFailure: ActivateDurabilityFailure, + ): Operation { + let value: T; + try { + value = yield* execute(); + } catch (error) { + throw activateFailure(classify(error)); + } + const result: ProtocolResult = { status: "ok", value }; + yield* publish(result); + return result; + }, +}; + +/** + * What this throw is, in the vocabulary the engine already fences on. + * + * The search order is core's own at the Files boundary: a durability failure + * keeps its identity, then an infrastructure failure keeps its data, and only + * something that is neither becomes a fresh `protocol` invariant carrying no + * cause, message or host value. A handler that threw an arbitrary object is + * precisely the case where nothing it produced can be trusted. + */ +function classify(error: unknown): Error { + if (error instanceof StaleInputError) { + return error; + } + if (error instanceof Error && asFilesFatal(error) !== undefined) { + return error; + } + return new FilesInvariantError("protocol"); +} diff --git a/packages/git/src/deno/retained-repository/provider.ts b/packages/git/src/deno/retained-repository/provider.ts new file mode 100644 index 000000000..880f56d7b --- /dev/null +++ b/packages/git/src/deno/retained-repository/provider.ts @@ -0,0 +1,140 @@ +/** + * One REPL entry's recording: what the entry executing right now installs. + * + * `run-composition/provider.ts` answers the same question for `xmd run`, and the + * two differ in exactly one way that matters here. An ordinary run's provider is + * installed per execution and remembers nothing; an entry's is installed per + * *entry* and retains what that entry did, so an entry interrupted partway + * through can be reopened and continue instead of repeating itself. + * + * It is established inside the scope the invocation already owns, through + * core's `Execution.init`, so everything it acquires belongs to that execution: + * the Git session, its isolation directory, the ambient discovery and the lease + * all end when the entry does. Establishing one while the command is still + * assembling itself would make a passive view — a cold history somebody is only + * reading — discover a checkout and take its lease, and a passive view reaches + * no initialization at all. + * + * ## What it is not + * + * This increment records authored and generated file writes, authored deletions + * and the directory an authored `` ensures, in the checkout the person is + * standing in. There is no managed checkout, no `` or `` + * selection, no local Git operation and nothing external. The components for + * those stay declared where they already are and refuse through their own + * unchanged missing-provider defaults. + * + * ## Installing it reaches nothing + * + * No directory, no lease, no `git`, no filesystem read. The ambient checkout is + * discovered by the first mutation that needs to know whether it is inside one, + * and an entry that mutates nothing never looks. Being outside a checkout is + * remembered as "there is none" rather than refused, so an entry writing files + * beside the document runs exactly as it would anywhere else. + */ + +import { useScope } from "effection"; +import type { Operation, Scope } from "effection"; +import { cwd, Files } from "@executablemd/runtime"; +import { denoRepositoryHost } from "../composition/host.ts"; +import type { RepositoryHost } from "../composition/host.ts"; +import { gitSession } from "../composition/git.ts"; +import type { GitSession } from "../composition/git.ts"; +import { discoverAmbientCheckout } from "../run-composition/ambient.ts"; +import { lazily } from "../run-composition/lazy.ts"; +import { useCommonDirectoryLease } from "./leases.ts"; +import type { ReplAmbientCheckout } from "../../retained-repository/records.ts"; +import { currentCoroutine, selectedCheckout } from "./local.ts"; +import { replFileMiddleware } from "./files.ts"; +import type { ReplLocalRecorder } from "./files.ts"; + +/** What a caller may state about one entry's recording. */ +export interface ReplRecordingOptions { + /** + * The host native Git runs through. + * + * Production states none and gets this platform's. A suite states one so it + * can drive this exact assembly without a `git` subprocess. + */ + readonly host?: RepositoryHost; +} + +/** + * Record this execution's file work in the checkout it is standing in. + * + * Called once per executing entry from core initialization, inside the scope + * the invocation owns. It installs middleware and nothing else: the directory + * discovery starts from is read here through the contextual environment Api + * rather than from the process, because a session opened inside a checkout is + * working in that one, and a nested execution that composed its own working + * directory is discovered from that. + */ +export function* useReplRepositoryRecording(options: ReplRecordingOptions = {}): Operation { + const host = options.host ?? denoRepositoryHost(); + const directory = yield* cwd(); + + // The entry task. Every capability below is acquired into it, so the session, + // its directory and the lease last as long as the entry does rather than as + // long as whichever element first needed them. + const entry: Scope = yield* useScope(); + + // The Git session's root is also `HOME`, so Git reads no configuration + // belonging to whoever is running the command. + const useGit = lazily(entry, function* (): Operation { + return gitSession(host, yield* host.useDirectory()); + }); + + const discover = lazily(entry, function* (): Operation { + return yield* discoverAmbientCheckout(yield* useGit(), directory); + }); + + // One lazy acquisition per shared Git directory, so the first mutation takes + // the lease and every later one in this entry reuses it. A borrower cancelled + // while waiting abandons its own wait and not the hold. Another live entry + // opens its own descriptor and is refused by the operating system. + const leases = new Map Operation>(); + + // Remembered per durable coroutine, absence included, because that is the + // grain the record has: the routing a coroutine saves sits at a position in + // *that* coroutine, and replay consumes it from there. + // + // Remembering it per entry instead is a defect, and a reproduced one. Two + // `` branches can both reach their first mutation before either has + // answered, so each saves routing at its own position — valid history. On + // continuation the branch that replayed second would find the entry already + // "knowing" where it was working, skip its own saved routing, and leave the + // cursor pointing at a selection where its next result belongs. The journal + // then refuses the whole continuation as divergent, and the work after the + // questions never runs. + // + // What stays shared is the live discovery below, which is the part that costs + // a `git` invocation. + const selected = new Map(); + + const recorder: ReplLocalRecorder = { + *selection(): Operation { + const coroutine = yield* currentCoroutine(); + const remembered = selected.get(coroutine); + if (remembered !== undefined) { + return remembered.checkout; + } + const checkout = yield* selectedCheckout({ cwd: directory, discover }); + selected.set(coroutine, { checkout }); + return checkout; + }, + *hold(commonDirectory: string): Operation { + let acquire = leases.get(commonDirectory); + if (acquire === undefined) { + acquire = lazily(entry, function* (): Operation { + yield* useCommonDirectoryLease(commonDirectory); + }); + leases.set(commonDirectory, acquire); + } + yield* acquire(); + }, + }; + + // The outermost band, so what is recorded is the result the whole composition + // answered with rather than one layer's own. + yield* Files.around(replFileMiddleware(recorder), { at: "max" }); +} diff --git a/packages/git/src/deno/run-composition/ambient.ts b/packages/git/src/deno/run-composition/ambient.ts index d46917bb9..99aceb61b 100644 --- a/packages/git/src/deno/run-composition/ambient.ts +++ b/packages/git/src/deno/run-composition/ambient.ts @@ -86,16 +86,30 @@ function* canonical(path: string): Operation { } } +/** Which checkout a directory is in, and which repository that checkout belongs to. */ +export interface AmbientCheckout { + /** The canonical root of the checkout the invocation started in. */ + readonly checkoutRoot: string; + /** The canonical common Git directory, which identifies the repository. */ + readonly commonDirectory: string; +} + /** - * Discover the ambient repository from this directory, or answer `undefined`. + * Which checkout this directory is in, or `undefined` when it is in none. + * + * The two identities and nothing else. A caller that only needs to know where + * it is standing — which is what a file mutation needs — asks no question about + * HEAD, the origin, the branch or a publication destination, so a directory + * whose commit this host cannot resolve is still a checkout a file may be + * written in. * * Every step is a local Git question. Nothing here contacts a remote, opens a * credential or writes anything. */ -export function* discoverAmbientRepository( +export function* discoverAmbientCheckout( git: GitSession, from: string, -): Operation { +): Operation { const reportedRoot = yield* git.read(["rev-parse", "--show-toplevel"], from); if (reportedRoot === undefined) { return undefined; @@ -118,6 +132,28 @@ export function* discoverAmbientRepository( if (commonDirectory === undefined) { return undefined; } + return Object.freeze({ checkoutRoot, commonDirectory }); +} + +/** + * Discover the ambient repository from this directory, or answer `undefined`. + * + * Everything a Repository identity needs on top of the checkout: the object + * format, the commit HEAD named, the admitted origin and the branch a pull + * request defaults its base to. + * + * Every step is a local Git question. Nothing here contacts a remote, opens a + * credential or writes anything. + */ +export function* discoverAmbientRepository( + git: GitSession, + from: string, +): Operation { + const found = yield* discoverAmbientCheckout(git, from); + if (found === undefined) { + return undefined; + } + const { checkoutRoot, commonDirectory } = found; const objectFormat = yield* readObjectFormat(git, checkoutRoot); const head = yield* resolveCommit(git, checkoutRoot, "HEAD"); diff --git a/packages/git/src/deno/run-composition/lazy.ts b/packages/git/src/deno/run-composition/lazy.ts new file mode 100644 index 000000000..b770315e2 --- /dev/null +++ b/packages/git/src/deno/run-composition/lazy.ts @@ -0,0 +1,58 @@ +/** + * One capability, acquired at most once, owned by the scope that installed it. + * + * Shared by the two native providers that hold environment-dependent state for + * longer than the element that first needed it: the ordinary run's repository + * provider and a REPL entry's. Installing either must reach nothing — no + * directory, no lease, no `git`, no identity read — so what each capability + * costs is paid by the first operation that actually needs it, and by nothing + * else. + * + * Four properties, and each one is why this is not a plain memoized function: + * + * - **Single-flight.** The first ask starts the acquisition; an ask that + * arrives while it is still running awaits the same task rather than starting + * a second. Two concurrent elements share one answer. + * - **Owned by the installation scope.** The work runs in `owner`, not in + * whichever element happened to ask first, and then *suspends* — because a + * task that returns releases what it acquired. A temporary directory, a + * session and a lease therefore last as long as the owner does, rather than + * as long as the element that first needed them. + * - **Settled answers are answers.** An absent ambient repository and a host + * that cannot state a commit identity are values, not failures, and are + * remembered as such: the second ask does not rediscover them. + * - **Nothing half-made is observable.** A caller cancelled while waiting + * abandons its own wait and not the acquisition; a failed acquisition is + * remembered as failed, and every later ask is told the same thing rather + * than retrying into partially made state. + */ + +import { race, suspend, withResolvers } from "effection"; +import type { Operation, Scope, WithResolvers } from "effection"; + +export function lazily(owner: Scope, acquire: () => Operation): () => Operation { + let settled: + | { readonly ready: WithResolvers; readonly failed: WithResolvers } + | undefined; + return function* (): Operation { + if (settled === undefined) { + const ready = withResolvers(); + const failed = withResolvers(); + settled = { ready, failed }; + owner.run(function* () { + let value: T; + try { + value = yield* acquire(); + } catch (error) { + failed.reject(error instanceof Error ? error : new Error(String(error))); + return; + } + ready.resolve(value); + // Held for the scope's life. Returning here would end the task, and + // ending a task releases everything it acquired. + yield* suspend(); + }); + } + return yield* race([settled.ready.operation, settled.failed.operation]); + }; +} diff --git a/packages/git/src/deno/run-composition/placement.ts b/packages/git/src/deno/run-composition/placement.ts index a9aecdfcc..a1bf26c51 100644 --- a/packages/git/src/deno/run-composition/placement.ts +++ b/packages/git/src/deno/run-composition/placement.ts @@ -23,6 +23,7 @@ */ import { createHash } from "node:crypto"; +import { tmpdir } from "node:os"; /** The layout, relative to whichever root the entrypoint chose. */ export const REPOSITORIES = "repositories"; @@ -81,3 +82,24 @@ export function metadataOf(slot: string): string { export function lockOf(root: string, kind: "repository" | "worktree", slot: string): string { return `${root}/${LOCKS}/${kind}/${digest(slot)}.lock`; } + +/** Where the locks keyed on a shared Git directory rather than on a slot live. */ +export const COMMON_DIRECTORIES = "xmd-common-lanes"; + +/** + * The lock sidecar for one shared Git directory. + * + * Outside both the checkout and the managed root, because the directory this + * protects is the person's own: a lock file written inside a checkout would be + * an untracked file appearing in their status, and one under the managed root + * would tie an ambient checkout's exclusion to a root an ambient run never + * creates. The host's temporary directory is where a lock that belongs to a + * running process belongs, and the digest is of the canonical common directory, + * so no arrangement of path characters names another repository's lock. + * + * Nothing unlinks it. Unlinking a locked path lets the next caller create and + * lock a different file at the same name while this lock is still held. + */ +export function commonDirectoryLock(commonDirectory: string): string { + return `${tmpdir()}/${COMMON_DIRECTORIES}/${digest(commonDirectory)}.lock`; +} diff --git a/packages/git/src/deno/run-composition/provider.ts b/packages/git/src/deno/run-composition/provider.ts index f661bad2a..2b73c1c30 100644 --- a/packages/git/src/deno/run-composition/provider.ts +++ b/packages/git/src/deno/run-composition/provider.ts @@ -39,7 +39,7 @@ * first operation that needs it. */ -import { race, suspend, useScope, withResolvers } from "effection"; +import { useScope } from "effection"; import type { Operation, Scope } from "effection"; import { randomUUID } from "node:crypto"; @@ -119,13 +119,11 @@ import { type RegisteredCheckout, } from "./operations.ts"; import { repositorySlot, worktreeSlot } from "./placement.ts"; +import { lazily } from "./lazy.ts"; import { realpath } from "node:fs/promises"; import { ensureDir } from "@effectionx/fs"; import { until } from "effection"; -/** What `withResolvers()` hands back, named so a capability can hold a pair. */ -type Resolvers = ReturnType>; - /** The canonical directory this path resolves to, or the path as written. */ function* canonicalPath(path: string): Operation { try { @@ -166,57 +164,6 @@ interface SelectedRepository { readonly commonDirectory: string; } -/** - * One capability, acquired at most once, owned by the installation scope. - * - * Installing this provider must reach nothing: no directory, no lease, no - * `git`, no identity read. What each capability costs is paid by the first - * operation that actually needs it, and by nothing else — a document that - * writes no repository component pays none of it. - * - * Four properties, and each one is why this is not a plain memoized function: - * - * - **Single-flight.** The first ask starts the acquisition; an ask that - * arrives while it is still running awaits the same task rather than starting - * a second. Two concurrent `` elements share one managed root. - * - **Owned by the installation scope.** The work runs in `owner`, not in - * whichever element happened to ask first, and then *suspends* — because a - * task that returns releases what it acquired. A temporary directory, a - * session and a lease therefore last as long as the execution does, rather - * than as long as the element that first needed them. - * - **Settled answers are answers.** An absent ambient repository and a host - * that cannot state a commit identity are values, not failures, and are - * remembered as such: the second ask does not rediscover them. - * - **Nothing half-made is observable.** A caller cancelled while waiting - * abandons its own wait and not the acquisition; a failed acquisition is - * remembered as failed, and every later ask is told the same thing rather - * than retrying into partially made state. - */ -function lazily(owner: Scope, acquire: () => Operation): () => Operation { - let settled: { readonly ready: Resolvers; readonly failed: Resolvers } | undefined; - return function* (): Operation { - if (settled === undefined) { - const ready = withResolvers(); - const failed = withResolvers(); - settled = { ready, failed }; - owner.run(function* () { - let value: T; - try { - value = yield* acquire(); - } catch (error) { - failed.reject(error instanceof Error ? error : new Error(String(error))); - return; - } - ready.resolve(value); - // Held for the scope's life. Returning here would end the task, and - // ending a task releases everything it acquired. - yield* suspend(); - }); - } - return yield* race([settled.ready.operation, settled.failed.operation]); - }; -} - /** * Install the ordinary repository vocabulary for the current scope and below. * diff --git a/packages/git/src/plugin.ts b/packages/git/src/plugin.ts index 3f30b2e85..85ce6bb56 100644 --- a/packages/git/src/plugin.ts +++ b/packages/git/src/plugin.ts @@ -28,8 +28,13 @@ import type { Operation } from "effection"; import { Plugin } from "@executablemd/core/api"; -import type { PluginInstallRequest, PluginInstallation } from "@executablemd/core/api"; +import type { + JournalAdmission, + PluginInstallRequest, + PluginInstallation, +} from "@executablemd/core/api"; import { useCompositionComponents } from "./composition/installation.ts"; +import { admitReplRepositoryHistory } from "./retained-repository/admission.ts"; import { gitHostIdentityAdmission, issueIdentityAdmission } from "./identities.ts"; /** @@ -192,8 +197,32 @@ export const gitPlugin: Plugin = Plugin({ // provider that performs what they name is attached by the host, lazily, // when a run actually has a Workspace to perform it in. yield* useCompositionComponents(); - return { admissions: [gitHostIdentityAdmission(), issueIdentityAdmission()] }; + return { + admissions: [ + gitHostIdentityAdmission(), + issueIdentityAdmission(), + // The REPL's own: what an entry retained about the checkout it was + // standing in has to be readable before any of it is replayed. It + // belongs to this contribution because an admission is applied inside + // each execution's journal read, and this is the trusted path that + // already declares for the command. It holds no native state and names + // no runtime: a Node session that retained nothing has nothing here to + // refuse. + ...(request.command === "repl" ? [replHistoryAdmission()] : []), + ], + }; }, }); +/** + * What a REPL entry requires of the history it is about to continue. + * + * Provider-neutral, and about integrity alone: every record of this feature's + * own types parses and is named by the request it carries. Which entry a record + * belongs to is the stream segment's question, not this one's. + */ +function replHistoryAdmission(): JournalAdmission { + return (retained) => admitReplRepositoryHistory(retained); +} + export default gitPlugin; diff --git a/packages/git/src/retained-repository/admission.ts b/packages/git/src/retained-repository/admission.ts new file mode 100644 index 000000000..0f94be0af --- /dev/null +++ b/packages/git/src/retained-repository/admission.ts @@ -0,0 +1,161 @@ +/** + * Whether one entry's retained history may be continued at all. + * + * Two questions, and only two. **Integrity**: every record of this feature's + * own types parses and is named by the request it carries. **Nothing else**: no + * checkout is inspected, no saved path is compared with the live one, and no + * result is handed out — replaying a completed operation is the durable engine's + * own positional matching, and this runs before any of it. + * + * Which entry a record belongs to is not asked here, because it is not this + * reader's question: an entry executes against its own `EntrySegmentStream` + * range, so the only records reaching this snapshot are the ones that range + * holds. A second label inside the record would be a second answer to a + * question the stream has already answered. + * + * A history that disagrees with the directory it was recorded in is an ordinary + * situation rather than a refusal. The person reading the screen resolves that. + * What is refused is a history this host cannot read as its own, because a + * record whose request was changed after the work was saved would let a saved + * result answer a question nobody asked. + * + * Repeated equal names are valid. One entry may write the same file with the + * same content twice, and those are two records at two ordered positions; a + * reader that deduplicated them by name would refuse a document that is allowed. + */ + +import { StaleInputError } from "@executablemd/durable-streams"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import type { Operation } from "effection"; +import { + parseReplLocalRecord, + parseReplSelectionRecord, + replLocalFingerprint, + replLocalRecordName, + replSelectionFingerprint, + replSelectionRecordName, + REPL_REPOSITORY_LOCAL, + REPL_REPOSITORY_SELECTION, + REPL_REPOSITORY_VERSION, + statedReplRecordVersion, +} from "./records.ts"; + +/** Why a history cannot be continued. */ +export type ReplRepositoryRefusal = + /** A record of this feature's type that this version cannot read. */ + | "malformed-record" + /** A record stating a version this host does not implement. */ + | "unknown-version" + /** A record whose name is not the name of the request it carries. */ + | "request-fingerprint"; + +/** + * A history this host cannot continue from. + * + * Fatal by construction: it extends `StaleInputError`, so it ends the attempt + * rather than becoming a value a document can print away and carry on past. + */ +export class ReplRepositoryHistoryError extends StaleInputError { + override name = "ReplRepositoryHistoryError"; + + readonly reason: ReplRepositoryRefusal; + + constructor(reason: ReplRepositoryRefusal, sentence: string) { + super(sentence); + this.reason = reason; + } +} + +/** + * Hold this entry's retained history to what this host can read, or refuse. + * + * `retained` is the exact snapshot canonical execution admitted, which for an + * entry is its own segment. Records of other features and of other durable + * types are not this reader's business and are passed over; a record of *this* + * feature's type that does not parse is, and refuses. + */ +export function* admitReplRepositoryHistory(retained: readonly DurableEvent[]): Operation { + for (const event of retained) { + if (event.type !== "yield") { + continue; + } + const type = event.description.type; + if (type !== REPL_REPOSITORY_LOCAL && type !== REPL_REPOSITORY_SELECTION) { + continue; + } + if (event.result.status !== "ok") { + continue; + } + const payload = event.result.value; + const stated = statedReplRecordVersion(payload); + if (stated !== undefined && stated !== REPL_REPOSITORY_VERSION) { + throw refusal( + "unknown-version", + `a ${type} record states version ${stated}, and this host implements version ` + + `${REPL_REPOSITORY_VERSION} only. Nothing is assumed about what the rest of it means.`, + ); + } + if (type === REPL_REPOSITORY_SELECTION) { + yield* admitSelection(payload, event.description.name); + continue; + } + yield* admitLocal(payload, event.description.name); + } +} + +function* admitLocal(payload: unknown, name: string): Operation { + const record = parseReplLocalRecord(payload); + if (record === undefined) { + throw malformed(REPL_REPOSITORY_LOCAL); + } + const fingerprint = yield* replLocalFingerprint({ + operation: record.operation, + request: record.request, + }); + if (record.fingerprint !== fingerprint) { + throw changed(REPL_REPOSITORY_LOCAL); + } + if (name !== replLocalRecordName({ operation: record.operation, fingerprint })) { + throw misnamed(REPL_REPOSITORY_LOCAL); + } +} + +function* admitSelection(payload: unknown, name: string): Operation { + const record = parseReplSelectionRecord(payload); + if (record === undefined) { + throw malformed(REPL_REPOSITORY_SELECTION); + } + const fingerprint = yield* replSelectionFingerprint({ cwd: record.cwd }); + if (name !== replSelectionRecordName({ fingerprint })) { + throw misnamed(REPL_REPOSITORY_SELECTION); + } +} + +function malformed(type: string): ReplRepositoryHistoryError { + return refusal( + "malformed-record", + `a ${type} record in this entry's history is not one this host can read, so what it says ` + + "about work already done is unknown.", + ); +} + +function changed(type: string): ReplRepositoryHistoryError { + return refusal( + "request-fingerprint", + `a retained ${type} carries a fingerprint that is not the digest of the request beside it, ` + + "so the request was changed after the work was saved and this history cannot say which of " + + "the two the saved work belongs to.", + ); +} + +function misnamed(type: string): ReplRepositoryHistoryError { + return refusal( + "request-fingerprint", + `a retained ${type} is journaled under a name that is not the name its own request ` + + "produces, so the record and the operation it claims to be do not match.", + ); +} + +function refusal(reason: ReplRepositoryRefusal, sentence: string): ReplRepositoryHistoryError { + return new ReplRepositoryHistoryError(reason, sentence); +} diff --git a/packages/git/src/retained-repository/records.ts b/packages/git/src/retained-repository/records.ts new file mode 100644 index 000000000..891f68257 --- /dev/null +++ b/packages/git/src/retained-repository/records.ts @@ -0,0 +1,430 @@ +/** + * What one REPL entry's local file and directory work looks like on the wire. + * + * A foreground REPL entry that writes a file inside the checkout the person is + * standing in records what the write produced. Reopen the entry and the saved + * result answers again, without touching the file — which is what lets an entry + * interrupted at a question be continued instead of repeated. + * + * Four operations are recorded and nothing else: a `` text read, a + * `` write, a ``, and the directory a `` ensures. A + * call outside the checkout is ordinary filesystem behavior and is recorded by + * nothing here, because giving it a record would place it in a checkout it is not in. + * Globs, path checks and temporary directories are recorded nowhere: a read is + * retained because a document binds its text and a continuation has to offer + * the same binding, and those three bind nothing. + * + * ## Identity is the request, and position is the journal's + * + * A record's name is the digest of the record version, the operation kind and + * the complete authored request — the working directory, the path, and for a + * write the exact content. Two entries that write the same text produce the same + * name, and that is allowed: each entry executes against its own segment of the + * one physical history, so a name another entry wrote is never inside the range + * this one replays. Two *equal* writes in one entry are one name at two ordered + * positions, and telling them apart is the durable engine's own positional + * matching rather than anything here: nothing in this module searches history by + * name, and a repeated name is valid. + * + * The request is named as the document wrote it. Nothing is canonicalized for + * naming, because a path resolved through the filesystem is a fact about the + * host at that moment and the question the document asked is not. + * + * ## What a result may carry + * + * The ordinary Files vocabulary and no more. A write keeps the validated + * success it was given, `publication` included; a delete and an ensure keep the + * absence of a payload that those operations answer with, rather than a write + * receipt they never produced. A refusal keeps the structural failure data the + * provider reported, so a continuation reconstructs the same `Result` the live + * call returned. + * + * Nothing else is retained — no native exception, no credential, no directory + * listing, no bytes read from another file and no handle. + */ + +import { Err, Ok, until } from "effection"; +import type { Operation, Result } from "effection"; +import type { Json } from "@executablemd/durable-streams"; +import { canonicalJson } from "@executablemd/workflow"; +import { + FILES_ERROR_MESSAGE, + FilesError, + parseFilesFailure, + parseFileWriteFailure, + parseFileWriteSuccess, +} from "@executablemd/runtime"; +import type { FilesErrorData, FileWriteSuccess } from "@executablemd/runtime"; +import { members } from "../composition/parse.ts"; + +/** The record shape this host writes and the only one it reads. */ +export const REPL_REPOSITORY_VERSION = 1; + +/** The durable type one completed local operation is journaled under. */ +export const REPL_REPOSITORY_LOCAL = "repl_repository_local"; + +/** The durable type the entry's ambient routing facts are journaled under. */ +export const REPL_REPOSITORY_SELECTION = "repl_repository_selection"; + +const RECORD_PREFIX = "executablemd.git.repl-repository"; + +/** Which local operation a record describes. */ +export type ReplLocalOperation = "file-read" | "file-write" | "file-delete" | "dir-ensure"; + +const LOCAL_OPERATIONS: readonly ReplLocalOperation[] = [ + "file-read", + "file-write", + "file-delete", + "dir-ensure", +]; + +/** + * What one recorded operation produced. + * + * A write's validated receipt, a read's exact text, or the absence a delete and + * an ensure answer with. Nothing else is a value this format carries. + */ +export type ReplLocalValue = FileWriteSuccess | string | undefined; + +/** What one recorded call asked for. */ +export interface ReplLocalRequest { + /** The contextual working directory the path is relative to. */ + readonly cwd: string; + /** The path exactly as the document wrote it. */ + readonly path: string; + /** The exact expanded content a write asked for; absent for the other three. */ + readonly content?: string; +} + +/** The routing facts the entry's ambient discovery found, or their absence. */ +export interface ReplAmbientCheckout { + /** The canonical root of the checkout the entry started in. */ + readonly checkoutRoot: string; + /** The canonical common Git directory, which identifies the repository. */ + readonly commonDirectory: string; +} + +/** One completed local operation, as a continuation reads it back. */ +export interface ReplLocalRecord { + readonly version: typeof REPL_REPOSITORY_VERSION; + readonly operation: ReplLocalOperation; + /** The digest of the version, operation kind and authored request. */ + readonly fingerprint: string; + readonly request: ReplLocalRequest; + /** + * What performing it produced, as the ordinary `Result` it was. + * + * A write carries the validated success it was given; a read carries the exact + * text it answered with; a delete and an ensure carry the absence those + * operations answer with. A refusal carries a `FilesError` rebuilt from the + * structural data the provider reported. + */ + readonly outcome: Result; +} + +/** The entry's saved ambient routing, as a continuation reads it back. */ +export interface ReplSelectionRecord { + readonly version: typeof REPL_REPOSITORY_VERSION; + /** The directory discovery started from. */ + readonly cwd: string; + /** What it found, or `null` for a directory inside no checkout. */ + readonly checkout: ReplAmbientCheckout | null; +} + +/** The stable name one local operation's record is journaled under. */ +export function replLocalRecordName(record: { + readonly operation: ReplLocalOperation; + readonly fingerprint: string; +}): string { + return canonicalJson([ + RECORD_PREFIX, + "local", + REPL_REPOSITORY_VERSION, + record.operation, + record.fingerprint, + ]); +} + +/** The stable name the entry's ambient routing record is journaled under. */ +export function replSelectionRecordName(record: { readonly fingerprint: string }): string { + return canonicalJson([RECORD_PREFIX, "selection", REPL_REPOSITORY_VERSION, record.fingerprint]); +} + +/** The digest of everything one authored mutation asks. */ +export function replLocalFingerprint(input: { + readonly operation: ReplLocalOperation; + readonly request: ReplLocalRequest; +}): Operation { + return digest({ + version: REPL_REPOSITORY_VERSION, + operation: input.operation, + request: requestJson(input.request), + }); +} + +/** The digest of the directory this entry's discovery starts from. */ +export function replSelectionFingerprint(input: { readonly cwd: string }): Operation { + return digest({ version: REPL_REPOSITORY_VERSION, selection: { cwd: input.cwd } }); +} + +/** The payload one completed local operation is journaled with. */ +export function replLocalRecordJson(input: { + readonly operation: ReplLocalOperation; + readonly fingerprint: string; + readonly request: ReplLocalRequest; + readonly outcome: Result; +}): Json { + const stated = { + version: REPL_REPOSITORY_VERSION, + operation: input.operation, + fingerprint: input.fingerprint, + request: requestJson(input.request), + }; + if (input.outcome.ok) { + const success = input.outcome.value; + if (success === undefined) { + return { ...stated, success: null }; + } + return { ...stated, success: typeof success === "string" ? success : { ...success } }; + } + return { ...stated, refusal: { ...failureData(input.outcome.error) } }; +} + +/** The payload this entry's ambient routing is journaled with. */ +export function replSelectionRecordJson(input: { + readonly cwd: string; + readonly checkout: ReplAmbientCheckout | undefined; +}): Json { + return { + version: REPL_REPOSITORY_VERSION, + cwd: input.cwd, + checkout: + input.checkout === undefined + ? null + : { + checkoutRoot: input.checkout.checkoutRoot, + commonDirectory: input.checkout.commonDirectory, + }, + }; +} + +/** + * The local operation this saved value describes, or `undefined`. + * + * Total, and exact about its members: a payload carrying anything this version + * did not write is not a record it can act on, because the operation may have + * happened and may not. The caller refuses rather than guessing. + */ +export function parseReplLocalRecord(value: unknown): ReplLocalRecord | undefined { + const refusal = members(value, ["version", "operation", "fingerprint", "request", "refusal"]); + const success = + refusal === undefined + ? members(value, ["version", "operation", "fingerprint", "request", "success"]) + : undefined; + const read = refusal ?? success; + if (read === undefined) { + return undefined; + } + if (read["version"] !== REPL_REPOSITORY_VERSION) { + return undefined; + } + const fingerprint = string(read["fingerprint"]); + const operation = parseReplLocalOperation(read["operation"]); + if (fingerprint === undefined || operation === undefined) { + return undefined; + } + const request = parseReplLocalRequest(read["request"], operation); + if (request === undefined) { + return undefined; + } + const outcome = + refusal === undefined + ? parseSuccess(read["success"], operation) + : parseRefusal(read["refusal"]); + if (outcome === undefined) { + return undefined; + } + return Object.freeze({ + version: REPL_REPOSITORY_VERSION, + operation, + fingerprint, + request, + outcome, + }); +} + +/** The ambient routing this saved value describes, or `undefined`. */ +export function parseReplSelectionRecord(value: unknown): ReplSelectionRecord | undefined { + const read = members(value, ["version", "cwd", "checkout"]); + if (read === undefined || read["version"] !== REPL_REPOSITORY_VERSION) { + return undefined; + } + const cwd = string(read["cwd"]); + if (cwd === undefined) { + return undefined; + } + const stated = read["checkout"]; + if (stated === null) { + return Object.freeze({ version: REPL_REPOSITORY_VERSION, cwd, checkout: null }); + } + const checkout = members(stated, ["checkoutRoot", "commonDirectory"]); + if (checkout === undefined) { + return undefined; + } + const checkoutRoot = string(checkout["checkoutRoot"]); + const commonDirectory = string(checkout["commonDirectory"]); + if (checkoutRoot === undefined || commonDirectory === undefined) { + return undefined; + } + return Object.freeze({ + version: REPL_REPOSITORY_VERSION, + cwd, + checkout: Object.freeze({ checkoutRoot, commonDirectory }), + }); +} + +/** The operation kind this value names, or `undefined`. */ +export function parseReplLocalOperation(value: unknown): ReplLocalOperation | undefined { + return LOCAL_OPERATIONS.find((operation) => operation === value); +} + +/** The version a payload states, or `undefined` when it states none. */ +export function statedReplRecordVersion(payload: unknown): number | undefined { + if (typeof payload !== "object" || payload === null || Array.isArray(payload)) { + return undefined; + } + const read = Reflect.get(payload, "version"); + return typeof read === "number" && Number.isSafeInteger(read) ? read : undefined; +} + +/** + * The authored request a saved record carries, or `undefined`. + * + * The content and the operation have to agree. A write without content, or a + * delete carrying some, is a record describing an operation this host does not + * perform, which says nothing trustworthy about work already done. + */ +function parseReplLocalRequest( + value: unknown, + operation: ReplLocalOperation, +): ReplLocalRequest | undefined { + const read = + operation === "file-write" + ? members(value, ["cwd", "path", "content"]) + : members(value, ["cwd", "path"]); + if (read === undefined) { + return undefined; + } + const cwd = string(read["cwd"]); + const path = anyString(read["path"]); + if (cwd === undefined || path === undefined) { + return undefined; + } + if (operation !== "file-write") { + return Object.freeze({ cwd, path }); + } + const content = anyString(read["content"]); + return content === undefined ? undefined : Object.freeze({ cwd, path, content }); +} + +/** + * The success a saved record carries, as the `Result` the operation produced. + * + * A write's success is the validated outcome data it was given, and a read's is + * the exact text. The other two answer with nothing at all, and `null` is how + * that is written down: a receipt manufactured for them would state what the + * operation never did. + */ +function parseSuccess( + value: unknown, + operation: ReplLocalOperation, +): Result | undefined { + if (operation === "file-read") { + return typeof value === "string" ? Ok(value) : undefined; + } + if (operation !== "file-write") { + return value === null ? Ok(undefined) : undefined; + } + const success = parseFileWriteSuccess(value); + return success === undefined ? undefined : Ok(success); +} + +/** The refusal a saved record carries, rebuilt as the failure it was. */ +function parseRefusal(value: unknown): Result | undefined { + const data = failureOf(carried(value)); + return data === undefined ? undefined : Err(new FilesError(data)); +} + +/** + * The structural data behind one ordinary failure. + * + * Read off the error and validated before it is written down: an error this + * boundary cannot describe is not a refusal it may save as one, so the caller's + * provider-contract classification is what reports it instead. + */ +function failureData(error: Error): FilesErrorData { + const data = failureOf(error); + if (data === undefined) { + throw new TypeError("a retained refusal carries no ordinary Files failure data"); + } + return data; +} + +/** Whether this error is one the record format can carry as a refusal. */ +export function retainableRefusal(error: Error): boolean { + return failureOf(error) !== undefined; +} + +/** + * The validated failure data one error carries, write or otherwise. + * + * A write's data is asked for first because its rules are the stricter pair: a + * phase, the target that phase implies, and which of a reason and a cleanup + * reason may appear beside them. + */ +function failureOf(error: unknown): FilesErrorData | undefined { + return parseFileWriteFailure(error) ?? parseFilesFailure(error); +} + +/** + * Saved failure data on a carrier, for the validators that read an error. + * + * The runtime's two validators take the error rather than the record, because + * at every other boundary the error is what they are handed. Carrying saved data + * on a plain one keeps them the single authority on whether a combination is one + * a consumer may read — a second implementation of the write phase and target + * rules beside them would be a second answer to the same question. + */ +function carried(data: unknown): Error { + return Object.assign(new Error(FILES_ERROR_MESSAGE), { data }); +} + +function requestJson(request: ReplLocalRequest): Json { + return request.content === undefined + ? { cwd: request.cwd, path: request.path } + : { cwd: request.cwd, path: request.path, content: request.content }; +} + +function* digest(value: Json): Operation { + const encoded = new TextEncoder().encode(canonicalJson(value)); + const computed = yield* until(crypto.subtle.digest("SHA-256", encoded)); + return Array.from(new Uint8Array(computed), (byte) => byte.toString(16).padStart(2, "0")).join( + "", + ); +} + +/** A non-empty string, which is what every identity member here is. */ +function string(value: unknown): string | undefined { + return typeof value === "string" && value !== "" ? value : undefined; +} + +/** + * Any string, which is what an authored path and authored content are. + * + * A document may write ``, and the empty path is what the + * ordinary lexical refusal is about. Refusing to read it back here would leave a + * continuation unable to parse a record this host itself wrote. + */ +function anyString(value: unknown): string | undefined { + return typeof value === "string" ? value : undefined; +} diff --git a/packages/git/tests/module-partition.test.ts b/packages/git/tests/module-partition.test.ts index da0dd3189..02a014bea 100644 --- a/packages/git/tests/module-partition.test.ts +++ b/packages/git/tests/module-partition.test.ts @@ -91,6 +91,8 @@ const PROVIDER_NEUTRAL: readonly string[] = [ "src/issue/records.ts", "src/issue/tracker.ts", "src/plugin.ts", + "src/retained-repository/admission.ts", + "src/retained-repository/records.ts", ]; /** @@ -156,10 +158,15 @@ const DENO_ADAPTER: readonly string[] = [ "src/deno/composition/subprocess.ts", "src/deno/composition/switch.ts", "src/deno/composition/worktree.ts", + "src/deno/retained-repository/files.ts", + "src/deno/retained-repository/leases.ts", + "src/deno/retained-repository/local.ts", + "src/deno/retained-repository/provider.ts", "src/deno/run-composition/ambient.ts", "src/deno/run-composition/checkouts.ts", "src/deno/run-composition/errors.ts", "src/deno/run-composition/identity.ts", + "src/deno/run-composition/lazy.ts", "src/deno/run-composition/leases.ts", "src/deno/run-composition/metadata.ts", "src/deno/run-composition/operations.ts", diff --git a/packages/git/tests/public-entrypoint.test.ts b/packages/git/tests/public-entrypoint.test.ts index 5c8a0b527..472d6222b 100644 --- a/packages/git/tests/public-entrypoint.test.ts +++ b/packages/git/tests/public-entrypoint.test.ts @@ -53,11 +53,12 @@ const INTERNAL: readonly string[] = [ const ROOT_PUBLISHED: readonly string[] = ["gitPlugin", "gitPluginDeclaresFor"]; /** - * GitHub names this package publishes on purpose. + * The names this package publishes to a Deno host on purpose. * * Listed so that losing one is a failure rather than a silent narrowing: a - * host configuring a ceiling needs the recognizer, and a document naming a - * provider needs the discriminator. + * host configuring a ceiling needs the recognizer, a document naming a + * provider needs the discriminator, and the REPL host needs the one operation + * an executing entry initializes its recording with. */ const PUBLISHED: readonly string[] = [ "GITHUB", @@ -69,6 +70,7 @@ const PUBLISHED: readonly string[] = [ "recognizesGitHubUrl", "useGitHubIssues", "useGitHubPullRequests", + "useReplRepositoryRecording", ]; /** diff --git a/packages/git/tests/retained-repository-lifecycle.test.ts b/packages/git/tests/retained-repository-lifecycle.test.ts new file mode 100644 index 000000000..d7d94c084 --- /dev/null +++ b/packages/git/tests/retained-repository-lifecycle.test.ts @@ -0,0 +1,703 @@ +/** + * Who holds a checkout while an entry works in it, and when they let go. + * + * Two entries moving files in one repository is what the exclusion exists to + * prevent, and one mechanism answers it: the kernel lock on a sidecar beside + * the repository. `flock` and `LockFileEx` both exclude *independently opened* + * handles, including two opened by one process, so the native descriptor is the + * whole live resource and there is no table of holders to keep. + * + * The rows that matter most are the entry-level ones. A refused acquisition has + * to leave the history it found exactly as it was: records already written stay + * written, the mutation that was about to happen gets no result of any kind, + * nothing after it runs, and a later attempt works once the holder has gone. A + * lease that leaked past its entry, or one that answered from a table instead of + * the kernel, would pass a helper test and fail that rule. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { + type Operation, + type Result, + ensure, + resource, + scoped, + sleep, + spawn, + suspend, + until, + withResolvers, +} from "effection"; +import { exists, readTextFile, rm, writeTextFile } from "@effectionx/fs"; +import { mkdir, mkdtemp, realpath } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { API, hostFilesHandler } from "@executablemd/runtime"; +import type { FilePathInput, FilesHandler, FileWriteInput } from "@executablemd/runtime"; +import { InMemoryStream } from "@executablemd/durable-streams"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import { collect, Elicitation } from "@executablemd/core"; +import { Execution } from "@executablemd/core/api"; +import { executeInstalled } from "@executablemd/core/host"; +import { useAdvisoryLock } from "@executablemd/workflow/deno"; +import { denoRepositoryHost } from "../src/deno/composition/host.ts"; +import { gitSession } from "../src/deno/composition/git.ts"; +import { useReplRepositoryRecording } from "../src/deno/retained-repository/provider.ts"; +import { admitReplRepositoryHistory } from "../src/retained-repository/admission.ts"; +import { + CommonDirectoryLeaseError, + useCommonDirectoryLease, +} from "../src/deno/retained-repository/leases.ts"; +import { commonDirectoryLock } from "../src/deno/run-composition/placement.ts"; +import { useCompositionComponents } from "../src/composition/installation.ts"; +import { REPL_REPOSITORY_LOCAL } from "../src/retained-repository/records.ts"; + +/** Let every ready task run, so "reached its wait" is about turns. */ +function* settled(turns = 8): Operation { + for (let turn = 0; turn < turns; turn += 1) { + yield* sleep(0); + } +} + +interface Fixture { + readonly root: string; + readonly checkout: string; + readonly elsewhere: string; +} + +/** Two seeded checkouts, so "another directory" is a real other repository. */ +function useFixture(): Operation { + return resource(function* (provide) { + const root = yield* until(realpath(yield* until(mkdtemp(join(tmpdir(), "repl-lease-"))))); + yield* ensure(() => rm(root, { recursive: true, force: true })); + const git = gitSession(denoRepositoryHost(), root); + const made: string[] = []; + for (const name of ["checkout", "elsewhere"]) { + const directory = join(root, name); + yield* until(mkdir(directory, { recursive: true })); + yield* git.run(["init", "-b", "main", directory], root); + yield* writeTextFile(join(directory, "seed.md"), "seed\n"); + made.push(directory); + } + yield* provide({ root, checkout: made[0] ?? "", elsewhere: made[1] ?? "" }); + }); +} + +/** The canonical common Git directory of this checkout. */ +function common(checkout: string): Operation { + return until(realpath(join(checkout, ".git"))); +} + +/** What holding this directory answered: nothing, or the refusal it raised. */ +function* held(directory: string): Operation { + try { + yield* useCommonDirectoryLease(directory); + return undefined; + } catch (error) { + return error instanceof Error ? error : new Error(String(error)); + } +} + +interface Counters { + writes: string[]; + deletes: string[]; +} + +function counters(): Counters { + return { writes: [], deletes: [] }; +} + +function countingFiles(counted: Counters): FilesHandler { + const host = hostFilesHandler(); + return { + checkFilePath: (input) => host.checkFilePath(input), + readTextFile: (input) => host.readTextFile(input), + globFiles: (input) => host.globFiles(input), + temporaryDirectory: () => host.temporaryDirectory(), + *writeTextFile(input: FileWriteInput) { + counted.writes.push(input.path); + return yield* host.writeTextFile(input); + }, + *deleteFile(input: FilePathInput) { + counted.deletes.push(input.path); + return yield* host.deleteFile(input); + }, + ensureDirectory: (input) => host.ensureDirectory(input), + }; +} + +interface Run { + readonly refusal: string | undefined; + readonly events: readonly DurableEvent[]; +} + +/** + * One entry, run to completion in this checkout. + * + * The entry host is the production one, so a second concurrent entry is refused + * by whatever actually refuses it rather than by anything this suite installs. + */ +function entry( + checkout: string, + source: string, + stream: InMemoryStream, + counted: Counters, + /** + * Work to do while this entry is standing at a question it never answers. + * + * Where a row holding the checkout open has to wait, and it has to wait while + * the *document* is live: the lease belongs to the scope core initialization + * established it in, and an entry whose document has finished has already + * released it. A source ending in `` is how a held entry is held in + * production too — somebody left it at a question — so this is that, with the + * answer never arriving. + */ + holding?: () => Operation, +): Operation { + return scoped(function* (): Operation { + const path = join(checkout, "entry.md"); + yield* writeTextFile(path, source); + yield* API.Env.around( + { + // deno-lint-ignore require-yield + *cwd() { + return checkout; + }, + }, + { at: "min" }, + ); + const files = countingFiles(counted); + yield* API.Files.around( + { + checkFilePath: ([input]) => files.checkFilePath(input), + readTextFile: ([input]) => files.readTextFile(input), + globFiles: ([input]) => files.globFiles(input), + temporaryDirectory: () => files.temporaryDirectory(), + writeTextFile: ([input]) => files.writeTextFile(input), + deleteFile: ([input]) => files.deleteFile(input), + ensureDirectory: ([input]) => files.ensureDirectory(input), + }, + { at: "min" }, + ); + yield* useCompositionComponents(); + yield* Execution.around({ + *init(_args, next): Operation { + yield* useReplRepositoryRecording(); + yield* next(); + }, + }); + if (holding !== undefined) { + // The question this entry's source ends at. Whoever asked gets no answer: + // the callback runs instead, and the entry stays where it is — holding + // the lease its initialization took — until that callback returns. + yield* Elicitation.around({ + *elicit(): Operation { + yield* holding(); + return yield* suspend(); + }, + }); + } + let refusal: string | undefined; + try { + yield* collect( + yield* executeInstalled({ path, stream }, [ + { admissions: [(retained) => admitReplRepositoryHistory(retained)] }, + ]), + ); + } catch (error) { + refusal = error instanceof Error ? error.message : String(error); + } + return { refusal, events: yield* stream.readAll() }; + }); +} + +function localRecords(events: readonly DurableEvent[]): number { + return events.filter( + (event) => event.type === "yield" && event.description.type === REPL_REPOSITORY_LOCAL, + ).length; +} + +/** The question a held entry stands at, never answered. */ +const QUESTION = 'Holding'; + +describe("an entry's hold on the checkout it is working in", () => { + it("E8: refuses a second live entry, preserving what the first recorded", function* () { + const fixture = yield* useFixture(); + const holding = counters(); + const blocked = counters(); + const stream = new InMemoryStream(); + const first = new InMemoryStream(); + + yield* scoped(function* (): Operation { + // One entry holding the checkout, kept open by a write it has already + // performed and a suspension after it. + const acquired = withResolvers(); + const release = withResolvers(); + yield* spawn(function* (): Operation { + yield* entry( + fixture.checkout, + ['held', QUESTION].join("\n"), + first, + holding, + function* (): Operation { + acquired.resolve(); + yield* release.operation; + }, + ); + }); + yield* acquired.operation; + expect(holding.writes).toEqual(["held.md"]); + const recorded = localRecords(yield* first.readAll()); + expect(recorded).toBe(1); + + // A second live entry of the same process, in the same checkout. It is + // refused by the operating system: the first entry's descriptor is a + // different handle on the same file. + const second = yield* entry( + fixture.checkout, + ['blocked', 'sentinel'].join( + "\n", + ), + stream, + blocked, + ); + + // Refused before anything was read or changed, so: no mutation, no + // sentinel after it, and no result of any kind written down for the + // operation that was about to happen. + expect(second.refusal).toContain("Another XMD entry or process holds this checkout"); + expect(blocked.writes).toEqual([]); + expect(yield* exists(join(fixture.checkout, "blocked.md"))).toBe(false); + expect(yield* exists(join(fixture.checkout, "sentinel.md"))).toBe(false); + expect(localRecords(second.events)).toBe(0); + + // And the records the first entry wrote are untouched. + expect(localRecords(yield* first.readAll())).toBe(recorded); + expect(yield* readTextFile(join(fixture.checkout, "held.md"))).toBe("held"); + + release.resolve(); + }); + + // The holder is gone, so a later attempt works against the environment then + // available. + const later = counters(); + const after = yield* entry( + fixture.checkout, + 'later', + new InMemoryStream(), + later, + ); + expect(after.refusal).toBe(undefined); + expect(later.writes).toEqual(["later.md"]); + }); + + it("E8: lets an entry in another repository through", function* () { + const fixture = yield* useFixture(); + const holding = counters(); + const other = counters(); + + yield* scoped(function* (): Operation { + const acquired = withResolvers(); + const release = withResolvers(); + yield* spawn(function* (): Operation { + yield* entry( + fixture.checkout, + ['held', QUESTION].join("\n"), + new InMemoryStream(), + holding, + function* (): Operation { + acquired.resolve(); + yield* release.operation; + }, + ); + }); + yield* acquired.operation; + + // A different common directory is a different question. + const elsewhere = yield* entry( + fixture.elsewhere, + 'free', + new InMemoryStream(), + other, + ); + expect(elsewhere.refusal).toBe(undefined); + expect(other.writes).toEqual(["free.md"]); + release.resolve(); + }); + }); + + it("E8: takes one handle for an entry's many mutations", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + + // Four mutations in one entry. Each asks for the lease; the entry's lazy + // single-flight acquisition is what makes that one native handle rather + // than four — a second independent handle would refuse itself. + const performed = yield* entry( + fixture.checkout, + [ + '', + 'one', + 'two', + '', + "", + ].join("\n"), + new InMemoryStream(), + counted, + ); + + expect(performed.refusal).toBe(undefined); + expect(counted.writes).toEqual(["one.md", "two.md"]); + expect(counted.deletes).toEqual(["one.md"]); + expect(localRecords(performed.events)).toBe(4); + }); + + it("E8: releases the checkout when the entry is torn down, and not before", function* () { + const fixture = yield* useFixture(); + const directory = yield* common(fixture.checkout); + const acquired = withResolvers(); + const release = withResolvers(); + + yield* scoped(function* (): Operation { + const holder = yield* spawn(function* (): Operation { + yield* useCommonDirectoryLease(directory); + acquired.resolve(); + yield* release.operation; + }); + // Asked for after the holder has it, not beside it: a probe racing the + // acquisition would be testing which of the two went first. + yield* acquired.operation; + expect(yield* scoped(() => held(directory))).not.toBe(undefined); + release.resolve(); + // Joined, not merely signalled: what proves the lease is gone is the + // holder's own teardown having run. + yield* holder; + }); + + expect(yield* scoped(() => held(directory))).toBe(undefined); + }); + + it("E8: refuses a second independently opened handle in this process", function* () { + const fixture = yield* useFixture(); + const directory = yield* common(fixture.checkout); + + yield* scoped(function* (): Operation { + const acquired = withResolvers(); + yield* spawn(function* (): Operation { + yield* useCommonDirectoryLease(directory); + acquired.resolve(); + yield* suspend(); + }); + yield* acquired.operation; + // The kernel, not a table: this is a second `open` of the same path in one + // process, which `flock` and `LockFileEx` both refuse. + const refused = yield* scoped(() => held(directory)); + expect(refused instanceof CommonDirectoryLeaseError).toBe(true); + }); + }); + + it("E8: refuses while another process holds the sidecar, and not after", function* () { + const fixture = yield* useFixture(); + const directory = yield* common(fixture.checkout); + const lock = commonDirectoryLock(directory); + const child = yield* holdingChild(lock); + + expect(yield* child.locked()).toBe("locked"); + expect(yield* scoped(() => held(directory))).not.toBe(undefined); + + // The child lets go, and the kernel is what says so: this waits for the + // process to be gone rather than for the close to be accepted. + yield* child.finish(); + + expect(yield* scoped(() => held(directory))).toBe(undefined); + // Nothing unlinked the sidecar: unlinking a locked path lets the next + // caller lock a different file at the same name. + expect(yield* exists(lock)).toBe(true); + }); + + it("E8: joins the child it owns when finishing is cancelled", function* () { + const fixture = yield* useFixture(); + const lock = commonDirectoryLock(yield* common(fixture.elsewhere)); + let probe: HoldingChild | undefined; + + yield* scoped(function* (): Operation { + // A child that holds the sidecar and does not exit when its standard + // input closes, so `finish()` gets past the close and suspends on the + // status wait — which is where a cancellation lands. + const child = yield* holdingChild(lock, { ignoreStdin: true }); + probe = child; + expect(yield* child.locked()).toBe("locked"); + yield* spawn(() => child.finish()); + // Let the finishing task reach its wait before this scope unwinds. + yield* settled(); + }); + + // The owner's teardown has returned. A fixture that switched its cleanup + // off on the way *into* those waits would have left a live process here, + // and a live process still holds the sidecar — so the lock being free is + // the independent evidence that the child was killed and joined. + expect(probe?.alive()).toBe(false); + yield* scoped(function* (): Operation { + expect(yield* useAdvisoryLock(lock)).not.toBe(undefined); + }); + }); + + it("E8: releases the advisory lock when its holder is cancelled", function* () { + const fixture = yield* useFixture(); + const lock = commonDirectoryLock(yield* common(fixture.elsewhere)); + const acquired = withResolvers(); + + yield* scoped(function* (): Operation { + const task = yield* spawn(function* (): Operation { + const file = yield* useAdvisoryLock(lock); + // Acquired before the halt, so this cancels a holder rather than a + // caller that had not reached the descriptor yet. + expect(file).not.toBe(undefined); + acquired.resolve(); + yield* suspend(); + }); + yield* acquired.operation; + yield* task.halt(); + }); + + // Available immediately afterwards, in this same process, where a + // still-open descriptor would still hold it. + yield* scoped(function* (): Operation { + expect(yield* useAdvisoryLock(lock)).not.toBe(undefined); + }); + }); + + it("E8: opens, locks, releases and closes each descriptor exactly once", function* () { + const fixture = yield* useFixture(); + const lock = commonDirectoryLock(yield* common(fixture.checkout)); + + yield* scoped(function* (): Operation { + const observed = yield* useObservedDescriptors(); + + // A held descriptor: nothing is released while the scope that took it is + // still running, and leaving it unlocks and closes once each. + yield* scoped(function* (): Operation { + expect(yield* useAdvisoryLock(lock)).not.toBe(undefined); + expect(observed.events()).toEqual(["open", "lock"]); + }); + expect(observed.events()).toEqual(["open", "lock", "unlock", "close"]); + + // A refused descriptor, against a real holder. + const acquired = withResolvers(); + yield* spawn(function* (): Operation { + expect(yield* useAdvisoryLock(lock)).not.toBe(undefined); + acquired.resolve(); + yield* suspend(); + }); + yield* acquired.operation; + observed.forget(); + + yield* scoped(function* (): Operation { + expect(yield* useAdvisoryLock(lock)).toBe(undefined); + // Closed already, inside the scope that asked: a descriptor holding + // nothing has nothing to release, and keeping it open until teardown + // would cost a refusal a handle it never acquired. Observing the close + // is the point — a refused descriptor left open holds no lock, so + // reacquiring later cannot reveal it. + expect(observed.events()).toEqual(["open", "refused", "close"]); + }); + // And not closed again on the way out. A second close throws, so the + // list staying put is what says cleanup handled the partial acquisition. + expect(observed.events()).toEqual(["open", "refused", "close"]); + + // A lock attempt that throws: the descriptor is closed once, and never + // unlocked — it was never held. + observed.forget(); + observed.failLock(); + yield* scoped(function* (): Operation { + let raised: Error | undefined; + try { + yield* useAdvisoryLock(lock); + } catch (error) { + raised = error instanceof Error ? error : new Error(String(error)); + } + expect(raised?.message).toBe("this host refused to lock"); + // One close and no unlock: the descriptor was never held, so there was + // nothing to release. Which of the two paths closed it — the catch, or + // the cleanup unwinding as the error propagates out of the resource — + // is deliberately not claimed here, because the two are simultaneous + // from outside. What is claimed is that exactly one of them did it. + expect(observed.events()).toEqual(["open", "lock-threw", "close"]); + }); + // And not closed a second time on the way out. + expect(observed.events()).toEqual(["open", "lock-threw", "close"]); + }); + }); +}); + +/** What this host was asked to do with each lock descriptor, in order. */ +interface ObservedDescriptors { + events(): readonly string[]; + /** Start a fresh list, so one row is not read against another's calls. */ + forget(): void; + /** Make the next lock attempt throw, which is a partial acquisition. */ + failLock(): void; +} + +/** + * Watch the native boundary the advisory helper reaches through. + * + * `advisory-lock.ts` reads its runtime off the global and depends on three + * methods of what `openSync` hands back, so wrapping that one function is + * enough to see every open, lock attempt, unlock and close without the + * production module gaining a seam of its own. The real descriptor still does + * the work; this only records what was asked of it. + * + * The restore is registered before the substitution, so a cancelled or failed + * row puts the host's own function back. + */ +function useObservedDescriptors(): Operation { + return resource(function* (provide) { + const runtime: unknown = Reflect.get(globalThis, "Deno"); + const original: unknown = Reflect.get(Object(runtime), "openSync"); + if (typeof original !== "function") { + throw new Error("this host opens no files, so there is no descriptor to observe"); + } + let events: string[] = []; + let throwing = false; + + yield* ensure(() => { + Reflect.set(Object(runtime), "openSync", original); + }); + + Reflect.set(Object(runtime), "openSync", (path: string, options: unknown) => { + const file = Reflect.apply(original, runtime, [path, options]); + events.push("open"); + return { + tryLockSync(exclusive: boolean): boolean { + if (throwing) { + events.push("lock-threw"); + throw new Error("this host refused to lock"); + } + const locked = Reflect.apply(Reflect.get(Object(file), "tryLockSync"), file, [exclusive]); + events.push(locked === true ? "lock" : "refused"); + return locked === true; + }, + unlockSync(): void { + events.push("unlock"); + Reflect.apply(Reflect.get(Object(file), "unlockSync"), file, []); + }, + close(): void { + events.push("close"); + Reflect.apply(Reflect.get(Object(file), "close"), file, []); + }, + }; + }); + + yield* provide({ + events: () => [...events], + forget(): void { + events = []; + }, + failLock(): void { + throwing = true; + }, + }); + }); +} + +/** A child process holding one sidecar, owned by the scope that started it. */ +interface HoldingChild { + /** What the child said once it had the lock. */ + locked(): Operation; + /** Let it go, and wait for the process to be gone. */ + finish(): Operation; + /** Whether this fixture still owns a process it has not joined. */ + alive(): boolean; +} + +/** + * Start a process that locks this path and waits. + * + * The reaping cleanup is registered *before* the child exists, which is rule + * 16's ownership order: a scope halted between the spawn and the registration + * would otherwise leave a process nothing owns. It waits for the child's own + * status rather than for the signal to be accepted, because a kill that was + * sent is not a process that has gone. + */ +function holdingChild( + lock: string, + options: { readonly ignoreStdin?: boolean } = {}, +): Operation { + return resource(function* (provide) { + let child: Deno.ChildProcess | undefined; + /** + * Set once the child's own status has resolved, and never before. + * + * `finish()` suspends twice — closing stdin, then waiting for the status — + * and a cancellation landing on either of those is exactly when this + * cleanup is needed. Declaring completion on the way *into* those waits + * would switch the cleanup off while the process was still running, which + * is registering cleanup before the spawn and then disowning the result. + */ + let gone = false; + yield* ensure(function* (): Operation { + if (child === undefined || gone) { + return; + } + try { + child.kill(); + } catch { + // Already gone, which is the ordinary case once stdin closed. + } + yield* until(child.status); + gone = true; + }); + + child = new Deno.Command(Deno.execPath(), { + args: [ + "eval", + [ + "const path = Deno.args[0];", + "await Deno.mkdir(path.slice(0, path.lastIndexOf('/')), { recursive: true });", + "const file = Deno.openSync(path, { read: true, write: true, create: true });", + "if (!file.tryLockSync(true)) { console.log('refused'); Deno.exit(1); }", + "console.log('locked');", + options.ignoreStdin === true + ? // Holds on past its standard input closing, so a `finish()` that + // has already closed stdin suspends on the status wait rather than + // completing. That is the cancellation boundary this fixture's own + // ownership has to survive. + // + // A pending timer rather than a bare never-resolving promise: with + // nothing on the event loop Deno reports an unresolvable top-level + // await and *exits*, which would release the lock on its own and + // make this fixture prove nothing. + "await new Promise((resolve) => setTimeout(resolve, 600000));" + : "await new Response(Deno.stdin.readable).text();", + ].join("\n"), + lock, + ], + stdin: "piped", + stdout: "piped", + // Piped so a killed child's own complaint stays out of the report. + stderr: "piped", + }).spawn(); + const started = child; + + const reader = started.stdout.getReader(); + yield* provide({ + *locked(): Operation { + const first = yield* until(reader.read()); + return new TextDecoder().decode(first.value).trim(); + }, + *finish(): Operation { + reader.releaseLock(); + yield* until(started.stdin.close()); + yield* until(started.status); + // Only now: the process has reported its own termination, so there is + // nothing left for the cleanup above to own. + gone = true; + }, + alive(): boolean { + return !gone; + }, + }); + }); +} diff --git a/packages/git/tests/retained-repository-local.test.ts b/packages/git/tests/retained-repository-local.test.ts new file mode 100644 index 000000000..198dd0424 --- /dev/null +++ b/packages/git/tests/retained-repository-local.test.ts @@ -0,0 +1,1027 @@ +/** + * What one REPL entry's local file work retains, replays and refuses to guess. + * + * Everything here runs a document through canonical execution with the entry + * host installed, so what is proven is the whole path: the authored component + * calls the document filesystem, the composition beneath answers, and the + * result is recorded once. A fixture that asserted on the record without the + * composition would prove the bookkeeping and nothing about the answer. + * + * The contract these rows hold the host to is the REPL's execution model. + * Replay restores recorded results; it does not restore the environment or + * require it to match. So the rows that matter most are the ones where the two + * disagree: a file changed behind a completed write, a checkout that is no + * longer there, and a write whose result was never acknowledged. + * + * Each scenario gets its own checkout, HOME and journal, because what most of + * them assert is what a *second* open of one history does. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { + Err, + Ok, + ensure, + race, + resource, + scoped, + sleep, + spawn, + suspend, + until, + withResolvers, +} from "effection"; +import type { Operation, Result } from "effection"; +import { exists, readTextFile, rm, writeTextFile } from "@effectionx/fs"; +import { mkdtemp, realpath, symlink, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + API, + cwd, + FILES_ERROR, + fileWriteSuccess, + filesFailure, + hostFilesHandler, +} from "@executablemd/runtime"; +import type { FilePathInput, FilesHandler, FileWriteInput } from "@executablemd/runtime"; +import { InMemoryStream } from "@executablemd/durable-streams"; +import type { DurableEvent, DurableStream, Json } from "@executablemd/durable-streams"; +import { Execution } from "@executablemd/core/api"; +import { executeInstalled } from "@executablemd/core/host"; +import { collect, Elicitation } from "@executablemd/core"; +import { denoRepositoryHost } from "../src/deno/composition/host.ts"; +import { gitSession } from "../src/deno/composition/git.ts"; +import { useReplRepositoryRecording } from "../src/deno/retained-repository/provider.ts"; +import { admitReplRepositoryHistory } from "../src/retained-repository/admission.ts"; +import { useCompositionComponents } from "../src/composition/installation.ts"; +import { + REPL_REPOSITORY_LOCAL, + REPL_REPOSITORY_SELECTION, + parseReplLocalRecord, +} from "../src/retained-repository/records.ts"; +import type { ReplLocalValue } from "../src/retained-repository/records.ts"; + +interface Fixture { + readonly root: string; + readonly checkout: string; +} + +/** One seeded checkout, with its own HOME, outside every other scenario's. */ +function useFixture(): Operation { + return resource(function* (provide) { + const root = yield* until(realpath(yield* until(mkdtemp(join(tmpdir(), "repl-local-"))))); + yield* ensure(() => rm(root, { recursive: true, force: true })); + const checkout = join(root, "checkout"); + const git = gitSession(denoRepositoryHost(), root); + yield* git.run(["init", "-b", "main", checkout], root); + yield* until(writeFile(join(checkout, "seed.md"), "seed\n")); + yield* provide({ root, checkout }); + }); +} + +/** What the composition beneath the recorder was actually asked to do. */ +interface Counters { + reads: string[]; + writes: string[]; + deletes: string[]; + ensures: string[]; + /** Every `git` subcommand this entry's session ran. */ + commands: string[]; + /** Every working directory the recorder resolved through the filesystem. */ + resolved: number; +} + +function counters(): Counters { + return { reads: [], writes: [], deletes: [], ensures: [], commands: [], resolved: 0 }; +} + +/** + * The host filesystem, with the three mutations counted. + * + * Installed at `min`, the implementation slot, so the recorder's own middleware + * and anything a scenario puts between them compose above it exactly as they do + * in production. + */ +function countingFiles(counted: Counters): FilesHandler { + const host = hostFilesHandler(); + return { + checkFilePath: (input) => host.checkFilePath(input), + *readTextFile(input: FilePathInput) { + counted.reads.push(input.path); + return yield* host.readTextFile(input); + }, + globFiles: (input) => host.globFiles(input), + temporaryDirectory: () => host.temporaryDirectory(), + *writeTextFile(input: FileWriteInput) { + counted.writes.push(input.path); + return yield* host.writeTextFile(input); + }, + *deleteFile(input: FilePathInput) { + counted.deletes.push(input.path); + return yield* host.deleteFile(input); + }, + *ensureDirectory(input: FilePathInput) { + counted.ensures.push(input.path); + return yield* host.ensureDirectory(input); + }, + }; +} + +/** A repository host that counts the `git` this entry ran. */ +function countingHost(counted: Counters) { + const host = denoRepositoryHost(); + return { + git(invocation: Parameters[0]) { + counted.commands.push(invocation.args.join(" ")); + return host.git(invocation); + }, + useDirectory: () => host.useDirectory(), + }; +} + +interface Run { + readonly output: string; + readonly events: readonly DurableEvent[]; + /** What ended the entry, when something did. */ + readonly refusal: string | undefined; +} + +/** One entry, over one journal, in one checkout. */ +function run( + fixture: Fixture, + source: string, + stream: DurableStream, + counted: Counters, + options: { + readonly cwd?: string; + readonly between?: () => Operation; + } = {}, +): Operation { + return scoped(function* () { + const path = join(fixture.root, "entry.md"); + yield* writeTextFile(path, source); + const working = options.cwd ?? fixture.checkout; + yield* API.Env.around( + { + // deno-lint-ignore require-yield + *cwd() { + return working; + }, + }, + { at: "min" }, + ); + const files = countingFiles(counted); + yield* API.Files.around( + { + checkFilePath: ([input]) => files.checkFilePath(input), + readTextFile: ([input]) => files.readTextFile(input), + globFiles: ([input]) => files.globFiles(input), + temporaryDirectory: () => files.temporaryDirectory(), + writeTextFile: ([input]) => files.writeTextFile(input), + deleteFile: ([input]) => files.deleteFile(input), + ensureDirectory: ([input]) => files.ensureDirectory(input), + }, + { at: "min" }, + ); + // `` is this package's own vocabulary, and an entry that writes one + // has to be able to resolve the name. + yield* useCompositionComponents(); + if (options.between !== undefined) { + yield* options.between(); + } + // Registered the way the REPL host registers it: core initialization is + // what establishes the recording, inside the scope the invocation owns. + yield* Execution.around({ + *init(_args, next): Operation { + yield* useReplRepositoryRecording({ host: countingHost(counted) }); + yield* next(); + }, + }); + let refusal: string | undefined; + let output = ""; + try { + output = String( + yield* collect( + yield* executeInstalled({ path, stream, includes: [fixture.root] }, [ + // What the Git Plugin contributes for `repl`, attached directly: + // this suite assembles no command Plugins. + { admissions: [(retained) => admitReplRepositoryHistory(retained)] }, + ]), + ), + ); + } catch (error) { + refusal = error instanceof Error ? error.message : String(error); + } + return { output, events: yield* stream.readAll(), refusal }; + }); +} + +/** Every payload of one record type, in journal order. */ +function recordsOf(events: readonly DurableEvent[], type: string): readonly Json[] { + const held: Json[] = []; + for (const event of events) { + if (event.type === "yield" && event.description.type === type && event.result.status === "ok") { + held.push(event.result.value ?? null); + } + } + return held; +} + +/** The names, in order, this journal recorded local operations under. */ +function recordNames(events: readonly DurableEvent[]): readonly string[] { + return events + .filter((event) => event.type === "yield" && event.description.type === REPL_REPOSITORY_LOCAL) + .map((event) => (event.type === "yield" ? event.description.name : "")); +} + +/** + * A journal with its last recorded result removed. + * + * What an interruption leaves: work that happened and whose outcome was never + * acknowledged. There is nothing to replay for it, so a continuation takes the + * ordinary live path and may do it again. + */ +function interrupted(events: readonly DurableEvent[]): InMemoryStream { + for (let index = events.length - 1; index >= 0; index -= 1) { + const event = events[index]; + if ( + event !== undefined && + event.type === "yield" && + event.description.type === REPL_REPOSITORY_LOCAL + ) { + return new InMemoryStream(events.slice(0, index)); + } + } + throw new Error("the interruption fixture has no recorded result to remove"); +} + +/** Everything before the root Close, which is what a continuation reopens. */ +function unfinished(events: readonly DurableEvent[]): InMemoryStream { + return new InMemoryStream(events.filter((event) => event.type !== "close")); +} + +/** + * Data as it arrives from outside the type system. + * + * A provider is a contextual handler: at run time it can return whatever it + * likes, and the contract these rows exercise is precisely the one the types + * cannot enforce. Round-tripping through JSON is how a malformed value is built + * here without asserting a type it does not have — the same device + * `packages/core/tests/files-fatal.test.ts` uses for the outer validator. + */ +function fromOutside(value: unknown): T { + return JSON.parse(JSON.stringify(value)); +} + +/** + * The publication a recorded write's success names. + * + * A record carries a write receipt, a read's text or nothing at all, so the + * payload is narrowed here rather than at each assertion: a row about a write + * that found a string would be reading a record of something else. + */ +function written(outcome: Result | undefined): string | undefined { + if (outcome === undefined || !outcome.ok) { + return undefined; + } + const value = outcome.value; + return value === undefined || typeof value === "string" ? undefined : value.publication; +} + +/** What one record says about the operation it describes. */ +function localRecords(events: readonly DurableEvent[]) { + return recordsOf(events, REPL_REPOSITORY_LOCAL).map((payload) => { + const parsed = parseReplLocalRecord(payload); + if (parsed === undefined) { + throw new Error("a recorded local operation is not readable"); + } + return parsed; + }); +} + +describe("one entry's local file work", () => { + it("records an authored write, a delete and a Dir ensure once", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + + const first = yield* run( + fixture, + [ + "# entry", + "", + '', + 'first', + 'remove me', + '', + "", + ].join("\n"), + stream, + counted, + ); + + expect(first.refusal).toBe(undefined); + expect(yield* readTextFile(join(fixture.checkout, "drafts", "note.md"))).toBe("first"); + expect(yield* exists(join(fixture.checkout, "drafts", "obsolete.md"))).toBe(false); + // The ensure comes before the children, and the children are this Dir's. + expect(counted.ensures).toEqual(["drafts"]); + expect(counted.writes).toEqual(["note.md", "obsolete.md"]); + expect(counted.deletes).toEqual(["obsolete.md"]); + // One record per operation, and one routing record for the entry. + expect(recordsOf(first.events, REPL_REPOSITORY_LOCAL).length).toBe(4); + expect(recordsOf(first.events, REPL_REPOSITORY_SELECTION).length).toBe(1); + + // And a continuation performs none of the four again, Dir ensure included. + const replayed = counters(); + const again = yield* run( + fixture, + [ + "# entry", + "", + '', + 'first', + 'remove me', + '', + "", + ].join("\n"), + unfinished(first.events), + replayed, + ); + expect(again.refusal).toBe(undefined); + expect(replayed.ensures).toEqual([]); + expect(replayed.writes).toEqual([]); + expect(replayed.deletes).toEqual([]); + expect(replayed.commands).toEqual([]); + }); + + it("answers a completed operation from its record, against a changed file", function* () { + const fixture = yield* useFixture(); + const live = counters(); + const stream = new InMemoryStream(); + const source = 'first'; + + const first = yield* run(fixture, source, stream, live); + expect(first.refusal).toBe(undefined); + expect(live.writes).toEqual(["note.md"]); + + // The environment and the record now disagree, which the REPL's execution + // model allows: the person reading the screen is what resolves it. + yield* writeTextFile(join(fixture.checkout, "note.md"), "changed by somebody else"); + + const replayed = counters(); + const again = yield* run(fixture, source, unfinished(first.events), replayed); + + expect(again.refusal).toBe(undefined); + // Nothing was performed and nothing was resolved: no write, and no `git`. + expect(replayed.writes).toEqual([]); + expect(replayed.commands).toEqual([]); + // And the file the person changed is still theirs. + expect(yield* readTextFile(join(fixture.checkout, "note.md"))).toBe("changed by somebody else"); + }); + + it("performs an unacknowledged operation again", function* () { + const fixture = yield* useFixture(); + const live = counters(); + const stream = new InMemoryStream(); + const source = ['kept', 'lost'].join( + "\n", + ); + + const first = yield* run(fixture, source, stream, live); + expect(live.writes).toEqual(["kept.md", "lost.md"]); + + const again = counters(); + const continued = yield* run(fixture, source, interrupted(first.events), again); + + expect(continued.refusal).toBe(undefined); + // The acknowledged one replayed; the one whose result never landed ran. + expect(again.writes).toEqual(["lost.md"]); + }); + + it("leaves work outside every checkout ordinary, unrecorded and repeatable", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + const source = 'beside'; + + // Started where there is no checkout at all, which is the reachable + // outside case: an authored path may not escape the directory the run is + // working in, so a `../` target is refused lexically long before routing. + const performed = yield* run(fixture, source, stream, counted, { cwd: fixture.root }); + + expect(performed.refusal).toBe(undefined); + expect(yield* readTextFile(join(fixture.root, "beside.md"))).toBe("beside"); + expect(counted.writes).toEqual(["beside.md"]); + // The entry looked once, remembered that there is none, and recorded no + // operation — a record would claim a checkout this write is not in. + expect(recordsOf(performed.events, REPL_REPOSITORY_LOCAL)).toEqual([]); + expect(recordsOf(performed.events, REPL_REPOSITORY_SELECTION).length).toBe(1); + + // And because nothing was recorded, a continuation performs it again. + const again = counters(); + const continued = yield* run(fixture, source, unfinished(performed.events), again, { + cwd: fixture.root, + }); + expect(continued.refusal).toBe(undefined); + expect(again.writes).toEqual(["beside.md"]); + }); + + it("recognizes a checkout reached through a symlinked working directory", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + // A link to the checkout, which is how a host's temporary root is commonly + // reached — `/var` for `/private/var` on macOS. A checkout discovered by + // its real path does not appear to contain a file addressed through the + // link unless the working directory is resolved first. + const linked = join(fixture.root, "through-a-link"); + yield* until(symlink(fixture.checkout, linked)); + + const performed = yield* run(fixture, 'first', stream, counted, { + cwd: linked, + }); + + expect(performed.refusal).toBe(undefined); + expect(yield* readTextFile(join(fixture.checkout, "linked.md"))).toBe("first"); + // Routed as inside the checkout, which is what the record says. + expect(recordsOf(performed.events, REPL_REPOSITORY_LOCAL).length).toBe(1); + }); + + it("replays two equal writes in the order they were answered", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + const source = ['same', 'same'].join( + "\n", + ); + + const first = yield* run(fixture, source, stream, counted); + expect(counted.writes).toEqual(["twice.md", "twice.md"]); + + // One name at two ordered positions, which is valid: a reader that + // deduplicated by name would refuse a document that is allowed. + const names = recordNames(first.events); + expect(names.length).toBe(2); + expect(names[0]).toBe(names[1]); + + const replayed = counters(); + const again = yield* run(fixture, source, unfinished(first.events), replayed); + expect(again.refusal).toBe(undefined); + expect(replayed.writes).toEqual([]); + }); + + it("offers nested work before the mutation that encloses it", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + const source = 'inner'; + + const first = yield* run(fixture, source, stream, counted); + expect(first.refusal).toBe(undefined); + // A write's content expands before the write, so the nested one happened + // first — and the journal holds them in that order. + expect(counted.writes).toEqual(["inner.md", "outer.md"]); + const records = localRecords(first.events); + expect(records.map((record) => record.request.path)).toEqual(["inner.md", "outer.md"]); + + const replayed = counters(); + const again = yield* run(fixture, source, unfinished(first.events), replayed); + expect(again.refusal).toBe(undefined); + expect(replayed.writes).toEqual([]); + }); + + it("keeps two concurrent branches' equal requests on their own coroutines", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + // Two `` children of one `` are concurrent durable coroutines of + // the one that reached the ``, each with its own journal position. They + // write the same path with the same content, so the two records carry one + // name — and what tells them apart is which coroutine they belong to rather + // than anything this feature adds. + const source = [ + "", + 'same', + 'same', + "", + ].join("\n"); + + const first = yield* run(fixture, source, stream, counted); + expect(first.refusal).toBe(undefined); + expect(yield* readTextFile(join(fixture.checkout, "same.md"))).toBe("same"); + + // Two writes, one name, two coroutines: the name is what the request asks + // and the coroutine is where it was asked. + const writes = first.events.filter( + (event) => + event.type === "yield" && + event.description.type === REPL_REPOSITORY_LOCAL && + event.description.name.includes("file-write"), + ); + expect(writes.length).toBe(2); + const named = new Set( + writes.map((event) => (event.type === "yield" ? event.description.name : "")), + ); + expect(named.size).toBe(1); + const coroutines = new Set( + writes.map((event) => (event.type === "yield" ? event.coroutineId : "")), + ); + expect(coroutines.size).toBe(2); + expect(counted.writes).toEqual(["same.md", "same.md"]); + + // Repeated once: every operation replays on its own coroutine, and no + // native work happens again. + const replayed = counters(); + const again = yield* run(fixture, source, unfinished(first.events), replayed); + expect(again.refusal).toBe(undefined); + expect(replayed.writes).toEqual([]); + expect(replayed.ensures).toEqual([]); + expect(replayed.commands).toEqual([]); + }); + + it("continues a prefix each concurrent branch saved its own routing in", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + const question = 'Continue?'; + // Both branches mutate, so both save where they are working — each at its + // own position in its own coroutine. One of them reads a file first, so the + // two do not reach their first write in lockstep. Then both stop at a + // question, which is what leaves a prefix with no close in it. + const source = [ + "", + `A${question}`, + `B${question}`, + "", + 'new suffix', + ].join("\n"); + + const waiting = withResolvers(); + let asked = 0; + const live = yield* spawn(() => + run(fixture, source, stream, counted, { + *between(): Operation { + yield* Elicitation.around({ + *elicit(): Operation { + asked += 1; + if (asked === 2) { + waiting.resolve(); + } + // Nobody answers, so this is where the entry is when it is left. + return yield* suspend(); + }, + }); + }, + }), + ); + + yield* race([ + waiting.operation, + (function* (): Operation { + yield* sleep(10_000); + throw new Error("both branches never reached their questions"); + })(), + ]); + + // The prefix exactly as the interrupted run left it: nothing removed and + // nothing reordered. + const prefix = [...(yield* stream.readAll())]; + yield* live.halt(); + + expect(prefix.some((event) => event.type === "close")).toBe(false); + // Both branches wrote, and nothing else did. Sorted, because which of two + // concurrent branches reaches its write first is the scheduler's business + // and asserting one order here would make this row fail for a reason it is + // not about. + expect(counted.writes.toSorted()).toEqual(["a.md", "b.md"]); + // The condition that makes this row discriminating: two saved routings, on + // two different coroutines. A host that remembered routing per *entry* + // would skip one of them and the continuation below would be refused. + const selections = prefix.filter( + (event) => event.type === "yield" && event.description.type === REPL_REPOSITORY_SELECTION, + ); + expect(selections.length).toBe(2); + expect( + new Set(selections.map((event) => (event.type === "yield" ? event.coroutineId : ""))).size, + ).toBe(2); + + // Continued unchanged, with answers available this time. + const replayed = counters(); + let answered = 0; + const again = yield* run(fixture, source, new InMemoryStream(prefix), replayed, { + *between(): Operation { + yield* Elicitation.around({ + // deno-lint-ignore require-yield + *elicit(): Operation { + answered += 1; + return {}; + }, + }); + }, + }); + + expect(again.refusal).toBe(undefined); + // Both completed writes came from history, both questions were answerable, + // and the work after them ran once. + expect(replayed.writes).toEqual(["sentinel.md"]); + expect(answered).toBe(2); + expect(yield* readTextFile(join(fixture.checkout, "sentinel.md"))).toBe("new suffix"); + // Discovery stays shared. The suffix is new work in a coroutine that saved + // no routing, so it does look — once, for the whole continuation, rather + // than once per coroutine that asks. + expect(replayed.commands).toEqual(["rev-parse --show-toplevel", "rev-parse --git-common-dir"]); + }); + + it("records what the composition answered, without performing it itself", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + let answered = 0; + const source = 'staged'; + + // A provider that answers the write itself, exactly as a transactional one + // does. Installed beneath the recorder and above the host, so what the + // recorder keeps is the answer the whole composition gave. + const answering = function* (): Operation { + yield* API.Files.around( + { + // deno-lint-ignore require-yield + *writeTextFile(): Operation>> { + answered += 1; + return Ok(fileWriteSuccess("transaction-staged")); + }, + }, + { at: "min" }, + ); + }; + + const first = yield* run(fixture, source, stream, counted, { between: answering }); + expect(first.refusal).toBe(undefined); + expect(answered).toBe(1); + // Nothing reached the host, and the publication the answer named is kept. + expect(counted.writes).toEqual([]); + const records = localRecords(first.events); + expect(records.length).toBe(1); + expect(records[0]?.outcome.ok).toBe(true); + expect(written(records[0]?.outcome)).toBe("transaction-staged"); + + answered = 0; + const replayed = counters(); + const again = yield* run(fixture, source, unfinished(first.events), replayed, { + between: answering, + }); + expect(again.refusal).toBe(undefined); + // The same semantics, and the answering provider was never asked. + expect(answered).toBe(0); + expect(replayed.writes).toEqual([]); + }); + + it("keeps the received request as the identity when a provider transforms it", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + let delegated = 0; + const source = 'first'; + + const transforming = function* (): Operation { + yield* API.Files.around( + { + *writeTextFile([input], next) { + delegated += 1; + return yield* next({ ...input, path: "elsewhere.md", content: "TRANSFORMED" }); + }, + }, + { at: "min" }, + ); + }; + + const first = yield* run(fixture, source, stream, counted, { between: transforming }); + expect(first.refusal).toBe(undefined); + // Delegated once, and the host performed the provider's version of it. + expect(delegated).toBe(1); + expect(counted.writes).toEqual(["elsewhere.md"]); + expect(yield* readTextFile(join(fixture.checkout, "elsewhere.md"))).toBe("TRANSFORMED"); + // The record names what the document asked for. A downstream path is + // provider behavior, not a second authored identity. + const records = localRecords(first.events); + expect(records[0]?.request).toEqual({ + cwd: fixture.checkout, + path: "note.md", + content: "first", + }); + + delegated = 0; + const replayed = counters(); + yield* run(fixture, source, unfinished(first.events), replayed, { between: transforming }); + expect(delegated).toBe(0); + expect(replayed.writes).toEqual([]); + }); + + it("replays an ordinary refusal as the refusal it was", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + let refused = 0; + const source = ''; + + const refusing = function* (): Operation { + yield* API.Files.around( + { + // deno-lint-ignore require-yield + *deleteFile(): Operation> { + refused += 1; + return Err( + filesFailure({ operation: "delete", phase: "access", reason: "permission-denied" }), + ); + }, + }, + { at: "min" }, + ); + }; + + const first = yield* run(fixture, source, stream, counted, { between: refusing }); + expect(refused).toBe(1); + // Refused without delegating, and the file is still there. + expect(counted.deletes).toEqual([]); + expect(yield* exists(join(fixture.checkout, "seed.md"))).toBe(true); + // A refusal the document can read, and it is in the document's output. + expect(first.output).toContain("seed.md"); + const records = localRecords(first.events); + expect(records[0]?.outcome.ok).toBe(false); + + refused = 0; + const replayed = counters(); + const again = yield* run(fixture, source, unfinished(first.events), replayed, { + between: refusing, + }); + expect(again.output).toContain("seed.md"); + expect(refused).toBe(0); + expect(replayed.deletes).toEqual([]); + }); + + it("answers completed results with the checkout gone", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + const source = [ + '', + 'first', + '', + '', + "", + "{held}", + ].join("\n"); + + const first = yield* run(fixture, source, stream, counted); + expect(first.refusal).toBe(undefined); + // One ensure, one write, one read and one delete. + expect(localRecords(first.events).length).toBe(4); + expect(first.output).toContain("first"); + + // The checkout the records were made in is no longer there. A completed + // operation is still answered, because answering it reads nothing. + yield* rm(fixture.checkout, { recursive: true, force: true }); + + const replayed = counters(); + const again = yield* run(fixture, source, unfinished(first.events), replayed); + expect(again.refusal).toBe(undefined); + expect(replayed.reads).toEqual([]); + expect(replayed.writes).toEqual([]); + expect(replayed.deletes).toEqual([]); + expect(replayed.ensures).toEqual([]); + // No discovery, and no canonicalization of a directory that is gone. + expect(replayed.commands).toEqual([]); + // And the binding a continuation offers is the text the read answered with + // the first time, from a file that no longer exists. + expect(again.output).toContain("first"); + }); + it("leaves no completed result when the acknowledgement is cancelled", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const events: DurableEvent[] = []; + const reached = withResolvers(); + const held = withResolvers(); + const source = 'first'; + + // A barrier between the native work and its acknowledgement: the write has + // happened, and the append has not. + const barrier: DurableStream = { + *readAll(): Operation { + return [...events]; + }, + *append(event: DurableEvent): Operation { + if (event.type === "yield" && event.description.type === REPL_REPOSITORY_LOCAL) { + reached.resolve(); + yield* held.operation; + } + events.push(event); + }, + }; + + yield* scoped(function* () { + const task = yield* spawn(() => run(fixture, source, barrier, counted)); + yield* reached.operation; + yield* task.halt(); + }); + + // The work happened, and nothing says it did. + expect(counted.writes).toEqual(["note.md"]); + expect(recordsOf(events, REPL_REPOSITORY_LOCAL)).toEqual([]); + expect(yield* readTextFile(join(fixture.checkout, "note.md"))).toBe("first"); + + // So a continuation takes the live path and may do it again, which is the + // window this feature trades for not snapshotting anybody's working tree. + const again = counters(); + const continued = yield* run(fixture, source, new InMemoryStream([...events]), again); + expect(continued.refusal).toBe(undefined); + expect(again.writes).toEqual(["note.md"]); + }); + + it("keeps a failed acknowledgement fatal, and runs nothing after it", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const events: DurableEvent[] = []; + const rejecting: DurableStream = { + *readAll(): Operation { + return [...events]; + }, + *append(event: DurableEvent): Operation { + if (event.type === "yield" && event.description.type === REPL_REPOSITORY_LOCAL) { + throw new Error("the journal refused this event"); + } + events.push(event); + }, + }; + + const run1 = yield* run( + fixture, + [ + "", + 'first', + "", + 'sentinel', + ].join("\n"), + rejecting, + counted, + ); + + // The write happened; the durability failure is fatal; and the sentinel + // after it never ran, printing region or not. + expect(counted.writes).toEqual(["note.md"]); + expect(run1.refusal === undefined ? run1.output : run1.refusal).not.toContain("sentinel"); + expect(recordsOf(events, REPL_REPOSITORY_LOCAL)).toEqual([]); + expect(yield* exists(join(fixture.checkout, "sentinel.md"))).toBe(false); + }); + + it("keeps a malformed provider answer fatal, with no completion and no sentinel", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + + const malformed = function* (): Operation { + yield* API.Files.around( + { + // deno-lint-ignore require-yield + *writeTextFile(): Operation> { + // A refusal in no vocabulary this boundary can read. A provider + // that cannot describe what it did may not have done it, so this + // is a broken contract rather than an outcome to write down. + return Err(new Error("something went wrong")); + }, + }, + { at: "min" }, + ); + }; + + const performed = yield* run( + fixture, + [ + "", + 'first', + "", + 'sentinel', + ].join("\n"), + stream, + counted, + { between: malformed }, + ); + + // No ordinary completion was written for it, and nothing ran after it. + expect(recordsOf(performed.events, REPL_REPOSITORY_LOCAL)).toEqual([]); + expect(counted.writes).toEqual([]); + expect(yield* exists(join(fixture.checkout, "sentinel.md"))).toBe(false); + }); + + it("keeps a malformed unit success fatal, with no completion and no sentinel", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + + const malformed = function* (): Operation { + yield* API.Files.around( + { + // deno-lint-ignore require-yield + *deleteFile(): Operation> { + // A delete answers with nothing at all. A payload here describes an + // outcome this operation does not have, so the provider has broken + // its contract and may not have removed anything. + return fromOutside>({ ok: true, value: "unexpected" }); + }, + }, + { at: "min" }, + ); + }; + + const performed = yield* run( + fixture, + [ + "", + '', + "", + 'sentinel', + ].join("\n"), + stream, + counted, + { between: malformed }, + ); + + // Nothing was written down for it, and nothing ran after it. + expect(recordsOf(performed.events, REPL_REPOSITORY_LOCAL)).toEqual([]); + expect(counted.deletes).toEqual([]); + expect(yield* exists(join(fixture.checkout, "sentinel.md"))).toBe(false); + }); + + it("records a refusal it does not recognize as the ordinary generic one", function* () { + const fixture = yield* useFixture(); + const counted = counters(); + const stream = new InMemoryStream(); + const source = [ + "", + '', + "", + 'after', + ].join("\n"); + + const unknown = function* (): Operation { + yield* API.Files.around( + { + // deno-lint-ignore require-yield + *deleteFile(): Operation> { + // Readable data in this operation's own shape, naming a phase this + // version does not know. The provider *did* describe a failure, + // just not a recognized one, so the document carries on past the + // generic sentence — and nothing it planted travels with it. + return Err( + Object.assign(new Error("delete failed at /planted/absolute/path"), { + data: { type: FILES_ERROR, operation: "delete", phase: "nowhere" }, + }), + ); + }, + }, + { at: "min" }, + ); + }; + + const performed = yield* run(fixture, source, stream, counted, { between: unknown }); + + // An ordinary printed refusal, in the vocabulary's own words, and the work + // after it ran. + expect(performed.refusal).toBe(undefined); + expect(performed.output).toContain("the filesystem operation failed"); + expect(performed.output).not.toContain("/planted/absolute/path"); + expect(yield* readTextFile(join(fixture.checkout, "after.md"))).toBe("after"); + + // Recorded as the refusal it was normalized to, and replayed as that same + // refusal without the provider being asked again. + const records = localRecords(performed.events); + expect(records.length).toBe(2); + expect(records[0]?.outcome.ok).toBe(false); + + const replayed = counters(); + let asked = 0; + const again = yield* run(fixture, source, unfinished(performed.events), replayed, { + *between(): Operation { + yield* API.Files.around( + { + // deno-lint-ignore require-yield + *deleteFile(): Operation> { + asked += 1; + return Err( + Object.assign(new Error("delete failed"), { + data: { type: FILES_ERROR, operation: "delete", phase: "nowhere" }, + }), + ); + }, + }, + { at: "min" }, + ); + }, + }); + expect(again.refusal).toBe(undefined); + expect(again.output).toContain("the filesystem operation failed"); + expect(asked).toBe(0); + expect(replayed.deletes).toEqual([]); + }); +}); diff --git a/packages/git/tests/retained-repository-records.test.ts b/packages/git/tests/retained-repository-records.test.ts new file mode 100644 index 000000000..c2f02e9e0 --- /dev/null +++ b/packages/git/tests/retained-repository-records.test.ts @@ -0,0 +1,376 @@ +/** + * What a REPL entry's records say, and which histories this host will continue. + * + * Portable on purpose: these are the provider-neutral halves of the feature — + * the name, the request digest and the admission — so they are held to the same + * rules under every runtime the CLI starts on. A record format that + * only parsed where the adapter runs would be a format a Node session could + * write and never read back. + * + * Nothing here touches a filesystem or runs Git. The question is what the bytes + * mean and which of them this host refuses to act on. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import type { Operation, Result } from "effection"; +import { Err, Ok } from "effection"; +import type { DurableEvent, Json } from "@executablemd/durable-streams"; +import { fileWriteSuccess, filesFailure, fileWriteFailure } from "@executablemd/runtime"; +import { + parseReplLocalRecord, + parseReplSelectionRecord, + replLocalFingerprint, + replLocalRecordJson, + replLocalRecordName, + replSelectionFingerprint, + replSelectionRecordJson, + replSelectionRecordName, + REPL_REPOSITORY_LOCAL, + REPL_REPOSITORY_SELECTION, + REPL_REPOSITORY_VERSION, + retainableRefusal, +} from "../src/retained-repository/records.ts"; +import type { + ReplLocalOperation, + ReplLocalRequest, + ReplLocalValue, +} from "../src/retained-repository/records.ts"; +import { + admitReplRepositoryHistory, + ReplRepositoryHistoryError, +} from "../src/retained-repository/admission.ts"; + +const WRITE: ReplLocalRequest = { cwd: "/work", path: "note.md", content: "first" }; + +/** The publication a recorded write's success names, where it is one. */ +function receipt(outcome: Result | undefined): string | undefined { + if (outcome === undefined || !outcome.ok) { + return undefined; + } + const value = outcome.value; + return value === undefined || typeof value === "string" ? undefined : value.publication; +} + +/** One journaled local result, named the way the host names it. */ +function* localEvent( + operation: ReplLocalOperation, + request: ReplLocalRequest, + outcome = Ok(operation === "file-write" ? fileWriteSuccess("host-committed") : undefined), + name?: string, +): Operation { + const fingerprint = yield* replLocalFingerprint({ operation, request }); + return { + type: "yield", + coroutineId: "root", + description: { + type: REPL_REPOSITORY_LOCAL, + name: name ?? replLocalRecordName({ operation, fingerprint }), + }, + result: { + status: "ok", + value: replLocalRecordJson({ operation, fingerprint, request, outcome }), + }, + }; +} + +/** The refusal admitting this history produced, or nothing when it admitted. */ +function* refusalOf( + events: readonly DurableEvent[], +): Operation { + try { + yield* admitReplRepositoryHistory(events); + return undefined; + } catch (error) { + if (error instanceof ReplRepositoryHistoryError) { + return error; + } + throw error; + } +} + +describe("what one entry's records say", () => { + it("names an operation by its complete authored request", function* () { + const write = yield* replLocalFingerprint({ + operation: "file-write", + request: WRITE, + }); + const other = yield* replLocalFingerprint({ + operation: "file-write", + request: { ...WRITE, content: "second" }, + }); + const elsewhere = yield* replLocalFingerprint({ + operation: "file-write", + request: { ...WRITE, path: "other.md" }, + }); + const deleted = yield* replLocalFingerprint({ + operation: "file-delete", + request: { cwd: WRITE.cwd, path: WRITE.path }, + }); + + // The content, the path and the kind are all identity. + expect(new Set([write, other, elsewhere, deleted]).size).toBe(4); + // And the same request twice is the same name, which is what makes two + // equal writes one name at two ordered positions. + expect(yield* replLocalFingerprint({ operation: "file-write", request: WRITE })).toBe(write); + }); + + it("reads back a read's text, and refuses a payload that is not one", function* () { + const request = { cwd: "/work", path: "note.md" }; + const fingerprint = yield* replLocalFingerprint({ operation: "file-read", request }); + const payload = replLocalRecordJson({ + operation: "file-read", + fingerprint, + request, + outcome: Ok("first\n"), + }); + + const record = parseReplLocalRecord(payload); + // The exact text, which is what a continuation binds. Not a receipt: a read + // answers with what it read. + expect(record?.outcome.ok === true ? record.outcome.value : undefined).toBe("first\n"); + // A read's request carries no content, so one that does describes an + // operation this host does not perform. + expect( + parseReplLocalRecord({ ...Object(payload), request: { ...request, content: "first\n" } }), + ).toBe(undefined); + // And a success that is not text is not a read this version wrote: the + // absence a delete answers with, and a write's receipt, are both refused + // here exactly as a read's text is refused in their place. + expect(parseReplLocalRecord({ ...Object(payload), success: null })).toBe(undefined); + expect( + parseReplLocalRecord({ + ...Object(payload), + success: { type: "executablemd.files.write-success", publication: "host-committed" }, + }), + ).toBe(undefined); + + // A read's refusal is the ordinary one it was. + const refused = parseReplLocalRecord( + replLocalRecordJson({ + operation: "file-read", + fingerprint, + request, + outcome: Err(filesFailure({ operation: "read", phase: "access", reason: "missing" })), + }), + ); + expect(refused?.outcome.ok).toBe(false); + const data = + refused?.outcome.ok === false ? Reflect.get(refused.outcome.error, "data") : undefined; + expect(Reflect.get(Object(data), "operation")).toBe("read"); + expect(Reflect.get(Object(data), "reason")).toBe("missing"); + }); + + it("reads back a write success, a void success and a refusal", function* () { + const fingerprint = yield* replLocalFingerprint({ + operation: "file-write", + request: WRITE, + }); + + const written = parseReplLocalRecord( + replLocalRecordJson({ + operation: "file-write", + fingerprint, + request: WRITE, + outcome: Ok(fileWriteSuccess("transaction-staged")), + }), + ); + expect(written?.outcome.ok).toBe(true); + expect(receipt(written?.outcome)).toBe("transaction-staged"); + + const removed = parseReplLocalRecord( + replLocalRecordJson({ + operation: "file-delete", + fingerprint, + request: { cwd: WRITE.cwd, path: WRITE.path }, + outcome: Ok(undefined), + }), + ); + // Nothing at all, which is what a delete answers with. No receipt is + // manufactured for it. + expect(removed?.outcome.ok).toBe(true); + expect(removed?.outcome.ok === true && removed.outcome.value).toBe(undefined); + + const refused = parseReplLocalRecord( + replLocalRecordJson({ + operation: "file-write", + fingerprint, + request: WRITE, + outcome: Err(fileWriteFailure({ phase: "commit", reason: "no-space" })), + }), + ); + expect(refused?.outcome.ok).toBe(false); + // Rebuilt as a conforming failure, phase and target included. + const data = + refused?.outcome.ok === false ? Reflect.get(refused.outcome.error, "data") : undefined; + expect(Reflect.get(Object(data), "phase")).toBe("commit"); + expect(Reflect.get(Object(data), "reason")).toBe("no-space"); + }); + + it("refuses a payload carrying anything this version did not write", function* () { + const fingerprint = yield* replLocalFingerprint({ + operation: "file-write", + request: WRITE, + }); + const valid = replLocalRecordJson({ + operation: "file-write", + fingerprint, + request: WRITE, + outcome: Ok(fileWriteSuccess("host-committed")), + }); + expect(parseReplLocalRecord(valid)).not.toBe(undefined); + + // An extra member is a record this host did not write, so what the rest of + // it means is unknown. + expect(parseReplLocalRecord({ ...Object(valid), extra: 1 })).toBe(undefined); + // A write with no content, and a delete carrying some, each describe an + // operation this host does not perform. + expect( + parseReplLocalRecord({ ...Object(valid), request: { cwd: "/work", path: "note.md" } }), + ).toBe(undefined); + expect( + parseReplLocalRecord({ + ...Object(valid), + operation: "file-delete", + success: null, + }), + ).toBe(undefined); + // And a write whose success is the absence a delete answers with. + expect(parseReplLocalRecord({ ...Object(valid), success: null })).toBe(undefined); + }); + + it("says which refusals the record format can carry", function* () { + expect( + retainableRefusal(filesFailure({ operation: "delete", phase: "access", reason: "missing" })), + ).toBe(true); + expect(retainableRefusal(fileWriteFailure({ phase: "lexical", reason: "empty-path" }))).toBe( + true, + ); + // Anything else is a broken provider contract rather than an outcome. + expect(retainableRefusal(new Error("something went wrong"))).toBe(false); + }); + + it("reads back saved routing, and its absence", function* () { + const fingerprint = yield* replSelectionFingerprint({ cwd: "/work" }); + const found = parseReplSelectionRecord( + replSelectionRecordJson({ + cwd: "/work", + checkout: { checkoutRoot: "/work", commonDirectory: "/work/.git" }, + }), + ); + expect(found?.checkout).toEqual({ checkoutRoot: "/work", commonDirectory: "/work/.git" }); + + const absent = parseReplSelectionRecord( + replSelectionRecordJson({ cwd: "/elsewhere", checkout: undefined }), + ); + // Absence is a value this host remembers, not a failure it retries. + expect(absent?.checkout).toBe(null); + expect(replSelectionRecordName({ fingerprint }).length > 0).toBe(true); + }); +}); + +describe("which histories one entry will continue", () => { + it("admits its own records, including repeated equal names", function* () { + const once = yield* localEvent("file-write", WRITE); + const twice = yield* localEvent("file-write", WRITE); + expect(once.type === "yield" && twice.type === "yield").toBe(true); + // The same name at two ordered positions, which is what one entry writing + // the same file twice produces — and what two entries writing the same file + // produce as well. Nothing in a name says which entry wrote it, because + // which history an entry may replay is its own segment's range rather than + // a label inside the record. + expect(once.type === "yield" ? once.description.name : "").toBe( + twice.type === "yield" ? twice.description.name : "", + ); + expect(yield* refusalOf([once, twice])).toBe(undefined); + }); + + it("passes over records of other features and other types", function* () { + const foreign: DurableEvent = { + type: "yield", + coroutineId: "root", + description: { type: "exec", name: "echo" }, + result: { status: "ok", value: null }, + }; + expect( + yield* refusalOf([ + foreign, + yield* localEvent("dir-ensure", { + cwd: "/work", + path: "drafts", + }), + ]), + ).toBe(undefined); + }); + + it("refuses a record whose request no longer produces its name", function* () { + const event = yield* localEvent("file-write", WRITE); + // The request is changed after the work was saved, which is the one way a + // saved result could answer a question nobody asked. + const tampered: DurableEvent = { + ...event, + result: { + status: "ok", + value: { + ...Object(saved(event)), + request: { cwd: "/work", path: "note.md", content: "somebody else's" }, + }, + }, + }; + const refusal = yield* refusalOf([tampered]); + expect(refusal?.reason).toBe("request-fingerprint"); + }); + + it("refuses a record journaled under a name it does not produce", function* () { + const event = yield* localEvent("file-write", WRITE, undefined, "something-else"); + const refusal = yield* refusalOf([event]); + expect(refusal?.reason).toBe("request-fingerprint"); + }); + + it("refuses a record of this type it cannot read", function* () { + const malformed: DurableEvent = { + type: "yield", + coroutineId: "root", + description: { type: REPL_REPOSITORY_LOCAL, name: "whatever" }, + result: { + status: "ok", + value: { version: REPL_REPOSITORY_VERSION, operation: "file-write" }, + }, + }; + const refusal = yield* refusalOf([malformed]); + expect(refusal?.reason).toBe("malformed-record"); + }); + + it("refuses a record stating a version this host does not implement", function* () { + const later: DurableEvent = { + type: "yield", + coroutineId: "root", + description: { type: REPL_REPOSITORY_SELECTION, name: "whatever" }, + result: { status: "ok", value: { version: REPL_REPOSITORY_VERSION + 1, cwd: "/work" } }, + }; + const refusal = yield* refusalOf([later]); + expect(refusal?.reason).toBe("unknown-version"); + // A later version is told apart from a broken record, so the person is + // told which of the two they have. + expect(refusal?.message).toContain(`version ${REPL_REPOSITORY_VERSION + 1}`); + }); + + it("admits an empty history", function* () { + const events: readonly DurableEvent[] = []; + expect(yield* refusalOf(events)).toBe(undefined); + // The positive control: this admission can refuse, so admitting nothing is + // an answer rather than a scan that never looked. + const refusal = yield* refusalOf([ + yield* localEvent("file-write", WRITE, undefined, "not-its-own-name"), + ]); + expect(refusal?.reason).toBe("request-fingerprint"); + }); +}); + +/** The payload one journaled event carries, for a fixture that edits it. */ +function saved(event: DurableEvent): Json { + if (event.type !== "yield" || event.result.status !== "ok") { + throw new Error("the fixture event carries no payload"); + } + return event.result.value ?? null; +} diff --git a/packages/workflow/src/deno/advisory-lock.ts b/packages/workflow/src/deno/advisory-lock.ts index 726f2c0e4..39b22f86c 100644 --- a/packages/workflow/src/deno/advisory-lock.ts +++ b/packages/workflow/src/deno/advisory-lock.ts @@ -100,26 +100,47 @@ function release(file: AdvisoryLockFile): void { * The file is closed immediately when the lock is refused: a descriptor that * holds nothing has nothing to release, and keeping it open until the scope * ends would make a refusal cost the caller a handle it never acquired. + * + * The release is registered before the file is opened, like its waiting sibling + * below, so no acquisition and no cancellation between the open and the lock can + * leave an open or locked file nothing owns. Registering it afterwards is the + * ordering a descriptor outliving its owner comes from: `ensure` suspends, and + * an owner halted while it registers unwinds with no cleanup on it at all. */ export function useAdvisoryLock(path: string): Operation { return resource(function* (provide) { yield* ensureDir(dirname(path)); - const file = locking().openSync(path, { read: true, write: true, create: true }); + let file: AdvisoryLockFile | undefined; let locked = false; + let closed = false; + yield* ensure(() => { + if (file === undefined || closed) { + return; + } + closed = true; + if (locked) { + release(file); + } else { + file.close(); + } + }); + + file = locking().openSync(path, { read: true, write: true, create: true }); try { locked = file.tryLockSync(true); } catch (error) { + closed = true; file.close(); throw error; } if (!locked) { + closed = true; file.close(); yield* provide(undefined); return; } - yield* ensure(() => release(file)); yield* provide(file); }); } diff --git a/scripts/bench-repl-file-dir.ts b/scripts/bench-repl-file-dir.ts new file mode 100644 index 000000000..2796d5659 --- /dev/null +++ b/scripts/bench-repl-file-dir.ts @@ -0,0 +1,432 @@ +/** + * What retaining a REPL entry's file work costs, measured rather than asserted. + * + * Three columns, over the same authored document and the same seeded layout: + * + * - **ordinary** — what `xmd run` assembles for file work: the host filesystem + * provider and this package's component declarations, and nothing else. + * - **entry** — the same, plus the executing entry's repository host. Every + * mutation inside the checkout is recorded, and the timing includes the host's + * assembly and its first lazy acquisitions, because a person waiting for the + * first write waits for those too. + * - **continuation** — the same document again, over the journal the entry + * column wrote, with the root close removed. Every operation is complete, so + * this is the cost of answering from records rather than performing. + * + * The comparison is only fair if the work is identical, so each sample gets its + * own independently seeded checkout, prepared outside every timed region, and + * both live modes expand the same document in the same relative layout. There + * is no terminal here and nobody waits for a question: painting a screen and + * waiting for a person are not what this measures. + * + * It states no budget and no pass/fail ratio. What it produces is raw samples, + * medians and counters for a reviewer to read. + */ + +import { main, scoped, until } from "effection"; +import type { Operation } from "effection"; +import { exists, rm, writeTextFile } from "@effectionx/fs"; +import { mkdir, mkdtemp, realpath } from "node:fs/promises"; +import { execFile } from "node:child_process"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import process from "node:process"; +import { API } from "@executablemd/runtime"; +import type { FilePathInput, FilesHandler, FileWriteInput } from "@executablemd/runtime"; +import { hostFilesHandler } from "@executablemd/runtime"; +import { collect } from "@executablemd/core"; +import { executeInstalled } from "@executablemd/core/host"; +import { InMemoryStream } from "@executablemd/durable-streams"; +import type { DurableEvent } from "@executablemd/durable-streams"; +import { useCompositionComponents } from "../packages/git/src/composition/installation.ts"; +import { Execution } from "@executablemd/core/api"; +import { useReplRepositoryRecording } from "../packages/git/src/deno/retained-repository/provider.ts"; +import { admitReplRepositoryHistory } from "../packages/git/src/retained-repository/admission.ts"; +import { denoRepositoryHost } from "../packages/git/src/deno/composition/host.ts"; +import { commonDirectoryLock } from "../packages/git/src/deno/run-composition/placement.ts"; +import { + REPL_REPOSITORY_LOCAL, + REPL_REPOSITORY_SELECTION, +} from "../packages/git/src/retained-repository/records.ts"; + +/** What one invocation was asked to measure. */ +interface Settings { + readonly samples: number; + readonly operations: number; + readonly payloadBytes: number; + /** + * How much unrelated content the checkout holds, per comparison. + * + * One whole comparison per size, so the two are read beside each other: a + * cost that grew with content the document never named would be a cost of + * looking at the checkout, and these columns are what would show it. + */ + readonly unrelatedFiles: readonly number[]; +} + +function settings(argv: readonly string[]): Settings { + const read = (name: string, fallback: number): number => { + const found = argv.find((entry) => entry.startsWith(`--${name}=`)); + if (found === undefined) { + return fallback; + } + const value = Number(found.slice(`--${name}=`.length)); + if (!Number.isSafeInteger(value) || value <= 0) { + throw new Error(`--${name} takes a positive whole number, and was given ${found}`); + } + return value; + }; + const operations = read("operations", 40); + if (operations % 4 !== 0) { + throw new Error( + `--operations is split equally between directory ensures, reads, writes and deletions, ` + + `so it takes a multiple of four; ${operations} is not one`, + ); + } + const sizes = argv.find((entry) => entry.startsWith("--unrelated-files=")); + const unrelatedFiles = + sizes === undefined + ? [0] + : sizes + .slice("--unrelated-files=".length) + .split(",") + .map((entry) => { + const value = Number(entry); + if (!Number.isSafeInteger(value) || value < 0) { + throw new Error(`--unrelated-files takes whole counts, and was given ${entry}`); + } + return value; + }); + return { + samples: read("samples", 5), + operations, + payloadBytes: read("payload-bytes", 1024), + unrelatedFiles, + }; +} + +/** What the composition beneath was actually asked to do. */ +interface Counters { + reads: number; + writes: number; + deletes: number; + ensures: number; + /** Pure path admissibility checks, which touch no file. */ + checks: number; + /** Searches of the tree, which are the only traversal a document can ask for. */ + globs: number; + /** Every `git` this entry's session ran, which is what discovery costs. */ + commands: number; + /** Journaled records of this feature's own types, which are its appends. */ + records: number; + /** The UTF-8 size of the whole journal this run produced. */ + journalBytes: number; + /** Whether the entry took the checkout's lease during this run. */ + lease: number; +} + +function counters(): Counters { + return { + reads: 0, + writes: 0, + deletes: 0, + ensures: 0, + checks: 0, + globs: 0, + commands: 0, + records: 0, + journalBytes: 0, + lease: 0, + }; +} + +function countingFiles(counted: Counters): FilesHandler { + const host = hostFilesHandler(); + return { + *checkFilePath(input: FilePathInput) { + counted.checks += 1; + return yield* host.checkFilePath(input); + }, + *readTextFile(input: FilePathInput) { + counted.reads += 1; + return yield* host.readTextFile(input); + }, + *globFiles(input: Parameters[0]) { + counted.globs += 1; + return yield* host.globFiles(input); + }, + temporaryDirectory: () => host.temporaryDirectory(), + *writeTextFile(input: FileWriteInput) { + counted.writes += 1; + return yield* host.writeTextFile(input); + }, + *deleteFile(input: FilePathInput) { + counted.deletes += 1; + return yield* host.deleteFile(input); + }, + *ensureDirectory(input: FilePathInput) { + counted.ensures += 1; + return yield* host.ensureDirectory(input); + }, + }; +} + +function countingHost(counted: Counters) { + const host = denoRepositoryHost(); + return { + git(invocation: Parameters[0]) { + counted.commands += 1; + return host.git(invocation); + }, + useDirectory: () => host.useDirectory(), + }; +} + +/** + * The document both live modes expand. + * + * One `` per quarter of the operations, each holding one read of a file the + * fixture preseeded, one exact-size write and one deletion, so the four kinds + * happen in one order and one relative layout whichever mode is running. The + * read comes before the deletion because it reads the file the deletion + * removes: a read of an absent file is a refusal, and a refusal is a different + * measurement. + */ +function document(chosen: Settings): string { + const each = chosen.operations / 4; + const payload = "x".repeat(chosen.payloadBytes); + const lines: string[] = []; + for (let index = 0; index < each; index += 1) { + lines.push(``); + lines.push(``); + lines.push(`${payload}`); + lines.push(``); + lines.push(""); + } + return lines.join("\n"); +} + +/** One checkout, seeded identically for every sample and every mode. */ +function* seed(chosen: Settings, unrelated: number): Operation { + const root = yield* until(realpath(yield* until(mkdtemp(join(tmpdir(), "xmd-bench-"))))); + const checkout = join(root, "checkout"); + yield* until(mkdir(checkout, { recursive: true })); + yield* run("git", ["init", "-b", "main", checkout], root); + const each = chosen.operations / 4; + for (let index = 0; index < each; index += 1) { + const part = join(checkout, `part-${index}`); + yield* until(mkdir(part, { recursive: true })); + yield* writeTextFile(join(part, "preseeded.md"), "preseeded\n"); + } + // Content the document never names, in a subtree it never enters. What it is + // here to show is whether a cost grows with the size of the checkout. + if (unrelated > 0) { + const elsewhere = join(checkout, "unrelated"); + yield* until(mkdir(elsewhere, { recursive: true })); + const payload = "u".repeat(chosen.payloadBytes); + for (let index = 0; index < unrelated; index += 1) { + yield* writeTextFile(join(elsewhere, `file-${index}.md`), payload); + } + } + return checkout; +} + +function run(command: string, args: readonly string[], cwd: string): Operation { + return until( + new Promise((resolve, reject) => { + execFile(command, [...args], { cwd }, (error) => + error === null ? resolve() : reject(error), + ); + }), + ); +} + +/** One measured run: how long it took, and what it really did. */ +interface Measured { + readonly ms: number; + readonly counted: Counters; + readonly events: readonly DurableEvent[]; +} + +/** Expand the document once, in the mode named, and measure it. */ +function measure( + mode: "ordinary" | "entry" | "continuation", + checkout: string, + source: string, + retained: readonly DurableEvent[] = [], +): Operation { + return scoped(function* (): Operation { + const counted = counters(); + const stream = new InMemoryStream([...retained]); + const path = join(checkout, "entry.md"); + yield* writeTextFile(path, source); + yield* API.Env.around( + { + // deno-lint-ignore require-yield + *cwd() { + return checkout; + }, + }, + { at: "min" }, + ); + const files = countingFiles(counted); + yield* API.Files.around( + { + checkFilePath: ([input]) => files.checkFilePath(input), + readTextFile: ([input]) => files.readTextFile(input), + globFiles: ([input]) => files.globFiles(input), + temporaryDirectory: () => files.temporaryDirectory(), + writeTextFile: ([input]) => files.writeTextFile(input), + deleteFile: ([input]) => files.deleteFile(input), + ensureDirectory: ([input]) => files.ensureDirectory(input), + }, + { at: "min" }, + ); + yield* useCompositionComponents(); + + const lock = commonDirectoryLock(yield* until(realpath(join(checkout, ".git")))); + const before = yield* exists(lock); + + // Timed from here, so the entry setup and its first lazy acquisitions are + // inside the measurement rather than beside it. + const started = performance.now(); + if (mode === "ordinary") { + yield* collect(yield* executeInstalled({ path, stream }, [])); + } else { + // Registered the way the REPL host registers it: core initialization is + // what establishes the recording, which is why the registration is inside + // the timed region along with the first lazy acquisitions it leads to. + yield* Execution.around({ + *init(_args, next): Operation { + yield* useReplRepositoryRecording({ host: countingHost(counted) }); + yield* next(); + }, + }); + yield* collect( + yield* executeInstalled({ path, stream }, [ + { admissions: [(retained) => admitReplRepositoryHistory(retained)] }, + ]), + ); + } + const ms = performance.now() - started; + + if (!before && (yield* exists(lock))) { + counted.lease = 1; + } + const events = yield* stream.readAll(); + counted.journalBytes = new TextEncoder().encode(JSON.stringify(events)).length; + counted.records = events.filter( + (event) => + event.type === "yield" && + (event.description.type === REPL_REPOSITORY_LOCAL || + event.description.type === REPL_REPOSITORY_SELECTION), + ).length; + return { ms, counted, events }; + }); +} + +/** Everything before the root close, which is what a continuation reopens. */ +function unfinished(events: readonly DurableEvent[]): readonly DurableEvent[] { + return events.filter((event) => event.type !== "close"); +} + +function median(values: readonly number[]): number { + const sorted = values.toSorted((left, right) => left - right); + const middle = Math.floor(sorted.length / 2); + return sorted.length % 2 === 1 + ? (sorted[middle] ?? 0) + : ((sorted[middle - 1] ?? 0) + (sorted[middle] ?? 0)) / 2; +} + +function report(name: string, samples: readonly Measured[]): void { + const times = samples.map((sample) => sample.ms); + const counted = samples[0]?.counted ?? counters(); + console.log( + `${name.padEnd(13)} median ${median(times).toFixed(1).padStart(8)}ms ` + + `range ${Math.min(...times).toFixed(1)}–${Math.max(...times).toFixed(1)}ms`, + ); + console.log(`${"".padEnd(13)} raw ${times.map((value) => value.toFixed(1)).join(", ")}`); + console.log( + `${"".padEnd(13)} reads ${counted.reads} writes ${counted.writes} ` + + `deletes ${counted.deletes} ensures ${counted.ensures} checks ${counted.checks} ` + + `globs ${counted.globs}`, + ); + console.log( + `${"".padEnd(13)} git ${counted.commands} lease ${counted.lease} ` + + `records ${counted.records} journal ${counted.journalBytes} bytes`, + ); +} + +await main(function* (argv): Operation { + const chosen = settings(argv); + const source = document(chosen); + + console.log("# A REPL entry's file work, measured"); + console.log(""); + console.log(`samples ${chosen.samples}`); + const each = chosen.operations / 4; + console.log( + `operations ${chosen.operations} per sample ` + + `(${each} directory ensures, ${each} reads of preseeded files, ${each} writes of exactly ` + + `${chosen.payloadBytes} bytes, ${each} deletions of those preseeded files)`, + ); + console.log(`unrelated checkout content ${chosen.unrelatedFiles.join(", ")} files`); + console.log(`deno ${Deno.version.deno} v8 ${Deno.version.v8}`); + console.log(`host ${process.platform} ${process.arch}`); + + for (const unrelated of chosen.unrelatedFiles) { + console.log(""); + console.log( + `## ${unrelated} unrelated ${chosen.payloadBytes}-byte files under an untouched subtree`, + ); + console.log(""); + + // One unreported warmup per live mode, so neither column is charged for the + // module graph the other one already paid for. + for (const mode of ["ordinary", "entry"] as const) { + const checkout = yield* seed(chosen, unrelated); + yield* measure(mode, checkout, source); + yield* rm(join(checkout, ".."), { recursive: true, force: true }); + } + + const ordinary: Measured[] = []; + const entry: Measured[] = []; + const continuation: Measured[] = []; + + for (let sample = 0; sample < chosen.samples; sample += 1) { + // Alternated, so a drift in the machine across the run does not land on + // one column. + const order = + sample % 2 === 0 ? (["ordinary", "entry"] as const) : (["entry", "ordinary"] as const); + for (const mode of order) { + const checkout = yield* seed(chosen, unrelated); + const measured = yield* measure(mode, checkout, source); + (mode === "ordinary" ? ordinary : entry).push(measured); + if (mode === "entry") { + // The continuation runs against the journal that entry just wrote, in + // the checkout it wrote it in, with the lease sidecar removed so a + // second acquisition would be visible. Nothing holds it: the live run + // has ended. + const lock = commonDirectoryLock(yield* until(realpath(join(checkout, ".git")))); + yield* rm(lock, { force: true }); + continuation.push( + yield* measure("continuation", checkout, source, unfinished(measured.events)), + ); + } + yield* rm(join(checkout, ".."), { recursive: true, force: true }); + } + } + + report("ordinary", ordinary); + console.log(""); + report("entry", entry); + console.log(""); + report("continuation", continuation); + } + console.log(""); + console.log( + "A continuation performs no read, no mutation, no traversal, runs no `git` and takes no " + + "lease, which is what the zeroes above say. This states no budget and no ratio: it is raw " + + "evidence for a reviewer.", + ); +}); diff --git a/scripts/runtime-test-exclusions.ts b/scripts/runtime-test-exclusions.ts index fd0a8c93e..3c99c175c 100644 --- a/scripts/runtime-test-exclusions.ts +++ b/scripts/runtime-test-exclusions.ts @@ -608,6 +608,12 @@ const COMPILED_BINARY: RuntimeExclusion[] = [ "asks the compiled `dist/xmd` to load a Plugin module that is not embedded in it — a binary with no checkout, no node_modules and no module graph to add to, which only `deno compile` produces and no test shard builds; it belongs beside the other compiled-binary evidence the `smoke` job builds for through README.md#Build", issue: "https://github.com/taras/executable.md/issues/567", }, + { + path: "scripts/tests/repl-repositories-compiled.test.ts", + reason: + "drives the compiled `dist/xmd` through a real terminal: it starts a REPL inside a seeded checkout, leaves at a question and reopens the printed location to prove the binary carries the same entry host the source does. Only `deno compile` produces it and no test shard builds it, so it belongs beside the other compiled-binary evidence the `smoke` job builds for through README.md#Build", + issue: "https://github.com/taras/executable.md/issues/567", + }, ]; /** @@ -670,6 +676,24 @@ const DENO_ONLY_REPOSITORY_PROVIDER: RuntimeExclusion[] = [ "a nested execution's own provider instance, which needs the same kernel-released advisory lock the root's does, and real subprocesses to prove where a child stands", issue: DERIVED_SCOPE, }, + { + path: "packages/git/tests/retained-repository-local.test.ts", + reason: + "the subject is a REPL entry's native repository host: it discovers a real checkout with a `git` subprocess and records what a real filesystem answered. The record format, the owner and the admission are provider-neutral and are proven on every runtime by packages/git/tests/retained-repository-records.test.ts", + issue: DERIVED_SCOPE, + }, + { + path: "packages/git/tests/retained-repository-lifecycle.test.ts", + reason: + "holds a shared Git directory under a kernel-released exclusive advisory lock taken through the Deno runtime, and spawns a second Deno process to prove the cross-process half; Node and Bun expose no equivalent", + issue: DERIVED_SCOPE, + }, + { + path: "packages/cli/tests/repl-repositories.test.ts", + reason: + "drives the production entry-host factory, which only Deno and the compiled binary supply: its module graph reaches @executablemd/git/deno and so Workflow's storage adapter and node:sqlite. That Node and Bun start a REPL and perform ordinary File and Dir work without it is proven by packages/cli/tests/repl-repositories-pty.test.ts, which runs everywhere", + issue: DERIVED_SCOPE, + }, ]; const BUN_MISSING_NODE_SQLITE: RuntimeExclusion[] = [ diff --git a/scripts/tests/repl-repositories-compiled.test.ts b/scripts/tests/repl-repositories-compiled.test.ts new file mode 100644 index 000000000..6145c1259 --- /dev/null +++ b/scripts/tests/repl-repositories-compiled.test.ts @@ -0,0 +1,40 @@ +/** + * The same journey, against the compiled binary (#181 E1). + * + * A compiled `xmd` has no checkout to read from and no module graph to lose a + * module out of — it has whatever `deno compile` embedded. The entry host is + * assembled by `compiled.ts`, which is a different entrypoint from the one the + * source row runs, so a binary that was built without it would start a REPL, + * perform the file work and retain none of it: the defect would appear at a + * person's first reopened location rather than here. + * + * It runs the *same* journey as the source row, from one shared module, so the + * two cannot drift into asserting different things. + * + * It runs against `dist/xmd`, so `deno task build` has to have happened. A + * missing binary is reported as the setup it is rather than as a failure of the + * behavior. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { shellQuote } from "@executablemd/test-support/launch"; +import { exists } from "@effectionx/fs"; +import type { Operation } from "effection"; +import { fileURLToPath as fromFileUrl } from "node:url"; +import { join } from "node:path"; +import { replFilesJourney } from "../../packages/cli/tests/support/repl-files-journey.ts"; + +const ROOT = fromFileUrl(new URL("../..", import.meta.url)); +const BINARY = join(ROOT, "dist", "xmd"); + +describe("the compiled binary's REPL entry files", () => { + it("E1: carries the entry host, so a completed prefix answers again", function* (): Operation { + if (!(yield* exists(BINARY))) { + throw new Error( + `this row drives the compiled binary and ${BINARY} is not there. Run \`deno task build\` ` + + "first; a skipped row is missing evidence rather than a passing behavior.", + ); + } + yield* replFilesJourney((args) => [shellQuote(BINARY), ...args].join(" ")); + }); +}); diff --git a/scripts/tests/runtime-exclusions.test.ts b/scripts/tests/runtime-exclusions.test.ts index ccad4c612..4daa45e54 100644 --- a/scripts/tests/runtime-exclusions.test.ts +++ b/scripts/tests/runtime-exclusions.test.ts @@ -83,6 +83,7 @@ describe("runtime exclusions", () => { "scripts/tests/component-form-dispatch.test.ts", "scripts/tests/plan-component-compiled.test.ts", "scripts/tests/plugin-compiled.test.ts", + "scripts/tests/repl-repositories-compiled.test.ts", ]); }); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 7ef11a24c..01d95a131 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -8858,6 +8858,26 @@ restore: the read happens again against whatever the file says now, and the write happens again. Inside a wrapping `` that repetition is what the directory's replay refusal depends on (§6.11). +Those two paragraphs are about this component, which journals nothing. A *host* +may journal what its filesystem provider was asked to do, and one does: the +interactive `xmd repl` retains the reads and writes an entry performed in the +ambient checkout it was started in, so a partial continuation of that entry +answers a completed one from what it produced instead of reading or writing +again (`specs/repl-spec.md`). A retained read answers with the exact text it +read, so the binding its `as` prop produced is the recorded text rather than +whatever the file says at continuation time; a read the entry had not reached is +new work and reads the file as it is then. + +That changes nothing stated here. The component still performs no durable +effect, still calls `checkFilePath` before its children and `writeTextFile` +after them, still reports from the fixed vocabulary, and still treats a provider +that is absent, that refuses or that breaks its contract as fatal. Terminal +permission still applies to the operation the provider is asked to perform, +after any transformation, and a completed result is answered before that +question is reached. What differs is only what a host established for the +execution — through `Execution.init`, before any installation callback — and +under `xmd run` and a workflow run no host journals these calls at all. + --- ### 6.13.1 Removing a file: `` @@ -9005,6 +9025,17 @@ restores its recorded outcome and removes nothing a second time, even where the path exists again; a partial replay continues live from the retained Workspace frontier and performs the first unrecorded deletion once. +Under the interactive `xmd repl` the three mutation replays differ again, and +only where the entry is standing in a Git checkout. The host retains what its +provider answered, so continuing an interrupted entry answers a completed +deletion from its record and removes nothing — including where the path exists +again, because nothing is compared with the live filesystem +(`specs/repl-spec.md`). Outside that checkout nothing is retained and the +`xmd run` rules above apply as written. The component is the same component in +all three: form dispatch, failed-child behavior, containment, the atomic native +semantics, the printed and fatal failure split, and generated admission are +unchanged by which host is beneath it. + #### Runtime portability The component behaves identically under Deno, Node, Bun, and the compiled diff --git a/specs/repl-spec.md b/specs/repl-spec.md index 20d1eb6c5..be67fd7e1 100644 --- a/specs/repl-spec.md +++ b/specs/repl-spec.md @@ -765,6 +765,61 @@ Sessions to follow a conversation and coming back returns you to the entry you left. It never carries focus, pause state, how far a window is scrolled, form internals, rendered cells or this process's overlay. +### The files an entry wrote in the checkout you are standing in + +Start the REPL inside a Git checkout and an entry's file work is retained too, +under the ordinary records above and no record type of its own. Four things are +kept: a self-closing `` text read; a paired +`…` write, after its content has expanded; a self-closing +``; and the directory a paired `…` +ensures, before its content is entered. The reads, writes and deletions an +admitted `` fragment performs are kept on the same terms, because a +fragment's operations go to the same document filesystem an authored one does. + +What that buys is the one thing experimentation needs: leave an entry at a +question, come back to the printed location, and the work it already did is not +done again. The question is offered again, and the write after it happens for +the first time when you answer. + +A completed operation answers from what it produced, and reads nothing. So a +file somebody else changed in the meantime stays changed, a file somebody +deleted stays deleted, and a checkout you have since moved or removed does not +stop the entry continuing — only a *new* operation needs the checkout to be +there, and it fails or succeeds exactly as it would have the first time. The +honest cost of that is one window: work can happen and its outcome can fail to +be recorded, and a continuation then does that one operation again. Work that +*was* recorded before you left is answered from the record, so an edit somebody +made in the meantime survives the continuation untouched. + +A retained read is the same rule, which is worth saying plainly because its +answer is a value you can see. The binding a completed read produced is the text +it read when it ran: a question showing that binding still shows it after the +file has changed, because nothing re-reads the file to answer an operation that +is already done. Type a *new* entry that reads the same path and it reads what +is there now. What is retained is each operation, not a value the session +remembers for a path. + +A mutation anywhere else — beside the document, in a temporary directory, in any +directory that is not inside the checkout the entry started in — is ordinary +file work, is retained by nothing here, and happens again if the entry is +continued. Reads, searches and temporary directories are ordinary throughout. +The lexical path rules and the permission a `` already answers to are +unchanged, and they are not checkout validation: a path this run may not write +is refused the same way it is anywhere else. + +One entry at a time works in one repository. A second entry — in this process or +another — that needs the same checkout is told so immediately rather than +waiting: + +```text +Another XMD entry or process holds this checkout. End that work before trying +again. Nothing was read and nothing was changed. +``` + +Starting outside a checkout retains none of this and changes nothing else: the +entry performs ordinary file work, exactly as it would anywhere else, and never +looks for a repository twice. + ## What the head says the execution is doing Seven states and no eighth, in one precedence, each a fact about retained @@ -810,6 +865,19 @@ recorded output replaces it. ## Who owns what +Repository behavior belongs to the entry that is executing, and to nothing +wider. The command selects it; the entry's own task prepares it, once, inside +that task, and everything it takes — the Git session it reads through, where it +looked for a checkout, the claim on that repository — is released when the entry +is torn down. Each entry reads and writes its own range of the one history, so +the same text typed twice is two sets of records rather than one replayed, and +the entry after an interrupted one replays nothing of it however identical its +text. No second name for the entry appears in a record: which history an entry +may continue is the range it was given. Reading is not owning: a settled +history, a frozen position and a refused location prepare nothing, look for no +checkout and claim nothing. A host that registers no recording runs the same +file work with nothing retained, which is what Node and Bun do. + Components receive immutable view data and answer with a typed semantic action. They hold no events, no history, no session, no repository and no host operation, so a keystroke cannot reach the file except through the one place that owns the @@ -835,7 +903,19 @@ schedules no timer. ## Exclusions - Two processes writing one execution is unsupported. There is no lease protocol; - do not open the same location twice for writing. + do not open the same location twice for writing. The claim on a checkout is a + different thing: it says one entry at a time may mutate one repository, and it + says nothing about who may write this execution's history. +- Repository components are not part of this. ``, ``, the + local Git operations, `` and `` refuse in a REPL entry + exactly as they do under any host that installs no provider for them. What an + entry retains is file and directory work in the checkout it was started in, + and nothing is managed, cloned, staged, committed or published. +- Continuing an entry is explicit. Returning to the live head starts the one + unfinished entry, once; a frozen position starts nothing at all. +- A continuation does not reconcile. Nothing compares a saved result with what + is on disk, reports a difference or offers to repair one — if what you are + looking at is not what you expected, that is for you to read and act on. - `xmd repl` takes one optional location and the five Agent options above. It renders no file and takes no document reference. - `` is unavailable. This command has no terminal to give away — diff --git a/specs/workflow-workspace-spec.md b/specs/workflow-workspace-spec.md index 888e164ff..386d7ddfb 100644 --- a/specs/workflow-workspace-spec.md +++ b/specs/workflow-workspace-spec.md @@ -52,7 +52,7 @@ The host creates a WorkflowRun, gives it an implicit Workspace, streams the rendered document output and retains enough state to continue after the process exits. -## 2. Two execution environments +## 2. Three execution environments ### 2.1 `xmd run` @@ -124,6 +124,29 @@ environment. The same declarative components work in both environments. Durability comes from the workflow host, not from another spelling of `` or ``. +### 2.3 `xmd repl` + +The interactive REPL is specified in `specs/repl-spec.md` rather than here. One +distinction belongs in this document, because it is about mutation replay. + +A REPL entry performs ordinary file work against the caller's own filesystem, +like `xmd run`. What differs is that the host retains the writes, deletions and +`` ensures the entry performed in the ambient Git checkout it was started +in, so an interrupted entry can be continued: a completed operation answers from +its record and touches nothing, and only an operation with no record executes. +Nothing is compared with the live filesystem, nothing is rolled back and no +frontier is advanced — a recorded result and what is on disk may disagree, and +the person reading the screen is what resolves that. + +So the three environments retain mutation differently, and none of the other two +changes. `xmd run` journals no filesystem mutation at all. A workflow run's +mutations are Workspace effects inside a transaction, ordered by the frontier +§9 and §10 describe, and the REPL has none of that: no Workspace, no +transaction, no frontier and no logical filesystem. Repository components — +``, ``, the local Git operations, `` and +`` — are not available to a REPL entry and refuse there as they do +under any host that installs no provider for them. + ## 3. Workflow lifecycle The immutable workflow definition is established by the workflow-run contract: a @@ -1103,6 +1126,16 @@ only cwd. It neither selects nor replaces the Repository in scope, so a Git operation inside a directory beneath a selected checkout still addresses that same Repository. +Under the interactive `xmd repl` the ensure is retained where the target is +inside the checkout the entry was started in, so continuing an interrupted entry +answers a completed ensure from its record and asks the provider for nothing +(`specs/repl-spec.md`). Everything stated above is unchanged by that: the ensure +still precedes the cwd installation and every child, the refusal is still fixed +and sanitized, the cwd is still restored on success, failure and cancellation, +and `` still neither selects nor replaces the Repository in scope. A +generated fragment gains no `` there either — the REPL's write table is the +ordinary one, which does not hold it. + Self-closing `` is invalid, and which form it was written as comes from the invocation the engine issued rather than from the contextual `hasContent()` chain — a handler installed outside the invocation answers that chain first, From 6d82e41be49653c7c6121c254d636bd1e9614b5e Mon Sep 17 00:00:00 2001 From: Taras Mankovski Date: Fri, 16 Oct 2026 02:41:23 -0400 Subject: [PATCH 2/4] =?UTF-8?q?=F0=9F=90=9B=20Keep=20a=20retained=20read?= =?UTF-8?q?=20out=20of=20the=20mutation=20lease=20(#181,=201/N)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses the Architect's R1 and the Planner's B1–B3 on `a0306e5a`. Both read-lease probes reproduced before anything changed: the Architect's live read came back with the contention sentence, and the Planner's observed one hold where there should be none. **R1/B1.** `routed()` took the checkout's lease for every new in-checkout operation, reads included, so a read refused while another entry was working and made anybody else wait while it was only looking. Being retained and being exclusive are different questions: what the lease answers is that one entry at a time *changes* a repository. Only a new write, deletion or directory ensure acquires it now. A read is still routed, still recorded, still permission- checked, and a completed one still answers before any of that. The regression lives in the lifetime suite beside the other resource rows: one entry holds the lease at a question, a second entry reads the same file, and the read succeeds, is retained, and opens no descriptor — observed at the native function the advisory lock opens with. Restoring the unconditional hold fails it. **B2.** The configured Plugin copies the helper assembly at construction — scalars and the nested launcher environment — and initializes executions from that copy. The caller's object is neither mutated nor frozen; what an execution gets is what this host selected, not whatever the caller's reference says later. **B3.** The benchmark measures what it said it measured. Ordinary mode now initializes the production ordinary composition inside its own timed region, so both live columns are initialized executions. The lease column counts lock descriptors actually opened, at `Deno.openSync`, with the persistent sidecar left exactly where it is — the old column inferred acquisition from a file it had deleted first. Appends are counted apart from retained records, which is how the continuation column can say it appends the engine's closing event and no local result. The `globs` and `checks` columns say what they are, and the report states plainly that this harness does not observe `node:fs` and so cannot claim comprehensive traversal coverage. Its first numbers stand on their own; the R6 report is preserved unchanged. The REPL specification no longer says reads are ordinary throughout — a read inside the checkout is retained, and searches, temporary directories and outside-checkout reads keep their ordinary scope — and it describes exclusion as applying to a new mutation rather than to every entry that needs a checkout. The architecture says the same, and the exact contention sentence is unchanged. --- architecture.md | 19 +- packages/cli/src/deno-repositories.ts | 30 +++- .../git/src/deno/retained-repository/files.ts | 13 +- .../retained-repository-lifecycle.test.ts | 84 ++++++++- scripts/bench-repl-file-dir.ts | 166 +++++++++++++++--- specs/repl-spec.md | 28 +-- 6 files changed, 285 insertions(+), 55 deletions(-) diff --git a/architecture.md b/architecture.md index 99ed93a62..a2b20c0dd 100644 --- a/architecture.md +++ b/architecture.md @@ -6510,8 +6510,8 @@ edit survives, while work performed and not acknowledged is attempted again and may overwrite one. That window is what the REPL trades for not snapshotting anybody's working tree. -A shared Git directory admits one live entry at a time, and one mechanism says -so: a kernel-released exclusive lock on a hashed sidecar outside every checkout. +A shared Git directory admits one live *mutating* entry at a time, and one +mechanism says so: a kernel-released exclusive lock on a hashed sidecar outside every checkout. `flock` and `LockFileEx` both exclude *independently opened* handles, two in one process included, so a second entry here is refused by the operating system exactly as a second process is — the native descriptor is the whole live @@ -6520,10 +6520,15 @@ caller could substitute. It is taken through the execution's own lazy single-flight acquisition, so one entry holds one handle however many mutations ask for it, with the release registered before the descriptor is opened and the hold lasting until the entry is torn down rather than until an element returns. -A completed read acquires nothing: an observation is not a new exclusive writer. -It is not exclusion against arbitrary Git tools, a person editing the checkout, -or any process that does not participate in this sidecar. An entry that cannot -have the repository it needs is told so in a sentence, before anything is read, +Neither a read acquires it: not a completed one, which reaches nothing at all, +and not a new one, which goes to its provider while another entry holds the +lease. What the lease answers is that one entry at a time *changes* a +repository, and an observation is not a change — a read that took it would +refuse while somebody else was working, and make anybody else wait while it was +only looking. It is not exclusion against arbitrary Git tools, a person editing +the checkout, or any process that does not participate in this sidecar. An entry +whose new mutation cannot have the repository it needs is told so in a sentence, +before anything is changed, and that refusal is never saved as a completed result: a lease is busy only while somebody else is working, and recording "refused" would make a temporary condition the permanent answer every later continuation replays. @@ -6907,7 +6912,7 @@ Status is measured against main. | `GitQuery.resolve()` | verifies and resolves one Git revision expression contextually [Interface and evidence](specs/workflow-spec.md#7-the-git-capability--moved). | built on main | | workflow run storage | creates or compatibly finds one run by public run ID, retains its identity, state, document executions and filtered journal, and validates immutable Workspace roots through one provider-owned connection entry [Invariant](#workflow-run-storage); [Interface and evidence](specs/workflow-spec.md#9-retained-run-storage). | built on the #365 stack; the CLI lifecycle reaches it on the #366 stack | | caller-owned storage transaction | publishes several changes, including journal events, in one transaction nothing else enlists in [Invariant](#one-database-at-a-time-and-a-transaction-a-caller-can-hold). | built on main | -| REPL entry retained file work | one executing REPL entry's own recording, established by core initialization in the scope the invocation owns and ending with it: it retains the `` reads, `` writes, `` deletions and `` ensures that land inside the ambient checkout the entry was started in, authored and admitted-generated alike, because both cross the one document filesystem. A completed operation answers from its saved result without live path canonicalization, ambient discovery, a lease, a permission lookup or a filesystem call — so only *new* work needs the environment, a continuation still binds the text its read answered with while the file says something else, and a new entry reading that path sees what is there now. A record's name is the digest of the record version, the operation kind and the complete received request, so two equal requests in one entry are one name at two ordered positions and the durable engine's positional matching tells them apart; equal work in two entries is the same name, and each entry executing against its own stream segment is what keeps those apart. Recording is ordinary ancestor-max `Files` middleware, so a provider beneath that answers directly or transforms and delegates is kept as provider behavior while the received request stays the operation's identity. A shared Git directory admits one live entry at a time, held lazily for as long as the execution lasts and refused rather than queued; a completed read acquires nothing. There is no managed checkout, no `` or `` selection, no local Git operation and nothing external here; work outside the checkout stays ordinary and unrecorded, and a passive reading initializes nothing [Reference](#what-one-entry-retains-about-the-checkout-it-is-standing-in). | built on the #181 stack | +| REPL entry retained file work | one executing REPL entry's own recording, established by core initialization in the scope the invocation owns and ending with it: it retains the `` reads, `` writes, `` deletions and `` ensures that land inside the ambient checkout the entry was started in, authored and admitted-generated alike, because both cross the one document filesystem. A completed operation answers from its saved result without live path canonicalization, ambient discovery, a lease, a permission lookup or a filesystem call — so only *new* work needs the environment, a continuation still binds the text its read answered with while the file says something else, and a new entry reading that path sees what is there now. A record's name is the digest of the record version, the operation kind and the complete received request, so two equal requests in one entry are one name at two ordered positions and the durable engine's positional matching tells them apart; equal work in two entries is the same name, and each entry executing against its own stream segment is what keeps those apart. Recording is ordinary ancestor-max `Files` middleware, so a provider beneath that answers directly or transforms and delegates is kept as provider behavior while the received request stays the operation's identity. A shared Git directory admits one live *mutating* entry at a time, held lazily for as long as the execution lasts and refused rather than queued; no read acquires it, so a new read proceeds while another entry holds it. There is no managed checkout, no `` or `` selection, no local Git operation and nothing external here; work outside the checkout stays ordinary and unrecorded, and a passive reading initializes nothing [Reference](#what-one-entry-retains-about-the-checkout-it-is-standing-in). | built on the #181 stack | | live durable-operation coordinator | explicitly coordinates structured live execution with existing Yield publication while leaving replay and callback effects unchanged [Invariant](#live-effect-coordination). | built on the #365 stack | | Workspace coordination API | fails closed by default; replaceable context routes only a one-use provider selection, while the selected provider directly invokes an execution-owned credentialed capability for execution, publication and failure activation [Invariant](#live-effect-coordination). | built on the #365 stack; the Deno provider installs an adapter-private atomic handler | | shared Git-host reconciliation | Reconciles Git push and pull-request upsert through one request-only boundary whose private terminal alone settles the retained outcome. [Invariant](#external-effects-and-git-host-mutations); [Invariant](#issue-effects). | Built on #297; Push (#370) and PullRequest (#295) consume it. Issue (#296) has its own middleware boundary. | diff --git a/packages/cli/src/deno-repositories.ts b/packages/cli/src/deno-repositories.ts index 10dc8f5ca..cc003863d 100644 --- a/packages/cli/src/deno-repositories.ts +++ b/packages/cli/src/deno-repositories.ts @@ -53,6 +53,13 @@ export function denoRunRepositories( helper: HelperAssembly, root: string = DEFAULT_REPOSITORY_ROOT, ): Plugin { + // Copied here, at construction, rather than held as the caller's object. + // What an execution initializes with has to be what this host selected: a + // caller that kept a reference and changed a field afterwards would be + // configuring every later execution, including a child's, from outside the + // selection the command already made. The caller's own object is neither + // mutated nor frozen — this is a copy, and the copy is what is used. + const configured = snapshot(helper); return { // The bundled name, because this *is* the bundled Plugin: an operator // selecting `--plugin git` is naming this value, and two values under one @@ -85,7 +92,7 @@ export function denoRunRepositories( // isolated child that composed its own working directory is // discovered from that one. cwd: yield* cwd(), - helper, + helper: configured, }); yield* next(); }, @@ -95,6 +102,27 @@ export function denoRunRepositories( }; } +/** + * This deployment's helper configuration, copied by value. + * + * Scalars and the nested launcher environment both: the environment is the one + * member whose contents a caller could change after handing it over, and + * copying the object while sharing that record would copy nothing that matters. + * Absent members stay absent, because `undefined` and "not stated" are + * different answers to `helperCommand`. + */ +function snapshot(helper: HelperAssembly): HelperAssembly { + return Object.freeze({ + runtime: helper.runtime, + platform: helper.platform, + execPath: helper.execPath, + ...(helper.modulePath === undefined ? {} : { modulePath: helper.modulePath }), + ...(helper.launcherEnvironment === undefined + ? {} + : { launcherEnvironment: Object.freeze({ ...helper.launcherEnvironment }) }), + }); +} + /** * What a REPL entry's execution initializes: recording, and nothing else. * diff --git a/packages/git/src/deno/retained-repository/files.ts b/packages/git/src/deno/retained-repository/files.ts index 4db8186bd..03f6b76bd 100644 --- a/packages/git/src/deno/retained-repository/files.ts +++ b/packages/git/src/deno/retained-repository/files.ts @@ -13,6 +13,11 @@ * completed write answers from its receipt, each touching no file, however the * file has changed since. * + * A new mutation also takes the checkout's lease, and a new read does not. + * Being retained and being exclusive are different questions: what the lease + * answers is that one entry at a time changes a repository, so a read proceeds + * while another entry holds it and never holds it itself. + * * ## One route, one policy * * `` reads and writes, `` and `` all arrive as @@ -159,7 +164,13 @@ function* routed( if (checkout === undefined) { return yield* perform(); } - yield* recorder.hold(checkout.commonDirectory); + // Only a mutation takes the lease. A read is retained like the rest, but the + // lease is about one entry at a time *changing* a checkout, and an + // observation is not a change: a read that took it would refuse while + // somebody else is working, and refuse them while it is only looking. + if (operation !== "file-read") { + yield* recorder.hold(checkout.commonDirectory); + } return yield* recordedLocalOperation({ operation, request, perform }); } diff --git a/packages/git/tests/retained-repository-lifecycle.test.ts b/packages/git/tests/retained-repository-lifecycle.test.ts index d7d94c084..f84cac4ef 100644 --- a/packages/git/tests/retained-repository-lifecycle.test.ts +++ b/packages/git/tests/retained-repository-lifecycle.test.ts @@ -100,19 +100,23 @@ function* held(directory: string): Operation { } interface Counters { + reads: string[]; writes: string[]; deletes: string[]; } function counters(): Counters { - return { writes: [], deletes: [] }; + return { reads: [], writes: [], deletes: [] }; } function countingFiles(counted: Counters): FilesHandler { const host = hostFilesHandler(); return { checkFilePath: (input) => host.checkFilePath(input), - readTextFile: (input) => host.readTextFile(input), + *readTextFile(input: FilePathInput) { + counted.reads.push(input.path); + return yield* host.readTextFile(input); + }, globFiles: (input) => host.globFiles(input), temporaryDirectory: () => host.temporaryDirectory(), *writeTextFile(input: FileWriteInput) { @@ -130,6 +134,8 @@ function countingFiles(counted: Counters): FilesHandler { interface Run { readonly refusal: string | undefined; readonly events: readonly DurableEvent[]; + /** What the document rendered, for a row whose subject is a read's value. */ + readonly output: string; } /** @@ -199,16 +205,19 @@ function entry( }); } let refusal: string | undefined; + let output = ""; try { - yield* collect( - yield* executeInstalled({ path, stream }, [ - { admissions: [(retained) => admitReplRepositoryHistory(retained)] }, - ]), + output = String( + yield* collect( + yield* executeInstalled({ path, stream }, [ + { admissions: [(retained) => admitReplRepositoryHistory(retained)] }, + ]), + ), ); } catch (error) { refusal = error instanceof Error ? error.message : String(error); } - return { refusal, events: yield* stream.readAll() }; + return { refusal, events: yield* stream.readAll(), output }; }); } @@ -327,6 +336,67 @@ describe("an entry's hold on the checkout it is working in", () => { }); }); + it("E8: reads the held checkout, and takes no lease to do it", function* () { + const fixture = yield* useFixture(); + const holding = counters(); + const reading = counters(); + + yield* scoped(function* (): Operation { + const acquired = withResolvers(); + const release = withResolvers(); + yield* spawn(function* (): Operation { + yield* entry( + fixture.checkout, + ['held', QUESTION].join("\n"), + new InMemoryStream(), + holding, + function* (): Operation { + acquired.resolve(); + yield* release.operation; + }, + ); + }); + yield* acquired.operation; + + // Watched from here, after the holder has its descriptor: what this row + // is about is whether the reader opens one of its own. + const observed = yield* useObservedDescriptors(); + const stream = new InMemoryStream(); + const read = yield* entry( + fixture.checkout, + '\n{seen}', + stream, + reading, + ); + + // The read happened, against a checkout somebody else is holding for + // mutation. A lease taken here would have refused it. + expect(read.refusal).toBe(undefined); + expect(reading.reads).toEqual(["held.md"]); + expect(String(read.output)).toContain("held"); + // And it is retained, so a continuation answers it without looking. + expect(localRecords(read.events)).toBe(1); + // No descriptor at all: the lease is about one entry at a time changing a + // repository, and this one only looked. + expect(observed.events()).toEqual([]); + + release.resolve(); + }); + + // The holder is gone. A mutation now takes the lease it always did, which + // is the positive control for the observation above. + const after = counters(); + const observed = yield* useObservedDescriptors(); + const wrote = yield* entry( + fixture.checkout, + 'later', + new InMemoryStream(), + after, + ); + expect(wrote.refusal).toBe(undefined); + expect(observed.events().includes("lock")).toBe(true); + }); + it("E8: takes one handle for an entry's many mutations", function* () { const fixture = yield* useFixture(); const counted = counters(); diff --git a/scripts/bench-repl-file-dir.ts b/scripts/bench-repl-file-dir.ts index 2796d5659..ae34d4a12 100644 --- a/scripts/bench-repl-file-dir.ts +++ b/scripts/bench-repl-file-dir.ts @@ -3,8 +3,9 @@ * * Three columns, over the same authored document and the same seeded layout: * - * - **ordinary** — what `xmd run` assembles for file work: the host filesystem - * provider and this package's component declarations, and nothing else. + * - **ordinary** — what `xmd run` initializes: the host filesystem provider, + * this package's component declarations and the ordinary repository + * composition an ordinary run establishes through `Execution.init`. * - **entry** — the same, plus the executing entry's repository host. Every * mutation inside the checkout is recorded, and the timing includes the host's * assembly and its first lazy acquisitions, because a person waiting for the @@ -21,6 +22,23 @@ * * It states no budget and no pass/fail ratio. What it produces is raw samples, * medians and counters for a reviewer to read. + * + * ## What the counters are, and what they are not + * + * The document filesystem calls, the `git` invocations and the lock descriptors + * are observed at the boundaries this harness owns: middleware on `API.Files`, + * a counting repository host, and a wrapper around the one native function the + * advisory lock opens its descriptor with. Those are actual calls, not + * inferences — in particular the lease column counts descriptors this run + * opened on a sidecar, with the sidecar left exactly where it is, because a + * sidecar is persistent and its presence says nothing about who took it. + * + * `globs` counts the contextual `globFiles` calls a document made. It is *not* + * comprehensive traversal instrumentation: this harness does not observe + * `node:fs` directly, so it cannot say that nothing walked the tree by another + * route. Two columns with similar elapsed times are likewise not proof that + * neither traversed anything. What the content-size pair does say is that these + * observed counts and these times did not change when the checkout grew. */ import { main, scoped, until } from "effection"; @@ -38,12 +56,15 @@ import { collect } from "@executablemd/core"; import { executeInstalled } from "@executablemd/core/host"; import { InMemoryStream } from "@executablemd/durable-streams"; import type { DurableEvent } from "@executablemd/durable-streams"; +import { ensure, resource } from "effection"; +import { fileURLToPath } from "node:url"; import { useCompositionComponents } from "../packages/git/src/composition/installation.ts"; +import { useRunComposition } from "../packages/git/src/deno/run-composition/provider.ts"; +import type { HelperAssembly } from "../packages/git/src/deno/composition/credential-helper.ts"; import { Execution } from "@executablemd/core/api"; import { useReplRepositoryRecording } from "../packages/git/src/deno/retained-repository/provider.ts"; import { admitReplRepositoryHistory } from "../packages/git/src/retained-repository/admission.ts"; import { denoRepositoryHost } from "../packages/git/src/deno/composition/host.ts"; -import { commonDirectoryLock } from "../packages/git/src/deno/run-composition/placement.ts"; import { REPL_REPOSITORY_LOCAL, REPL_REPOSITORY_SELECTION, @@ -121,8 +142,12 @@ interface Counters { records: number; /** The UTF-8 size of the whole journal this run produced. */ journalBytes: number; - /** Whether the entry took the checkout's lease during this run. */ + /** Lock descriptors this run opened on a common-directory sidecar. */ lease: number; + /** Every durable event this run appended, engine events included. */ + appends: number; + /** Of those, the ones that are this feature's own local results. */ + localAppends: number; } function counters(): Counters { @@ -137,6 +162,8 @@ function counters(): Counters { records: 0, journalBytes: 0, lease: 0, + appends: 0, + localAppends: 0, }; } @@ -206,8 +233,11 @@ function document(chosen: Settings): string { return lines.join("\n"); } -/** One checkout, seeded identically for every sample and every mode. */ -function* seed(chosen: Settings, unrelated: number): Operation { +/** One checkout and managed root, seeded identically for every sample and mode. */ +function* seed( + chosen: Settings, + unrelated: number, +): Operation<{ readonly checkout: string; readonly managed: string }> { const root = yield* until(realpath(yield* until(mkdtemp(join(tmpdir(), "xmd-bench-"))))); const checkout = join(root, "checkout"); yield* until(mkdir(checkout, { recursive: true })); @@ -228,7 +258,11 @@ function* seed(chosen: Settings, unrelated: number): Operation { yield* writeTextFile(join(elsewhere, `file-${index}.md`), payload); } } - return checkout; + // Where an ordinary run would keep a managed repository, inside this + // invocation's own fixture: nothing here selects one, and a root under the + // real home would be shared state this measurement has no business touching. + const managed = join(root, "repositories"); + return { checkout, managed }; } function run(command: string, args: readonly string[], cwd: string): Operation { @@ -241,6 +275,60 @@ function run(command: string, args: readonly string[], cwd: string): Operation void }> { + return resource<{ stop: () => void }>(function* (provide) { + const runtime: unknown = Reflect.get(globalThis, "Deno"); + const original: unknown = Reflect.get(Object(runtime), "openSync"); + if (typeof original !== "function") { + throw new Error("this host opens no files, so there is no descriptor to observe"); + } + let watching = true; + yield* ensure(() => { + Reflect.set(Object(runtime), "openSync", original); + }); + Reflect.set(Object(runtime), "openSync", (...args: unknown[]) => { + const [path] = args; + if (watching && typeof path === "string" && path.endsWith(".lock")) { + counted.lease += 1; + } + return Reflect.apply(original as (...rest: unknown[]) => unknown, runtime, args); + }); + yield* provide({ + stop: () => { + watching = false; + }, + }); + }); +} + +/** + * The helper configuration an ordinary run is initialized with. + * + * The same shape the Deno entrypoint states. Nothing here reads a credential: + * what it describes is how a launcher would start this runtime, and no launcher + * is written unless an operation needs one. + */ +const HELPER: HelperAssembly = { + runtime: "source", + platform: process.platform === "win32" ? "windows" : "unix", + execPath: process.execPath, + modulePath: fileURLToPath( + new URL("../packages/cli/src/credential-helper-entry.ts", import.meta.url), + ), + launcherEnvironment: {}, +}; + /** One measured run: how long it took, and what it really did. */ interface Measured { readonly ms: number; @@ -252,11 +340,15 @@ interface Measured { function measure( mode: "ordinary" | "entry" | "continuation", checkout: string, + managed: string, source: string, retained: readonly DurableEvent[] = [], ): Operation { return scoped(function* (): Operation { const counted = counters(); + // Every lock descriptor this run opens, observed at the one native function + // `useAdvisoryLock` reaches for. The sidecar itself is left alone. + const descriptors = yield* useObservedLocks(counted); const stream = new InMemoryStream([...retained]); const path = join(checkout, "entry.md"); yield* writeTextFile(path, source); @@ -284,13 +376,24 @@ function measure( ); yield* useCompositionComponents(); - const lock = commonDirectoryLock(yield* until(realpath(join(checkout, ".git")))); - const before = yield* exists(lock); - // Timed from here, so the entry setup and its first lazy acquisitions are // inside the measurement rather than beside it. const started = performance.now(); if (mode === "ordinary") { + // What an ordinary run establishes, through the same seam and inside the + // same timed region: the comparison is between two initialized + // executions, and a baseline that initialized nothing would be charging + // the entry column for work the ordinary one also does. + yield* Execution.around({ + *init(_args, next): Operation { + yield* useRunComposition({ + root: managed, + cwd: checkout, + helper: HELPER, + }); + yield* next(); + }, + }); yield* collect(yield* executeInstalled({ path, stream }, [])); } else { // Registered the way the REPL host registers it: core initialization is @@ -310,10 +413,14 @@ function measure( } const ms = performance.now() - started; - if (!before && (yield* exists(lock))) { - counted.lease = 1; - } const events = yield* stream.readAll(); + counted.appends = events.length - retained.length; + counted.localAppends = events + .slice(retained.length) + .filter( + (event) => event.type === "yield" && event.description.type === REPL_REPOSITORY_LOCAL, + ).length; + descriptors.stop(); counted.journalBytes = new TextEncoder().encode(JSON.stringify(events)).length; counted.records = events.filter( (event) => @@ -352,8 +459,9 @@ function report(name: string, samples: readonly Measured[]): void { `globs ${counted.globs}`, ); console.log( - `${"".padEnd(13)} git ${counted.commands} lease ${counted.lease} ` + - `records ${counted.records} journal ${counted.journalBytes} bytes`, + `${"".padEnd(13)} git ${counted.commands} lock descriptors ${counted.lease} ` + + `appends ${counted.appends} (local results ${counted.localAppends}) ` + + `retained records ${counted.records} journal ${counted.journalBytes} bytes`, ); } @@ -384,8 +492,8 @@ await main(function* (argv): Operation { // One unreported warmup per live mode, so neither column is charged for the // module graph the other one already paid for. for (const mode of ["ordinary", "entry"] as const) { - const checkout = yield* seed(chosen, unrelated); - yield* measure(mode, checkout, source); + const { checkout, managed } = yield* seed(chosen, unrelated); + yield* measure(mode, checkout, managed, source); yield* rm(join(checkout, ".."), { recursive: true, force: true }); } @@ -399,18 +507,17 @@ await main(function* (argv): Operation { const order = sample % 2 === 0 ? (["ordinary", "entry"] as const) : (["entry", "ordinary"] as const); for (const mode of order) { - const checkout = yield* seed(chosen, unrelated); - const measured = yield* measure(mode, checkout, source); + const { checkout, managed } = yield* seed(chosen, unrelated); + const measured = yield* measure(mode, checkout, managed, source); (mode === "ordinary" ? ordinary : entry).push(measured); if (mode === "entry") { // The continuation runs against the journal that entry just wrote, in - // the checkout it wrote it in, with the lease sidecar removed so a - // second acquisition would be visible. Nothing holds it: the live run - // has ended. - const lock = commonDirectoryLock(yield* until(realpath(join(checkout, ".git")))); - yield* rm(lock, { force: true }); + // the checkout it wrote it in. The sidecar stays exactly where the + // live run left it: whether this run takes a lease is observed at the + // descriptor, so there is nothing to clear to make an acquisition + // visible. continuation.push( - yield* measure("continuation", checkout, source, unfinished(measured.events)), + yield* measure("continuation", checkout, managed, source, unfinished(measured.events)), ); } yield* rm(join(checkout, ".."), { recursive: true, force: true }); @@ -425,8 +532,11 @@ await main(function* (argv): Operation { } console.log(""); console.log( - "A continuation performs no read, no mutation, no traversal, runs no `git` and takes no " + - "lease, which is what the zeroes above say. This states no budget and no ratio: it is raw " + - "evidence for a reviewer.", + "A continuation performs no observed document read, write, delete or ensure, runs no `git`, " + + "opens no lock descriptor and appends no local result; the appends it does make are the " + + "engine's own closing events, which is why that column is counted apart from local " + + "results. The `checks` it does make are pure lexical path checks, which touch no file. " + + "This states no budget and no ratio, and the `globs` column is contextual calls rather " + + "than comprehensive traversal instrumentation: it is raw evidence for a reviewer.", ); }); diff --git a/specs/repl-spec.md b/specs/repl-spec.md index be67fd7e1..d2c1b5e1f 100644 --- a/specs/repl-spec.md +++ b/specs/repl-spec.md @@ -799,23 +799,29 @@ is already done. Type a *new* entry that reads the same path and it reads what is there now. What is retained is each operation, not a value the session remembers for a path. -A mutation anywhere else — beside the document, in a temporary directory, in any -directory that is not inside the checkout the entry started in — is ordinary -file work, is retained by nothing here, and happens again if the entry is -continued. Reads, searches and temporary directories are ordinary throughout. -The lexical path rules and the permission a `` already answers to are -unchanged, and they are not checkout validation: a path this run may not write -is refused the same way it is anywhere else. - -One entry at a time works in one repository. A second entry — in this process or -another — that needs the same checkout is told so immediately rather than -waiting: +Any of the four anywhere else — beside the document, in a temporary directory, +in any directory that is not inside the checkout the entry started in — is +ordinary file work, is retained by nothing here, and happens again if the entry +is continued. That includes a read: what is retained is a read *inside* the +checkout. Searches and temporary directories are ordinary throughout, wherever +they point. The lexical path rules and the permission a `` already answers +to are unchanged, and they are not checkout validation: a path this run may not +read or write is refused the same way it is anywhere else. + +One entry at a time *changes* one repository. A second entry — in this process +or another — whose next new write, deletion or directory is in a checkout +somebody else is changing is told so immediately rather than waiting: ```text Another XMD entry or process holds this checkout. End that work before trying again. Nothing was read and nothing was changed. ``` +Reading is not changing, so it is not excluded: an entry may read a file in a +checkout another entry is holding, and reading takes nothing that would stop +that entry carrying on. Only a new write, deletion or directory waits on the +hold, and a completed operation of any kind takes nothing at all. + Starting outside a checkout retains none of this and changes nothing else: the entry performs ordinary file work, exactly as it would anywhere else, and never looks for a repository twice. From ecd08632f52b94dd6bb433c1a942197f03147acd Mon Sep 17 00:00:00 2001 From: Taras Mankovski Date: Fri, 16 Oct 2026 03:55:34 -0400 Subject: [PATCH 3/4] =?UTF-8?q?=F0=9F=93=9D=20Say=20what=20the=20host=20se?= =?UTF-8?q?lects,=20not=20what=20it=20used=20to=20pass=20(#181,=201/N)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four statements this change had left untrue, found while preparing the publication of #902 and fixed rather than deferred. Section 9.6 described a repository installer as a `runXmd` parameter, with the argument in the sample and a paragraph about what Node and Bun pass instead. Under the accepted C1 composition there is no such parameter: the bundled Plugin is one CLI-private Context whose default is the provider-neutral value, a runtime-named entrypoint binds a configured native variant before any command is dispatched, and the enumeration of how the commands differ now says what each one initializes — a test root capturing the selected value for its `host="run"` child, and `xmd repl` initializing none of it because an entry's recording is registered by the selected REPL host. `architecture.md` listed that same parameter beside the service installer and the upgrade assembly in the sentence about where standard input comes from. Two runtime exclusions described the withdrawn design: one named a record owner this feature no longer has, and one named an entry-host factory the entrypoint no longer supplies. `repl-profile.ts` goes back to the base. Its interface renaming and its third paragraph were for an input the amendment withdrew, so the file carried prose about a member it does not declare. No behavior changes. Prose, one revert, and the enumerations they belong to. --- architecture.md | 2 +- packages/cli/src/repl-profile.ts | 18 +++------ scripts/runtime-test-exclusions.ts | 4 +- specs/executable-mdx-spec.md | 60 ++++++++++++++++++------------ specs/repl-spec.md | 6 +-- 5 files changed, 48 insertions(+), 42 deletions(-) diff --git a/architecture.md b/architecture.md index a2b20c0dd..c3252021b 100644 --- a/architecture.md +++ b/architecture.md @@ -174,7 +174,7 @@ document argument of exactly `-` means standard input instead. Standard input is the one of the three the shared CLI cannot reach for itself. A process's own stdin is host-specific state, so each runtime-named entrypoint supplies the operation that reads it, as a value passed into the CLI beside the -service installer, the upgrade assembly and the repository installer. It is not +service installer and the upgrade assembly. It is not a Context, a contextual Api, a public export, a syntax entry, a component or a document capability: nothing an authored document can name reaches it, replaces it, or asks it for a second read. The CLI calls it at most once per invocation, diff --git a/packages/cli/src/repl-profile.ts b/packages/cli/src/repl-profile.ts index 7ab84b20e..a4156294f 100644 --- a/packages/cli/src/repl-profile.ts +++ b/packages/cli/src/repl-profile.ts @@ -38,9 +38,8 @@ import type { CommandPlugins } from "./plugin-host.ts"; const REPL_INCLUDES: readonly string[] = Object.freeze(["components", "."]); /** - * What the caller states and this assembler cannot work out for itself. + * The two facts a proof states and production leaves to the host. * - * The first two are facts a proof states and production leaves to the host. * `planAgentContext` and `installAgentProviderStack` already take the first: * production states none and both get the real ACPX runtime, while a journey * states a scriptable one so it can drive this exact assembler and this exact @@ -48,13 +47,8 @@ const REPL_INCLUDES: readonly string[] = Object.freeze(["components", "."]); * keeps its conversations, which production leaves at the host default and a * proof points at a directory it created itself — a suite that used the default * would read and remove directories under the developer's own home. - * - * The third is the opposite way round: production states it and a proof usually - * does not. Only a runtime whose module graph can reach the Deno adapter has an - * entry host to offer, so the entrypoint that knows supplies it and the shared - * command path carries it as data. */ -export interface ReplProfileInputs { +export interface ReplProfileSeams { readonly acp?: AcpxProviderDependencies; readonly planWriterRoot?: string; } @@ -101,13 +95,13 @@ export interface ReplExecutionProfile { export function* assembleReplProfile( stack: AgentStack, plugins: CommandPlugins, - inputs: ReplProfileInputs = {}, + seams: ReplProfileSeams = {}, ): Operation { - yield* installAgentProviderStack(stack, inputs.acp); + yield* installAgentProviderStack(stack, seams.acp); const plan = yield* planComponentDeclaration({ surface: "component", - ...(inputs.planWriterRoot === undefined ? {} : { planWriterRoot: inputs.planWriterRoot }), + ...(seams.planWriterRoot === undefined ? {} : { planWriterRoot: seams.planWriterRoot }), includes: REPL_INCLUDES, // The command's own Plugin assembly, not a second installation of it: two // would be two catalogs claiming the same names. @@ -115,7 +109,7 @@ export function* assembleReplProfile( // The same seam `planAgentContext` already takes: production states none and // gets the real ACPX runtime, and a proof states a scriptable one so a // journey can drive this exact assembler rather than a copy of it. - context: planAgentContext(stack, inputs.acp), + context: planAgentContext(stack, seams.acp), ...(stack.sessions === undefined ? {} : { sessions: stack.sessions }), host: yield* useScope(), // Plan review is answered by the REPL's own Elicit provider, which the diff --git a/scripts/runtime-test-exclusions.ts b/scripts/runtime-test-exclusions.ts index 3c99c175c..977a31721 100644 --- a/scripts/runtime-test-exclusions.ts +++ b/scripts/runtime-test-exclusions.ts @@ -679,7 +679,7 @@ const DENO_ONLY_REPOSITORY_PROVIDER: RuntimeExclusion[] = [ { path: "packages/git/tests/retained-repository-local.test.ts", reason: - "the subject is a REPL entry's native repository host: it discovers a real checkout with a `git` subprocess and records what a real filesystem answered. The record format, the owner and the admission are provider-neutral and are proven on every runtime by packages/git/tests/retained-repository-records.test.ts", + "the subject is a REPL entry's native repository host: it discovers a real checkout with a `git` subprocess and records what a real filesystem answered. The record format and the admission are provider-neutral and are proven on every runtime by packages/git/tests/retained-repository-records.test.ts", issue: DERIVED_SCOPE, }, { @@ -691,7 +691,7 @@ const DENO_ONLY_REPOSITORY_PROVIDER: RuntimeExclusion[] = [ { path: "packages/cli/tests/repl-repositories.test.ts", reason: - "drives the production entry-host factory, which only Deno and the compiled binary supply: its module graph reaches @executablemd/git/deno and so Workflow's storage adapter and node:sqlite. That Node and Bun start a REPL and perform ordinary File and Dir work without it is proven by packages/cli/tests/repl-repositories-pty.test.ts, which runs everywhere", + "drives the production entry host, which only Deno and the compiled binary register: its module graph reaches @executablemd/git/deno and so Workflow's storage adapter and node:sqlite. That Node and Bun start a REPL and perform ordinary File and Dir work without it is proven by packages/cli/tests/repl-repositories-pty.test.ts, which runs everywhere", issue: DERIVED_SCOPE, }, ]; diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 01d95a131..84f243feb 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -10887,9 +10887,9 @@ export function* useTerminalOutput(): Operation { how an eval block compiles, how a service process is hosted, or which runtime it is on**. A runtime-named entrypoint — `deno.ts`, `node.ts`, `bun.ts`, `compiled.ts` — installs its -`API.Env` providers with `{ at: "min" }` and passes the matching service -installer, its own upgrade assembly and its own repository installer to -`runXmd(args, installService, upgrade, installRepositories, ...)`: +`API.Env` providers with `{ at: "min" }`, selects the bundled Plugin this host +runs with, and passes the matching service installer and its own upgrade +assembly to `runXmd(args, installService, upgrade, ...)`: ```typescript yield* API.Env.around( @@ -10904,11 +10904,11 @@ yield* API.Env.around( }, { at: "min" }, ); +yield* BundledRunPlugin.set(denoRunRepositories(HELPER)); yield* runXmd( args, useDenoService, UPGRADE, - denoRunRepositories(HELPER), () => useDenoWorkflowHost(HELPER), useMachineSessions(), ); @@ -10922,26 +10922,38 @@ document nothing through which a release could be read, downloaded or replaced, and inheriting those components by omission is impossible. The complete contract is [`xmd upgrade`](./upgrade-command-spec.md). -The repository installer is required for the same reason and states a different -fact: what this runtime can *operate*, rather than what it is. Deno and the -compiled binary pass the live provider — `denoRunRepositories(HELPER)`, built -from the same credential-helper assembly the workflow host uses — and Node and -Bun pass `unsupportedRepositories`. Every runtime registers the same twelve -declarations either way, so `xmd syntax` describes one language everywhere and a -runtime that operates none of it reports an absent provider before a lock, a -credential, a subprocess or a request exists (§5.3, §8.1). `` is canonical -core's rather than one of those declarations, so it is operational on every -runtime that installs a Files provider. - -Being *supplied* by the entrypoint and being *used* are different things, and -the commands differ: - -- an ordinary `xmd run` installs a fresh provider instance for that execution; -- `xmd test` installs none for its own root document, and hands the - entrypoint's installer to a `host="run"` child — which is an ordinary run - whatever command hosts it, and receives an instance of its own; -- a workflow execution installs none at all, because its Workspace attachment - owns repositories for the run. +The bundled Plugin is a selection rather than a parameter, and it states a +different fact: what this runtime can *operate*, rather than what it is. One +CLI-private Context holds it and its default is the provider-neutral Git +Plugin; Deno and the compiled binary bind a configured native variant — +`denoRunRepositories(HELPER)`, built from the same credential-helper assembly +the workflow host uses — before any command is dispatched, and Node and Bun +leave the default in place. The shared command path reads the value and imports +no adapter. What the Context holds is inert code and configuration, with no +provider, lease, session or identity in it: the native variant delegates the +neutral Plugin's whole install and adds ordinary initialization middleware for a +run, so every runtime registers the same twelve declarations either way, `xmd +syntax` describes one language everywhere, and a runtime that operates none of +it reports an absent provider before a lock, a credential, a subprocess or a +request exists (§5.3, §8.1). `` is canonical core's rather than one of +those declarations, so it is operational on every runtime that installs a Files +provider. + +Being *selected* by the entrypoint and being *initialized* are different things, +and the commands differ: + +- an ordinary `xmd run` initializes a fresh provider instance for that + execution; +- `xmd test` activates no bundled Plugin for its own root document, and captures + the selected value into the settings a `host="run"` child builds its host + from — an ordinary run whatever command hosts it, which installs that value in + its own scope and receives an instance of its own; +- a workflow execution initializes none at all, because its Workspace + attachment owns repositories for the run; +- `xmd repl` initializes none either. What an entry does to files in the ambient + checkout it was started in is retained by the entry's own recording, which the + selected REPL host registers rather than this Plugin + ([the REPL](./repl-spec.md)). The service installer is invoked only for `xmd run` and `xmd test`, immediately before `execute()`. Help, inspection and agent-worker paths never install or diff --git a/specs/repl-spec.md b/specs/repl-spec.md index d2c1b5e1f..869dff948 100644 --- a/specs/repl-spec.md +++ b/specs/repl-spec.md @@ -874,14 +874,14 @@ recorded output replaces it. Repository behavior belongs to the entry that is executing, and to nothing wider. The command selects it; the entry's own task prepares it, once, inside that task, and everything it takes — the Git session it reads through, where it -looked for a checkout, the claim on that repository — is released when the entry +looked for a checkout, the hold on that repository — is released when the entry is torn down. Each entry reads and writes its own range of the one history, so the same text typed twice is two sets of records rather than one replayed, and the entry after an interrupted one replays nothing of it however identical its text. No second name for the entry appears in a record: which history an entry may continue is the range it was given. Reading is not owning: a settled history, a frozen position and a refused location prepare nothing, look for no -checkout and claim nothing. A host that registers no recording runs the same +checkout and hold nothing. A host that registers no recording runs the same file work with nothing retained, which is what Node and Bun do. Components receive immutable view data and answer with a typed semantic action. @@ -909,7 +909,7 @@ schedules no timer. ## Exclusions - Two processes writing one execution is unsupported. There is no lease protocol; - do not open the same location twice for writing. The claim on a checkout is a + do not open the same location twice for writing. The hold on a checkout is a different thing: it says one entry at a time may mutate one repository, and it says nothing about who may write this execution's history. - Repository components are not part of this. ``, ``, the From e6420b45570161cf88201a419d8c41a38d60d1a9 Mon Sep 17 00:00:00 2001 From: Taras Mankovski Date: Fri, 16 Oct 2026 04:14:05 -0400 Subject: [PATCH 4/4] =?UTF-8?q?=F0=9F=94=A8=20Run=20each=20new=20proof=20o?= =?UTF-8?q?nly=20where=20its=20runtime=20can=20host=20it=20(#181,=201/N)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two CI failures, each one a gate the local Deno matrix does not have. **The Node typecheck.** `pnpm exec tsc --project tsconfig.node.json` is the only place `lib: ES2022` is enforced, and it reported five errors: the lifetime proof names `Deno.ChildProcess` and `Deno.Command`, and the local proof's concurrent row called `toSorted`, which is ES2023 and compiles under Deno alone. The lifetime proof joins the three tests already excluded from that project for reaching `Deno` directly — its subject is a kernel-released advisory lock and a second process taking the same sidecar — and the sort is copied before sorting, the way `module-partition.test.ts` already does it. **Bun.** The C1 fixture was a row inside the shared `testing-execution-host.test.ts`, which runs everywhere, and it imports the configured native Plugin — `@executablemd/git/deno`, whose graph reaches `node:sqlite`. Bun has no such module, so locally the whole file fails to load and on CI the child's repository work failed instead. Reproduced both ways before moving anything. The proof is now its own file, excluded for Node and Bun beside the other Deno-only repository subjects, and the portable half — that a test root hands its `host="run"` child the value the host selected, reinstalled in the child's own scope after its working directory — stays where it was and still runs on all three runtimes. No production code changes. Verified: the Node typecheck gate exit 0; the trimmed shared suite 21 passed under Bun, 21 under Node and 21 under Deno; the moved proof 1 passed under Deno; runtime exclusions and the manifest rows pass; lint and check exit 0. --- .../testing-execution-host-native.test.ts | 216 ++++++++++++++++++ .../cli/tests/testing-execution-host.test.ts | 196 +--------------- .../tests/retained-repository-local.test.ts | 5 +- scripts/runtime-test-exclusions.ts | 6 + tsconfig.node.json | 7 +- 5 files changed, 232 insertions(+), 198 deletions(-) create mode 100644 packages/cli/tests/testing-execution-host-native.test.ts diff --git a/packages/cli/tests/testing-execution-host-native.test.ts b/packages/cli/tests/testing-execution-host-native.test.ts new file mode 100644 index 000000000..b36626352 --- /dev/null +++ b/packages/cli/tests/testing-execution-host-native.test.ts @@ -0,0 +1,216 @@ +/** + * Why this proof is its own file. + * + * It imports the configured native Plugin, which comes from + * `@executablemd/git/deno` — a graph that reaches `node:sqlite`. A runtime + * without that module cannot load this module at all, and one that could load + * it still has no Deno adapter to initialize. The runtime exclusions keep this + * file out of the Node and Bun suites, which is why it is not one more row in + * `testing-execution-host.test.ts`: every row there is host-neutral and runs + * everywhere. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { Ok, until } from "effection"; +import type { Operation } from "effection"; +import { realpath } from "node:fs/promises"; +import { join } from "node:path"; +import { spawnSync } from "node:child_process"; +import process from "node:process"; +import { fileURLToPath } from "node:url"; +import { Execution } from "@executablemd/core/api"; +import type { Plugin, PluginInstallation } from "@executablemd/core/api"; +import { cwd } from "@executablemd/runtime"; +import { cliBase } from "@executablemd/test-support/launch"; +import { useTempDirectory } from "@executablemd/test-support/temp"; +import type { HelperAssembly } from "@executablemd/git/credential-helper"; +import { planComponentDeclaration } from "../src/plan-component.ts"; +import { testingExecutionHost } from "../src/testing-host.ts"; +import type { TestingHostSettings } from "../src/testing-host.ts"; +import { BundledRunPlugin } from "../src/run-profile.ts"; +import { denoRunRepositories } from "../src/deno-repositories.ts"; + +/** + * C1 — the native host an isolated run child actually initializes. + * + * A test root is not a run profile: it activates no bundled Plugin and has no + * operational repository provider of its own. Its `` + * child *is* an ordinary run, so it has to receive what this host selected — + * and what it receives is the inert Plugin value the command captured before + * any document code ran, never a live provider and never whatever a document + * put in the Context afterwards. + * + * This is asked of a configured native variant rather than the provider-neutral + * default, because the defect worth catching is a child initialized with the + * wrong host's Plugin: one hard-coded to the statically bundled value would + * satisfy every assertion about names while reaching a managed root nobody + * configured. + */ +describe("C1 — the native host a run child initializes", () => { + /** A bare remote a child may clone without reaching a network. */ + function* useBareRemote(home: string): Operation { + const parent = yield* until(realpath(yield* useTempDirectory("xmd-c1-remote-"))); + const seed = join(parent, "seed"); + git(["init", "--initial-branch=main", seed], parent, home); + git(["commit", "--allow-empty", "-m", "first"], seed, home); + const bare = join(parent, "remote.git"); + git(["clone", "--bare", "--", seed, bare], parent, home); + return bare; + } + + /** The configured native Plugin, with each execution it initializes recorded. */ + function observedNative(initialized: string[], root: string): Plugin { + const native = denoRunRepositories(HELPER, root); + return { + name: native.name, + *install(request): Operation { + const install = native.install; + const declared = install === undefined ? undefined : yield* install(request); + if (declared === undefined) { + return undefined; + } + yield* Execution.around({ + *init(_args, next): Operation { + initialized.push(`${request.command}@${yield* cwd()}`); + yield* next(); + }, + }); + return declared; + }, + }; + } + + it("E9: gives a test root's run child the captured configured Plugin, once", function* () { + const home = yield* useTempDirectory("xmd-c1-home-"); + const remote = yield* useBareRemote(home); + // This fixture's own managed root, so where a clone lands is observable and + // nothing reaches the tree a production run owns. + const managed = yield* until(realpath(yield* useTempDirectory("xmd-c1-managed-"))); + const root = join(managed, "repositories"); + // Where the child stands: a directory the parent chose, which is what a + // `` around the `` selects. + const workspace = yield* until(realpath(yield* useTempDirectory("xmd-c1-work-"))); + + const initialized: string[] = []; + const settings = hostSettings({ bundledPlugin: observedNative(initialized, root) }); + const host = testingExecutionHost(settings); + + // A document could replace the selection after the command captured it. + // What it replaces it with here would be visible if the child read it: this + // one declares for every document command and records a different word. + const replaced: string[] = []; + yield* BundledRunPlugin.set({ + name: "@executablemd/git", + *install(): Operation { + yield* Execution.around({ + *init(_args, next): Operation { + replaced.push("replaced"); + yield* next(); + }, + }); + return {}; + }, + }); + + let rendered = ""; + const settlement = yield* host.runChild({ + request: { + host: "run", + source: + `\n` + + "selected {repository}\n", + props: {}, + journal: "transient", + collectJournal: false, + configuration: [], + }, + run: undefined, + cwd: workspace, + *chunk(text: string): Operation { + rendered += text; + }, + }); + rendered += settlement.output; + + // The child ran, and the repository work an ordinary run can do worked. + expect(settlement.outcome.kind).toBe("settled"); + expect(settlement.outcome.kind === "settled" ? settlement.outcome.result.ok : false).toBe(true); + // Exactly one initialization, for command `run`, standing where the parent + // put the child rather than where this process is. + expect(initialized).toEqual([`run@${workspace}`]); + // And it was the captured value that did it: the Context the document + // replaced afterwards never reached the child. + expect(replaced).toEqual([]); + // The clone landed under the root this fixture configured, which is what + // proves the *configured* Plugin arrived rather than a default one. + expect(rendered).toContain(root); + expect(rendered).toContain("selected "); + }); +}); + +/** + * What this suite's host is, stated the way the Deno entrypoint states it. + * + * The same shape `deno.ts` builds, because what the configured Plugin holds is + * host configuration: a fixture that invented a different one would be + * measuring its own invention. + */ +const HELPER: HelperAssembly = { + runtime: "source", + platform: process.platform === "win32" ? "windows" : "unix", + execPath: process.execPath, + modulePath: fileURLToPath(new URL("../src/credential-helper-entry.ts", import.meta.url)), + launcherEnvironment: {}, +}; + +/** Git, run with configuration nobody's machine contributes to. */ +function git(args: readonly string[], directory: string, home: string): string { + const outcome = spawnSync("git", [...args], { + cwd: directory, + env: { + ...(process.env.PATH === undefined ? {} : { PATH: process.env.PATH }), + HOME: home, + GIT_CONFIG_GLOBAL: "/dev/null", + GIT_CONFIG_SYSTEM: "/dev/null", + GIT_CONFIG_NOSYSTEM: "1", + GIT_TERMINAL_PROMPT: "0", + LC_ALL: "C", + GIT_AUTHOR_NAME: "Fixture", + GIT_AUTHOR_EMAIL: "fixture@example.invalid", + GIT_COMMITTER_NAME: "Fixture", + GIT_COMMITTER_EMAIL: "fixture@example.invalid", + }, + encoding: "utf8", + }); + if (outcome.status !== 0) { + throw new Error(`git ${args.join(" ")} failed: ${outcome.stderr}`); + } + return outcome.stdout; +} + +/** The host settings a command assembles, with one member under test. */ +function hostSettings(chosen: { readonly bundledPlugin: Plugin }): TestingHostSettings { + return { + includes: [], + plugins: [], + pluginArgs: [], + secretDetection: false, + // deno-lint-ignore require-yield + installService: function* (): Operation {}, + bundledPlugin: chosen.bundledPlugin, + testAgentWorker: Ok([...cliBase(), "test-agent"]), + planDeclaration: (request) => + planComponentDeclaration({ + surface: "component", + includes: [], + context: request.context, + ...(request.planWriterRoot === undefined ? {} : { planWriterRoot: request.planWriterRoot }), + host: request.host, + ...(request.observePlanWriter === undefined + ? {} + : { observePlanWriter: request.observePlanWriter }), + installElicitation: request.installElicitation, + }), + }; +} diff --git a/packages/cli/tests/testing-execution-host.test.ts b/packages/cli/tests/testing-execution-host.test.ts index 81b5896df..f9d8175a5 100644 --- a/packages/cli/tests/testing-execution-host.test.ts +++ b/packages/cli/tests/testing-execution-host.test.ts @@ -28,17 +28,7 @@ import { PLAN_WRITER_INSTRUCTIONS, DEFAULT_PLAN_WRITER_ROOT } from "../src/plan- import type { PlanWriterObservation } from "../src/plan-writer-profile.ts"; import { testingExecutionHost } from "../src/testing-host.ts"; import { planComponentDeclaration, planComponentDescription } from "../src/plan-component.ts"; -import { BUNDLED_PLUGIN, BundledRunPlugin } from "../src/run-profile.ts"; -import { denoRunRepositories } from "../src/deno-repositories.ts"; -import { Execution } from "@executablemd/core/api"; -import type { Plugin, PluginInstallation } from "@executablemd/core/api"; -import { cwd } from "@executablemd/runtime"; -import { useTempDirectory } from "@executablemd/test-support/temp"; -import { spawnSync } from "node:child_process"; -import process from "node:process"; -import { fileURLToPath } from "node:url"; -import type { HelperAssembly } from "@executablemd/git/credential-helper"; -import type { TestingHostSettings } from "../src/testing-host.ts"; +import { BUNDLED_PLUGIN } from "../src/run-profile.ts"; import reviewPlugin from "@executablemd/code-review-agent"; function doc(...lines: string[]): string { @@ -1157,187 +1147,3 @@ describe("execution initialization in isolated run children", () => { expect(result.stdout + result.stderr).not.toContain("❌"); }); }); - -/** - * C1 — the native host an isolated run child actually initializes. - * - * A test root is not a run profile: it activates no bundled Plugin and has no - * operational repository provider of its own. Its `` - * child *is* an ordinary run, so it has to receive what this host selected — - * and what it receives is the inert Plugin value the command captured before - * any document code ran, never a live provider and never whatever a document - * put in the Context afterwards. - * - * This is asked of a configured native variant rather than the provider-neutral - * default, because the defect worth catching is a child initialized with the - * wrong host's Plugin: one hard-coded to the statically bundled value would - * satisfy every assertion about names while reaching a managed root nobody - * configured. - */ -describe("C1 — the native host a run child initializes", () => { - /** A bare remote a child may clone without reaching a network. */ - function* useBareRemote(home: string): Operation { - const parent = yield* until(realpath(yield* useTempDirectory("xmd-c1-remote-"))); - const seed = join(parent, "seed"); - git(["init", "--initial-branch=main", seed], parent, home); - git(["commit", "--allow-empty", "-m", "first"], seed, home); - const bare = join(parent, "remote.git"); - git(["clone", "--bare", "--", seed, bare], parent, home); - return bare; - } - - /** The configured native Plugin, with each execution it initializes recorded. */ - function observedNative(initialized: string[], root: string): Plugin { - const native = denoRunRepositories(HELPER, root); - return { - name: native.name, - *install(request): Operation { - const install = native.install; - const declared = install === undefined ? undefined : yield* install(request); - if (declared === undefined) { - return undefined; - } - yield* Execution.around({ - *init(_args, next): Operation { - initialized.push(`${request.command}@${yield* cwd()}`); - yield* next(); - }, - }); - return declared; - }, - }; - } - - it("E9: gives a test root's run child the captured configured Plugin, once", function* () { - const home = yield* useTempDirectory("xmd-c1-home-"); - const remote = yield* useBareRemote(home); - // This fixture's own managed root, so where a clone lands is observable and - // nothing reaches the tree a production run owns. - const managed = yield* until(realpath(yield* useTempDirectory("xmd-c1-managed-"))); - const root = join(managed, "repositories"); - // Where the child stands: a directory the parent chose, which is what a - // `` around the `` selects. - const workspace = yield* until(realpath(yield* useTempDirectory("xmd-c1-work-"))); - - const initialized: string[] = []; - const settings = hostSettings({ bundledPlugin: observedNative(initialized, root) }); - const host = testingExecutionHost(settings); - - // A document could replace the selection after the command captured it. - // What it replaces it with here would be visible if the child read it: this - // one declares for every document command and records a different word. - const replaced: string[] = []; - yield* BundledRunPlugin.set({ - name: "@executablemd/git", - *install(): Operation { - yield* Execution.around({ - *init(_args, next): Operation { - replaced.push("replaced"); - yield* next(); - }, - }); - return {}; - }, - }); - - let rendered = ""; - const settlement = yield* host.runChild({ - request: { - host: "run", - source: - `\n` + - "selected {repository}\n", - props: {}, - journal: "transient", - collectJournal: false, - configuration: [], - }, - run: undefined, - cwd: workspace, - *chunk(text: string): Operation { - rendered += text; - }, - }); - rendered += settlement.output; - - // The child ran, and the repository work an ordinary run can do worked. - expect(settlement.outcome.kind).toBe("settled"); - expect(settlement.outcome.kind === "settled" ? settlement.outcome.result.ok : false).toBe(true); - // Exactly one initialization, for command `run`, standing where the parent - // put the child rather than where this process is. - expect(initialized).toEqual([`run@${workspace}`]); - // And it was the captured value that did it: the Context the document - // replaced afterwards never reached the child. - expect(replaced).toEqual([]); - // The clone landed under the root this fixture configured, which is what - // proves the *configured* Plugin arrived rather than a default one. - expect(rendered).toContain(root); - expect(rendered).toContain("selected "); - }); -}); - -/** - * What this suite's host is, stated the way the Deno entrypoint states it. - * - * The same shape `deno.ts` builds, because what the configured Plugin holds is - * host configuration: a fixture that invented a different one would be - * measuring its own invention. - */ -const HELPER: HelperAssembly = { - runtime: "source", - platform: process.platform === "win32" ? "windows" : "unix", - execPath: process.execPath, - modulePath: fileURLToPath(new URL("../src/credential-helper-entry.ts", import.meta.url)), - launcherEnvironment: {}, -}; - -/** Git, run with configuration nobody's machine contributes to. */ -function git(args: readonly string[], cwd: string, home: string): string { - const outcome = spawnSync("git", [...args], { - cwd, - env: { - ...(process.env.PATH === undefined ? {} : { PATH: process.env.PATH }), - HOME: home, - GIT_CONFIG_GLOBAL: "/dev/null", - GIT_CONFIG_SYSTEM: "/dev/null", - GIT_CONFIG_NOSYSTEM: "1", - GIT_TERMINAL_PROMPT: "0", - LC_ALL: "C", - GIT_AUTHOR_NAME: "Fixture", - GIT_AUTHOR_EMAIL: "fixture@example.invalid", - GIT_COMMITTER_NAME: "Fixture", - GIT_COMMITTER_EMAIL: "fixture@example.invalid", - }, - encoding: "utf8", - }); - if (outcome.status !== 0) { - throw new Error(`git ${args.join(" ")} failed: ${outcome.stderr}`); - } - return outcome.stdout; -} - -/** The host settings a command assembles, with one member under test. */ -function hostSettings(chosen: { readonly bundledPlugin: Plugin }): TestingHostSettings { - return { - includes: [], - plugins: [], - pluginArgs: [], - secretDetection: false, - // deno-lint-ignore require-yield - installService: function* (): Operation {}, - bundledPlugin: chosen.bundledPlugin, - testAgentWorker: Ok([...cliBase(), "test-agent"]), - planDeclaration: (request) => - planComponentDeclaration({ - surface: "component", - includes: [], - context: request.context, - ...(request.planWriterRoot === undefined ? {} : { planWriterRoot: request.planWriterRoot }), - host: request.host, - ...(request.observePlanWriter === undefined - ? {} - : { observePlanWriter: request.observePlanWriter }), - installElicitation: request.installElicitation, - }), - }; -} diff --git a/packages/git/tests/retained-repository-local.test.ts b/packages/git/tests/retained-repository-local.test.ts index 198dd0424..13fcfa735 100644 --- a/packages/git/tests/retained-repository-local.test.ts +++ b/packages/git/tests/retained-repository-local.test.ts @@ -594,8 +594,9 @@ describe("one entry's local file work", () => { // Both branches wrote, and nothing else did. Sorted, because which of two // concurrent branches reaches its write first is the scheduler's business // and asserting one order here would make this row fail for a reason it is - // not about. - expect(counted.writes.toSorted()).toEqual(["a.md", "b.md"]); + // not about. Copied before sorting: `toSorted` is ES2023 and the Node + // typecheck's lib is ES2022. + expect([...counted.writes].sort()).toEqual(["a.md", "b.md"]); // The condition that makes this row discriminating: two saved routings, on // two different coroutines. A host that remembered routing per *entry* // would skip one of them and the continuation below would be refused. diff --git a/scripts/runtime-test-exclusions.ts b/scripts/runtime-test-exclusions.ts index 977a31721..10f67676f 100644 --- a/scripts/runtime-test-exclusions.ts +++ b/scripts/runtime-test-exclusions.ts @@ -688,6 +688,12 @@ const DENO_ONLY_REPOSITORY_PROVIDER: RuntimeExclusion[] = [ "holds a shared Git directory under a kernel-released exclusive advisory lock taken through the Deno runtime, and spawns a second Deno process to prove the cross-process half; Node and Bun expose no equivalent", issue: DERIVED_SCOPE, }, + { + path: "packages/cli/tests/testing-execution-host-native.test.ts", + reason: + "its subject is the configured native Plugin an isolated run child initializes, which comes from @executablemd/git/deno and so from Workflow's storage adapter and node:sqlite. What a child does with the *selected* value — that a test root hands its host=\"run\" child what the host chose, reinstalled in the child's own scope after its working directory — is proven portably by packages/cli/tests/testing-execution-host.test.ts, which runs everywhere", + issue: DERIVED_SCOPE, + }, { path: "packages/cli/tests/repl-repositories.test.ts", reason: diff --git a/tsconfig.node.json b/tsconfig.node.json index ddaf8742b..44331d6ca 100644 --- a/tsconfig.node.json +++ b/tsconfig.node.json @@ -128,6 +128,11 @@ // canonicalizes against the host filesystem and asks the exact observed // path for its version through `Deno.Command`. The runtime exclusions keep // the same test out of the Node and Bun execution suites. - "packages/runtime/tests/executable-observer.test.ts" + "packages/runtime/tests/executable-observer.test.ts", + // A REPL entry's hold on a checkout is a kernel-released advisory lock, so + // this proof observes `Deno.openSync` directly and spawns a second + // `Deno.Command` to take the same sidecar from another process. The runtime + // exclusions keep the same test out of the Node and Bun execution suites. + "packages/git/tests/retained-repository-lifecycle.test.ts" ] }