diff --git a/.gitignore b/.gitignore index 49206df..dd63c53 100644 --- a/.gitignore +++ b/.gitignore @@ -12,6 +12,8 @@ .idea/ # Local AI-assistant configuration and state. +# .mcp.json is each contributor's client wiring, not project source; see docs/mcp-clients.md. +/.mcp.json /.agent/ /.agents/ /.ai/ diff --git a/CHANGELOG.md b/CHANGELOG.md index a7542a8..c21803d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,17 @@ All notable changes to tagged releases are documented here. +## Unreleased + +- Add an EXPERIMENTAL evaluator (ADR-0007): `judgment-pack experimental evaluate` and the + `experimental_evaluate` MCP tool apply the JPS Core §§7–8 experiment, as pinned by the + specification's RFC 0006 (Draft), to one conformant pack, one facts document, and an optional + tri-state evidence-availability input, returning a disposition (kind, reasons, handoff) and a + trace. It claims NO evaluator conformance — JPS `0.1.0-draft` forbids such claims — authorizes + nothing, evaluates only fully conformant packs, and may change or be removed without + compatibility promise. The nine walked instances committed in RFC 0006's appendix run as this + surface's acceptance tests. Validation behavior, claims, and exit classes are unchanged. + ## 0.1.0 - 2026-07-25 - Add a `NOTICE` file identifying Brian Jin as the copyright holder and carrying diff --git a/README.md b/README.md index 5c9b9b0..c3bccbe 100644 --- a/README.md +++ b/README.md @@ -12,10 +12,13 @@ schemas, and conformance corpus are owned by the separate [`judgment-pack-spec`](https://github.com/Judgment-Pack/judgment-pack-spec) repository, and any independent implementation that passes the published conformance corpus is equally valid. -The runtime **validates documents**. It does not evaluate rules, choose an outcome, fetch a source, -authorize an action, or establish truth, organizational authority, safety, or operational fitness. -This matches the scope of JPS `0.1.0-draft`, which defines carrier, structural, and semantic -document conformance only and defines no evaluator conformance class. +The runtime **validates documents**. It does not fetch a source, authorize an action, or establish +truth, organizational authority, safety, or operational fitness. This matches the scope of JPS +`0.1.0-draft`, which defines carrier, structural, and semantic document conformance only and +defines no evaluator conformance class. Its one evaluation surface is explicitly EXPERIMENTAL: +`judgment-pack experimental evaluate` (and the `experimental_evaluate` MCP tool) runs the +specification's §§7–8 experiment per [ADR-0007](docs/adr/0007-experimental-evaluator.md), claims +no conformance, and may change or be removed without compatibility promise. The command binary is `judgment-pack`; release archives also ship a `jpack` short alias for the same program. @@ -27,14 +30,18 @@ judgment-pack version judgment-pack spec validate judgment-pack spec test-conformance [suite] judgment-pack spec schema +judgment-pack spec examples [name] judgment-pack mcp +judgment-pack experimental evaluate (EXPERIMENTAL; no conformance claim) ``` The namespace is `judgment-pack spec`, not `judgment-pack jps`. JPS remains the name of the specification and the prefix of its provisional diagnostic codes. The `mcp` command serves the same offline operations to a Model Context Protocol client over stdio, -so an agent can validate documents as a tool call; see [docs/agent-testing.md](docs/agent-testing.md). +so an agent can validate documents as a tool call; see +[docs/mcp-clients.md](docs/mcp-clients.md) for per-client setup and +[docs/agent-testing.md](docs/agent-testing.md) for the agent-driven testing protocol. ## Install a tagged release diff --git a/docs/adr/0003-mcp-integration-and-testing-surface.md b/docs/adr/0003-mcp-integration-and-testing-surface.md index 63c37a1..99bcc6b 100644 --- a/docs/adr/0003-mcp-integration-and-testing-surface.md +++ b/docs/adr/0003-mcp-integration-and-testing-surface.md @@ -8,6 +8,10 @@ deciders: Brian Jin > **Accepted.** Phase 0 passed and Phase 1 is built: `internal/mcp` is a hand-rolled stdio MCP > server wired as the `judgment-pack mcp` subcommand. +> +> **Amended by [0007](0007-experimental-evaluator.md):** the surface gains exactly one explicitly +> experimental evaluation tool (`experimental_evaluate`); the validation tools still evaluate +> nothing. ## Context and problem statement diff --git a/docs/adr/0007-experimental-evaluator.md b/docs/adr/0007-experimental-evaluator.md new file mode 100644 index 0000000..f542281 --- /dev/null +++ b/docs/adr/0007-experimental-evaluator.md @@ -0,0 +1,93 @@ +--- +status: proposed +date: 2026-07-27 +deciders: Brian Jin +--- + +# Ship an experimental evaluator behind an explicit experimental surface + +> **Proposed.** Implement JPS Core's §§7–8 experiment — as pinned by the specification's +> [RFC 0006 (Draft)](https://github.com/Judgment-Pack/judgment-pack-spec/blob/main/rfcs/0006-evaluator-conformance.md) +> — as `judgment-pack experimental evaluate` and the `experimental_evaluate` MCP tool, claiming no +> evaluator conformance. + +## Context and problem statement + +The specification published RFC 0006 (Draft): a proposed normative evaluator conformance class +whose acceptance is gated on evidence from two independent implementations. Core `0.1.0-draft` +§3.4 forbids any evaluator-conformance claim, and the specification's tooling-architecture doc +pre-states the guardrails for exactly this situation: an experimental evaluator must be +unmistakably labeled, produce no conformance claim, and remain outside the default validation +path. Someone has to build the first implementation to generate the RFC's evidence; this runtime +is the natural first implementer. Separately, agent-driven authoring (ADR-0006) today proves that +a pack is *well-formed* but not that its logic is *correct* — an evaluator closes that loop. + +## Decision drivers + +- Generate RFC 0006's acceptance evidence: real implementation experience against its pinned + semantics and its nine committed instances. +- Preserve the runtime's identity: stateless, keyless, offline `bytes -> result`; evaluation is + still a pure function over supplied inputs, with no store and no side effect. +- Never contaminate the conformance surface: validation behavior, claims, and exit classes stay + untouched; the experiment must be impossible to mistake for a standard. + +## Considered options + +- **A. An `experimental` command namespace in this runtime** — CLI `judgment-pack experimental + evaluate` plus one clearly-labeled MCP tool. +- **B. A separate experimental binary or repository.** +- **C. Wait for the RFC to be accepted first.** + +## Decision outcome + +Chosen option: **A**. Option C is circular — acceptance requires implementations. Option B +fragments distribution for no isolation gain: the guardrail is the labeled, non-default surface, +not a separate artifact, and the shared engine seam (one core, per ADR-0002) is the point. + +Settled constraints: + +- **Semantics:** §§7–8 exactly as pinned by RFC 0006's sketch — three-valued conditions including + the §7 preamble's `literal` and §7.5 evidence presence; required-evidence check restated as + `missing-required-evidence` iff presence is false and `unknown` iff presence is unknown with + none false; ordered comparison defined only over §2.2 grammar-conforming JSON strings (JSON + numbers yield `unknown`); conflict never tie-broken; §8's order treated as contract, not + algorithm. +- **Inputs:** a pack (validated to full document conformance first — a non-conformant pack is + refused as an error, never evaluated), one JSON facts document, an optional tri-state + evidence-availability document keyed by declared requirement ids (omitted key = `unknown`; + undeclared key = input error), and the supported-extension set. +- **Output:** RFC 0006's disposition (`kind`, `outcomeId`, deduplicated `reasons`, `handoff`) + plus a minimal trace, in an envelope that carries `"experimental": true` and + `"conformanceClaim": "none"`. Producing a disposition exits 0 — the disposition is data, not a + process verdict; operational failures use the existing exit classes. +- **Errors are not dispositions:** invalid pack, unsupported required extension, malformed or + oversized inputs, and undeclared evidence keys are explicit errors. +- **Labeling:** the command group, tool name (`experimental_evaluate`), help text, tool + description, and result envelope all carry the experimental marker and the no-claim statement. + Diagnostic codes minted for this surface (`JPS-EVALUATION-*`) stay `codeStability: + "provisional"` like every other code; the experimental marking is carried by the envelope + (`experimental`, `conformanceClaim`) and the surface naming, not by a new stability tier. +- **Scope:** single-pack evaluation only. Judgment Graph composition (spec RFC 0002) and planner + selection (spec RFC 0004) are explicitly out of scope — the graph composes the dispositions + this surface produces and cannot be implemented before their semantics exist. + +This amends [0003](0003-mcp-integration-and-testing-surface.md) narrowly: the existing validation +and description tools still evaluate nothing, and the MCP surface as a whole no longer "evaluates nothing" — it +gains exactly one tool that evaluates *experimentally and says so*. + +### Consequences + +- Good, because RFC 0006 gains its first implementation and executable evidence (the nine + instances become this surface's acceptance tests). +- Good, because agents can close the logic-correctness loop (author → validate → evaluate against + sample facts) entirely offline and keyless. +- Bad, because an experimental surface invites misuse as if it were conformant; mitigated by + labeling in every artifact and by §3.4's permanent prohibition for `0.1.0-draft`. +- Bad, because the surface may change or be removed without compatibility promise as RFC 0006 + evolves; that is the meaning of experimental. + +## More information + +Semantics source: JPS Core §§7–8 and spec RFC 0006's Specification sketch and appendix. Guardrails: +the specification's tooling-architecture document. Follows ADR-0002 (one core), ADR-0003 (MCP +surface), and ADR-0006 (client-held lifecycle; evaluation inputs cross the wire as text). diff --git a/docs/adr/README.md b/docs/adr/README.md index 8f02335..4474298 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -35,3 +35,4 @@ when** it became that way. architecture.md links out to the relevant ADR rather | [0004](0004-decline-http-api.md) | Permanently decline an in-runtime HTTP API | accepted | | [0005](0005-single-jps-diagnostic-code-prefix.md) | Use a single `JPS-` prefix for diagnostic codes | accepted | | [0006](0006-authoring-lifecycle-in-the-client.md) | The authoring lifecycle lives in the client; the runtime is a stateless oracle | accepted | +| [0007](0007-experimental-evaluator.md) | Ship an experimental evaluator behind an explicit experimental surface | proposed | diff --git a/docs/agent-testing.md b/docs/agent-testing.md index b77124a..c767ab8 100644 --- a/docs/agent-testing.md +++ b/docs/agent-testing.md @@ -20,8 +20,9 @@ CLI loop here needs neither an SDK nor a running server. - A `judgment-pack` binary (installed or built below). - Any agent client that can run a shell command and read and write files, configured with **your - own** model API key. The runtime holds no key, opens no network connection, and evaluates nothing, - so the key lives entirely in the client. Any of the many such clients works — for example Claude + own** model API key. The runtime holds no key and opens no network connection — the key lives + entirely in the client. (Its only evaluation surface is the explicitly experimental, + non-conformance-claiming evaluator of ADR-0007.) Any of the many such clients works — for example Claude Code, Cline, Cursor, or Continue. ## Get the validator diff --git a/docs/architecture.md b/docs/architecture.md index 8875f76..8c035a1 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -38,8 +38,10 @@ required-extension capability check one result model ──► human or versioned JSON renderer ``` -Each layer stops when the preceding layer fails. No layer evaluates a condition, resolves a -decision outcome, fetches a locator, loads an extension, or authorizes an action. +Each layer stops when the preceding layer fails. No validation layer evaluates a condition, +resolves a decision outcome, fetches a locator, loads an extension, or authorizes an action; +evaluation exists only behind the explicitly experimental surface of +[ADR-0007](adr/0007-experimental-evaluator.md). ## Packages @@ -50,6 +52,7 @@ decision outcome, fetches a locator, loads an extension, or authorizes an action - `internal/fssecure` opens selected local files defensively and enforces bounded regular-file reads. - `internal/result` defines machine output version 1 and exit classes. - `internal/describe` composes the machine descriptions the CLI and MCP server share, so neither drifts. +- `internal/evaluation` implements the EXPERIMENTAL, non-conformance-claiming JPS §§7–8 evaluator (ADR-0007). - `internal/cli` owns commands, streams, and human/JSON rendering. - `internal/mcp` adapts the offline core onto Model Context Protocol tools over stdio. - `tools/sync-spec-artifacts` is an explicit maintainer-only snapshot importer. diff --git a/docs/authoring-lifecycle.md b/docs/authoring-lifecycle.md index 3734d8f..0386fb4 100644 --- a/docs/authoring-lifecycle.md +++ b/docs/authoring-lifecycle.md @@ -12,7 +12,8 @@ The runtime is a **stateless oracle**: `bytes -> result`. Every step below consu > *Is what you now hold conformant?* It cannot distinguish Create from Update, because both arrive as "validate this document." It keeps -no store, holds no credential, opens no network connection, and evaluates nothing. So the authoring +no store, holds no credential, and opens no network connection; its one evaluation surface is the +explicitly EXPERIMENTAL `experimental_evaluate` (ADR-0007), which claims no conformance. So the authoring loop — the part that holds a document, edits it, saves it, and deletes it — lives entirely in the **client**. That is the whole of ADR-0006, and everything here follows from it. @@ -167,8 +168,9 @@ remains. - No store: it never saves, names, lists, overwrites, or deletes user content. - No credential and no network: the client owns the key; the server opens no connection. - No path over the wire: MCP documents are values (text), never references to the server's filesystem. -- No evaluation: it validates conformance; it never resolves a condition, chooses an outcome, or - authorizes anything. +- No evaluation in the authoring loop: validation never resolves a condition, chooses an outcome, + or authorizes anything. The separate `experimental_evaluate` tool evaluates experimentally, says + so in every payload, and still authorizes nothing. ## See also diff --git a/docs/mcp-clients.md b/docs/mcp-clients.md new file mode 100644 index 0000000..e95851c --- /dev/null +++ b/docs/mcp-clients.md @@ -0,0 +1,111 @@ +# Connecting MCP clients + +`judgment-pack mcp` serves the offline validator to any Model Context Protocol client over stdio: +no port, no credential, no network. The client supplies its own model and API key; the runtime +never sees either (see [ADR-0003](adr/0003-mcp-integration-and-testing-surface.md) and +[ADR-0006](adr/0006-authoring-lifecycle-in-the-client.md)). + +Every client below launches the same binary. Install a released `judgment-pack` (see the +[README](../README.md)) and make sure the command the client launches resolves — if `judgment-pack` +is not on the PATH the client's launcher inherits, use the absolute path instead; a wrong path is +the most common failure and most clients report it poorly. + +Client configuration is personal wiring, not project source, so the local config files below are +gitignored in this repository. Copy a snippet, don't commit one. + +## The tools + +| Tool | Purpose | +| --- | --- | +| `validate` | Carrier, structural, and semantic conformance for one document | +| `test_conformance` | Run a version-pinned conformance corpus | +| `get_schema` | The exact bundled JSON Schema, with digest | +| `describe_runtime` | Versions and artifact provenance | +| `list_examples` / `get_example` | The embedded valid fixtures, read-only | +| `experimental_evaluate` | EXPERIMENTAL (ADR-0007): the §§7–8 experiment; claims no conformance | + +None of these evaluate, decide, or authorize anything, except `experimental_evaluate`, which +evaluates experimentally and says so in every payload. + +## Claude Code + +One command, from any directory: + +```bash +claude mcp add --scope user judgment-pack -- judgment-pack mcp +``` + +Or per project, in `.mcp.json` at the project root (gitignored here): + +```json +{ + "mcpServers": { + "judgment-pack": { + "type": "stdio", + "command": "judgment-pack", + "args": ["mcp"] + } + } +} +``` + +Restart the session, approve the server when prompted, and confirm with `/mcp`. + +## OpenAI Codex CLI + +In `~/.codex/config.toml`: + +```toml +[mcp_servers.judgment-pack] +command = "judgment-pack" +args = ["mcp"] +``` + +Or equivalently: `codex mcp add judgment-pack -- judgment-pack mcp`. Start `codex` and confirm +with `/mcp`. All tools are read-only except the experimental evaluator, so an automatic approval +mode is reasonable. + +## GitHub Copilot (VS Code) + +In `.vscode/mcp.json` at the workspace root (gitignored here) — note the root key is `servers`, +not `mcpServers`: + +```json +{ + "servers": { + "judgment-pack": { + "type": "stdio", + "command": "judgment-pack", + "args": ["mcp"] + } + } +} +``` + +For a user-level server instead, run **MCP: Open User Configuration** from the Command Palette. +Copilot uses MCP tools in agent mode; see the +[VS Code MCP documentation](https://code.visualstudio.com/docs/agents/reference/mcp-configuration) +and [GitHub's Copilot MCP guide](https://docs.github.com/copilot/customizing-copilot/using-model-context-protocol/extending-copilot-chat-with-mcp) +for the current UI flow. The Copilot CLI has its own +[MCP configuration](https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/add-mcp-servers). + +## Remote-only clients + +Some clients (the ChatGPT desktop app's connectors, Microsoft Copilot Studio) cannot launch a +local process and require a public HTTPS MCP endpoint. This runtime deliberately does not serve +HTTP ([ADR-0004](adr/0004-decline-http-api.md)); reaching those clients means an external +stdio-to-HTTP bridge plus a tunnel, which is developer tooling outside this repository, or a +separately governed hosted distribution. + +## Troubleshooting + +- **Server fails to start:** run `judgment-pack mcp` in a terminal yourself; if it starts and + waits on stdin, the binary is fine and the problem is the client's command path or config file. +- **Config edits not picked up:** clients read MCP configuration at session start — restart the + session. +- **Old binary:** `judgment-pack version` must report a release that has the `mcp` command + (v0.1.0 or later); a stale build on the PATH is easy to pick up by mistake. + +The end-to-end authoring loop these tools support — create, read, update, delete, with the +runtime as a stateless oracle — is described in [authoring-lifecycle.md](authoring-lifecycle.md), +and the agent-driven testing protocol in [agent-testing.md](agent-testing.md). diff --git a/internal/cli/app.go b/internal/cli/app.go index d2fc29f..8226958 100644 --- a/internal/cli/app.go +++ b/internal/cli/app.go @@ -15,6 +15,7 @@ import ( "github.com/Judgment-Pack/judgment-pack-runtime/internal/conformance" "github.com/Judgment-Pack/judgment-pack-runtime/internal/describe" "github.com/Judgment-Pack/judgment-pack-runtime/internal/display" + "github.com/Judgment-Pack/judgment-pack-runtime/internal/evaluation" "github.com/Judgment-Pack/judgment-pack-runtime/internal/fssecure" "github.com/Judgment-Pack/judgment-pack-runtime/internal/mcp" "github.com/Judgment-Pack/judgment-pack-runtime/internal/result" @@ -75,7 +76,7 @@ func (a *App) rootCommand() *cobra.Command { root := &cobra.Command{ Use: "judgment-pack", Short: "Judgment Pack Specification tooling", - Long: "Judgment Pack Specification tooling. The spec commands validate JPS documents; they do not evaluate decisions or authorize actions.", + Long: "Judgment Pack Specification tooling. The spec commands validate JPS documents; they do not evaluate decisions or authorize actions. The experimental namespace is the one exception: it evaluates experimentally, claims no conformance, and may change without notice.", SilenceErrors: true, SilenceUsage: true, Version: result.CLIVersion, @@ -90,10 +91,97 @@ func (a *App) rootCommand() *cobra.Command { root.SetVersionTemplate("judgment-pack {{.Version}}\n") root.CompletionOptions.DisableDefaultCmd = true root.PersistentFlags().BoolVar(&a.pretty, "pretty", false, "indent JSON output") - root.AddCommand(a.versionCommand(), a.specCommand(), a.mcpCommand()) + root.AddCommand(a.versionCommand(), a.specCommand(), a.mcpCommand(), a.experimentalCommand()) return root } +func (a *App) experimentalCommand() *cobra.Command { + experimental := &cobra.Command{ + Use: "experimental", + Short: "Experimental operations that claim no conformance and may change or vanish", + Long: "Experimental operations (ADR-0007). Nothing under this command claims any JPS conformance; JPS 0.1.0-draft §3.4 forbids evaluator-conformance claims outright. Behavior may change or be removed without compatibility promise.", + Args: cobra.NoArgs, + RunE: func(command *cobra.Command, _ []string) error { + return command.Help() + }, + } + experimental.AddCommand(a.evaluateCommand()) + return experimental +} + +func (a *App) evaluateCommand() *cobra.Command { + format := "human" + factsPath := "" + evidencePath := "" + supported := []string{} + command := &cobra.Command{ + Use: "evaluate ", + Short: "EXPERIMENTAL: apply the JPS §§7-8 experiment to one pack; no conformance claim", + Long: "Apply the experimental JPS Core §§7-8 resolution model, as pinned by the specification's RFC 0006 (Draft), to one conformant pack and one facts document. The result is a disposition, not a conformance claim, an authorization, or an executed action; producing any disposition exits 0.", + Args: cobra.ExactArgs(1), + RunE: func(_ *cobra.Command, args []string) error { + if err := validateFormat(format); err != nil { + return a.operational("experimental evaluate", format, result.ExitInvocation, "JPS-INVOCATION-FORMAT", err.Error()) + } + if factsPath == "" { + return a.operational("experimental evaluate", format, result.ExitInvocation, "JPS-INVOCATION-FACTS", "--facts is required: one JSON facts document (a file path, or - for standard input).") + } + if args[0] == "-" && factsPath == "-" { + return a.operational("experimental evaluate", format, result.ExitInvocation, "JPS-INVOCATION-STDIN", "The pack and the facts document cannot both be standard input.") + } + if evidencePath == "-" { + return a.operational("experimental evaluate", format, result.ExitInvocation, "JPS-INVOCATION-STDIN", "The evidence document cannot be standard input; pass a file path.") + } + for _, input := range []string{args[0], factsPath, evidencePath} { + if input != "" && input != "-" && (strings.Contains(input, "://") || fssecure.IsRemotePath(input)) { + return a.operational("experimental evaluate", format, result.ExitInvocation, "JPS-INVOCATION-INPUT", "URL and remote filesystem inputs are not supported; use local files or standard input.") + } + } + pack, err := a.readPack(args[0], carrier.HardMaxBytes) + if err != nil { + return a.evaluateReadFailure(format, "pack", err) + } + facts, err := a.readPack(factsPath, carrier.HardMaxBytes) + if err != nil { + return a.evaluateReadFailure(format, "facts", err) + } + var evidence []byte + if evidencePath != "" { + evidence, err = fssecure.ReadRegular(evidencePath, carrier.HardMaxBytes) + if err != nil { + return a.evaluateReadFailure(format, "evidence", err) + } + if len(evidence) == 0 { + return a.operational("experimental evaluate", format, result.ExitInvocation, "JPS-EVALUATION-INPUT-JSON", "The evidence file is empty; supply a JSON object, or omit --evidence to treat every requirement as unknown.") + } + } + evaluator := evaluation.NewEngine(a.engine) + output, failure := evaluator.Evaluate(pack, facts, evidence, supported, "experimental evaluate") + if failure != nil { + return a.operational("experimental evaluate", format, failure.ExitCode, failure.Code, failure.Message) + } + if err := a.renderEvaluation(format, output); err != nil { + return &handledExit{code: result.ExitIO} + } + return nil + }, + } + command.Flags().StringVar(&format, "format", format, "output format: human or json") + command.Flags().StringVar(&factsPath, "facts", factsPath, "JSON facts document: a file path, or - for standard input (required)") + command.Flags().StringVar(&evidencePath, "evidence", evidencePath, "optional tri-state evidence availability: {\"\": \"present\"|\"absent\"|\"unknown\"}") + command.Flags().StringArrayVar(&supported, "supported-extension", supported, "extension name this consumer supports (repeatable)") + return command +} + +// evaluateReadFailure reports a failed evaluation-input read, keeping the +// byte-limit case as informative as spec validate's. +func (a *App) evaluateReadFailure(format, name string, err error) error { + if errors.Is(err, fssecure.ErrTooLarge) { + return a.operational("experimental evaluate", format, result.ExitIO, "JPS-RESOURCE-INPUT-BYTE-LIMIT", fmt.Sprintf("The %s input exceeds the hard %d-byte limit.", name, carrier.HardMaxBytes)) + } + return a.operational("experimental evaluate", format, result.ExitIO, "JPS-INPUT-READ", fmt.Sprintf("The %s input could not be read as one bounded regular file or standard input stream.", name)) +} + func (a *App) specCommand() *cobra.Command { spec := &cobra.Command{ Use: "spec", @@ -111,8 +199,8 @@ func (a *App) specCommand() *cobra.Command { func (a *App) mcpCommand() *cobra.Command { return &cobra.Command{ Use: "mcp", - Short: "Serve the offline validator to an MCP client over stdio", - Long: "Run a Model Context Protocol server on standard input and output. It exposes the offline validation, conformance, and description operations as MCP tools; it holds no credential, opens no network connection, and evaluates nothing.", + Short: "Serve the offline validator and one experimental tool to an MCP client over stdio", + Long: "Run a Model Context Protocol server on standard input and output. It exposes the offline validation, conformance, and description operations as MCP tools, plus one explicitly EXPERIMENTAL evaluation tool (experimental_evaluate; ADR-0007) that claims no conformance. It holds no credential, opens no network connection, and authorizes nothing.", Args: cobra.NoArgs, RunE: func(_ *cobra.Command, _ []string) error { server := mcp.NewServer(a.engine, a.runner) @@ -456,15 +544,23 @@ func requestedFormat(args []string) string { } func requestedCommand(args []string) string { + // One scan, first command group wins, so an operand that happens to spell a + // group name (for example a file called "spec") cannot relabel the command. for index, argument := range args { - if argument != "spec" || index+1 >= len(args) { - continue - } - switch args[index+1] { - case "validate", "test-conformance", "schema", "examples": - return "spec " + args[index+1] - default: + switch argument { + case "spec": + if index+1 < len(args) { + switch args[index+1] { + case "validate", "test-conformance", "schema", "examples": + return "spec " + args[index+1] + } + } return "spec" + case "experimental": + if index+1 < len(args) && args[index+1] == "evaluate" { + return "experimental evaluate" + } + return "experimental" } } for _, argument := range args { diff --git a/internal/cli/app_test.go b/internal/cli/app_test.go index 85c1e69..97ac9a9 100644 --- a/internal/cli/app_test.go +++ b/internal/cli/app_test.go @@ -436,3 +436,41 @@ func TestMCPSubcommandRespondsToInitialize(t *testing.T) { t.Fatalf("unexpected initialize response: %#v", response) } } + +// The experimental evaluate command produces a labeled disposition and exits 0 +// for any produced disposition; its invocation guards use the standard codes. +func TestExperimentalEvaluateCommand(t *testing.T) { + pack := filepath.Join("..", "evaluation", "testdata", "data-request-intake-triage.json") + dir := t.TempDir() + facts := filepath.Join(dir, "facts.json") + if err := os.WriteFile(facts, []byte(`{"request":{"type":"data-access","completeness":"complete","appropriateness":"hard-fail","embargoedInformationToUnauthorizedRecipients":false}}`), 0o644); err != nil { + t.Fatal(err) + } + evidence := filepath.Join(dir, "evidence.json") + if err := os.WriteFile(evidence, []byte(`{"intake-form":"present","sponsor-endorsement":"present"}`), 0o644); err != nil { + t.Fatal(err) + } + + code, stdout, stderr := runTest(t, []string{"experimental", "evaluate", pack, "--facts", facts, "--evidence", evidence, "--format", "json"}, "") + if code != 0 || stderr != "" { + t.Fatalf("exit=%d stderr=%q stdout=%q", code, stderr, stdout) + } + var output map[string]any + if err := json.Unmarshal([]byte(stdout), &output); err != nil { + t.Fatal(err) + } + if output["experimental"] != true || output["conformanceClaim"] != "none" { + t.Fatalf("evaluation output must be labeled experimental with no claim: %v", output) + } + disposition := output["disposition"].(map[string]any) + if disposition["kind"] != "outcome" || disposition["outcomeId"] != "decline-redirect" { + t.Fatalf("disposition = %v", disposition) + } + + // --facts is required. + code, stdout, _ = runTest(t, []string{"experimental", "evaluate", pack, "--format", "json"}, "") + if code != result.ExitInvocation { + t.Fatalf("missing --facts must be an invocation error, got exit=%d", code) + } + assertDiagnosticCode(t, stdout, "JPS-INVOCATION-FACTS") +} diff --git a/internal/cli/render.go b/internal/cli/render.go index 59069bf..3977e25 100644 --- a/internal/cli/render.go +++ b/internal/cli/render.go @@ -132,6 +132,50 @@ func (a *App) renderExample(format string, output result.Example) error { return nil } +func (a *App) renderEvaluation(format string, output result.Evaluation) error { + if format == "json" { + return a.writeJSON(output) + } + fmt.Fprintln(a.out, "EXPERIMENTAL evaluation (no conformance claim; JPS 0.1.0-draft defines no evaluator conformance)") + switch output.Disposition.Kind { + case "outcome": + fmt.Fprintf(a.out, "disposition: outcome %s\n", display.Sanitize(output.Disposition.OutcomeID)) + default: + fmt.Fprintf(a.out, "disposition: %s (%s)\n", display.Sanitize(output.Disposition.Kind), display.Sanitize(strings.Join(output.Disposition.Reasons, ", "))) + } + if output.Disposition.Handoff.State == "requested" { + if target := output.Disposition.Handoff.Target; target != nil { + fmt.Fprintf(a.out, "handoff: requested -> %s %q\n", display.Sanitize(target.Kind), display.Sanitize(target.Name)) + } else { + fmt.Fprintln(a.out, "handoff: requested (no declared destination)") + } + } + for _, entry := range output.Trace { + note := entry.Condition + if entry.Suppressed { + note = "suppressed" + } + if entry.Skipped { + note = "skipped (forced outcome)" + } + detail := "" + if entry.Effect != "" { + detail = " effect=" + entry.Effect + } + if entry.Outcome != "" { + detail += " outcome=" + entry.Outcome + } + if entry.OnUnknown != "" { + detail += " onUnknown=" + entry.OnUnknown + } + fmt.Fprintf(a.out, "trace: %s %s: %s%s\n", display.Sanitize(entry.Stage), display.Sanitize(entry.ID), display.Sanitize(note), display.Sanitize(detail)) + } + if output.Artifact != nil { + fmt.Fprintf(a.out, "artifacts: %s · sha256 %s\n", display.Sanitize(output.Artifact.Provenance), output.Artifact.BundleDigest) + } + return nil +} + func joinNonEmpty(values ...string) string { output := []string{} for _, value := range values { diff --git a/internal/evaluation/condition.go b/internal/evaluation/condition.go new file mode 100644 index 0000000..9fc8a43 --- /dev/null +++ b/internal/evaluation/condition.go @@ -0,0 +1,356 @@ +package evaluation + +import ( + "encoding/json" + "math/big" + "regexp" + "strconv" + "strings" +) + +// tri is the three-valued condition result of the §7 experiment. +type tri int + +const ( + triFalse tri = iota + triTrue + triUnknown +) + +func (t tri) String() string { + switch t { + case triTrue: + return "true" + case triFalse: + return "false" + default: + return "unknown" + } +} + +func triFromBool(value bool) tri { + if value { + return triTrue + } + return triFalse +} + +// evalCondition applies §7's experimental interpretation, as pinned by spec +// RFC 0006, to one condition node. The pack has full document conformance, so +// node shapes are schema-guaranteed; anything unexpected still degrades to +// unknown ("unsupported operators or operand shapes ... produce unknown"). +func evalCondition(node any, facts any, evidence map[string]tri) tri { + condition, ok := node.(map[string]any) + if !ok { + return triUnknown + } + switch condition["op"] { + case "literal": + if value, ok := condition["value"].(bool); ok { + return triFromBool(value) + } + return triUnknown + case "all": + return evalAll(condition["conditions"], facts, evidence) + case "any": + return evalAny(condition["conditions"], facts, evidence) + case "not": + switch evalCondition(condition["condition"], facts, evidence) { + case triTrue: + return triFalse + case triFalse: + return triTrue + default: + return triUnknown + } + case "fact": + return evalFact(condition, facts) + case "evidence-present": + name, ok := condition["evidenceRequirement"].(string) + if !ok { + return triUnknown + } + presence, declared := evidence[name] + if !declared { + return triUnknown + } + return presence + default: + return triUnknown + } +} + +// evalAll is strong three-valued conjunction (§7.1): false if any child is +// false, true if every child is true, unknown otherwise. +func evalAll(children any, facts any, evidence map[string]tri) tri { + items, ok := children.([]any) + if !ok { + return triUnknown + } + sawUnknown := false + for _, item := range items { + switch evalCondition(item, facts, evidence) { + case triFalse: + return triFalse + case triUnknown: + sawUnknown = true + } + } + if sawUnknown { + return triUnknown + } + return triTrue +} + +// evalAny is strong three-valued disjunction (§7.2): true if any child is +// true, false if every child is false, unknown otherwise. +func evalAny(children any, facts any, evidence map[string]tri) tri { + items, ok := children.([]any) + if !ok { + return triUnknown + } + sawUnknown := false + for _, item := range items { + switch evalCondition(item, facts, evidence) { + case triTrue: + return triTrue + case triUnknown: + sawUnknown = true + } + } + if sawUnknown { + return triUnknown + } + return triFalse +} + +// evalFact selects a value from the facts document by RFC 6901 pointer and +// applies the declared operator (§7.4). A pointer that does not resolve, an +// unsupported operand shape, or an incomparable ordered pair produces unknown. +func evalFact(condition map[string]any, facts any) tri { + path, ok := condition["path"].(string) + if !ok { + return triUnknown + } + value, resolved := resolvePointer(facts, path) + if !resolved { + return triUnknown + } + operand := condition["value"] + switch condition["operator"] { + case "equals": + equal, determinable := jsonEqual(value, operand) + if !determinable { + return triUnknown + } + return triFromBool(equal) + case "not-equals": + // §7.4: not-equals is the Boolean inverse of equals only when equality + // can be determined. + equal, determinable := jsonEqual(value, operand) + if !determinable { + return triUnknown + } + return triFromBool(!equal) + case "in": + items, ok := operand.([]any) + if !ok { + return triUnknown + } + sawUndeterminable := false + for _, item := range items { + equal, determinable := jsonEqual(value, item) + if !determinable { + sawUndeterminable = true + continue + } + if equal { + return triTrue + } + } + if sawUndeterminable { + return triUnknown + } + return triFalse + case "greater-than", "greater-than-or-equal", "less-than", "less-than-or-equal": + comparison, comparable := decimalCompare(value, operand) + if !comparable { + return triUnknown + } + switch condition["operator"] { + case "greater-than": + return triFromBool(comparison > 0) + case "greater-than-or-equal": + return triFromBool(comparison >= 0) + case "less-than": + return triFromBool(comparison < 0) + default: + return triFromBool(comparison <= 0) + } + default: + return triUnknown + } +} + +// resolvePointer resolves an RFC 6901 JSON Pointer against a decoded JSON +// document. The empty string selects the root. It reports false for any +// pointer that does not resolve, including an invalid array traversal and the +// "-" past-the-end token. +func resolvePointer(document any, pointer string) (any, bool) { + if pointer == "" { + return document, true + } + if !strings.HasPrefix(pointer, "/") { + return nil, false + } + current := document + for _, token := range strings.Split(pointer[1:], "/") { + token = strings.ReplaceAll(strings.ReplaceAll(token, "~1", "/"), "~0", "~") + switch container := current.(type) { + case map[string]any: + value, ok := container[token] + if !ok { + return nil, false + } + current = value + case []any: + index, ok := arrayIndex(token, len(container)) + if !ok { + return nil, false + } + current = container[index] + default: + return nil, false + } + } + return current, true +} + +// arrayIndex parses an RFC 6901 array-reference token: decimal digits with no +// leading zero (other than "0" itself), in range. "-" refers past the end and +// never resolves. +func arrayIndex(token string, length int) (int, bool) { + if token == "" || token == "-" { + return 0, false + } + if len(token) > 1 && token[0] == '0' { + return 0, false + } + for _, r := range token { + if r < '0' || r > '9' { + return 0, false + } + } + index, err := strconv.Atoi(token) + if err != nil || index >= length { + return 0, false + } + return index, true +} + +// jsonEqual is §7.4's type-preserving JSON equality: no coercion between +// types; numbers compare by mathematical value; arrays in order; objects by +// member name and value regardless of order. The second result reports +// whether equality could be determined at all: a syntactically valid JSON +// number the arithmetic cannot represent (for example an exponent beyond +// big.Rat's range) makes the comparison undeterminable rather than silently +// false — §7.4: "incomparable values ... produce unknown". +func jsonEqual(a, b any) (bool, bool) { + switch left := a.(type) { + case nil: + return b == nil, true + case bool: + right, ok := b.(bool) + return ok && left == right, true + case string: + right, ok := b.(string) + return ok && left == right, true + case json.Number: + right, ok := b.(json.Number) + if !ok { + return false, true + } + if left.String() == right.String() { + // Identical tokens are equal regardless of representability. + return true, true + } + leftRat, leftOK := new(big.Rat).SetString(left.String()) + rightRat, rightOK := new(big.Rat).SetString(right.String()) + if !leftOK || !rightOK { + return false, false + } + return leftRat.Cmp(rightRat) == 0, true + case []any: + right, ok := b.([]any) + if !ok || len(left) != len(right) { + return false, true + } + sawUndeterminable := false + for i := range left { + equal, determinable := jsonEqual(left[i], right[i]) + if !determinable { + sawUndeterminable = true + continue + } + if !equal { + return false, true + } + } + if sawUndeterminable { + return false, false + } + return true, true + case map[string]any: + right, ok := b.(map[string]any) + if !ok || len(left) != len(right) { + return false, true + } + sawUndeterminable := false + for key, value := range left { + other, present := right[key] + if !present { + return false, true + } + equal, determinable := jsonEqual(value, other) + if !determinable { + sawUndeterminable = true + continue + } + if !equal { + return false, true + } + } + if sawUndeterminable { + return false, false + } + return true, true + default: + return false, true + } +} + +// decimalPattern is the §2.2 decimal grammar: an optional minus, an integer +// part with no leading zero, and an optional fraction. Exponents, leading +// plus signs, leading zeroes, NaN, and infinities are not admitted. +var decimalPattern = regexp.MustCompile(`^-?(0|[1-9][0-9]*)(\.[0-9]+)?$`) + +// decimalCompare implements RFC 0006's pinned ordering: an ordered comparison +// is defined iff both values are JSON strings conforming to the §2.2 decimal +// grammar, compared by mathematical value. Any other value — including a JSON +// number — is not comparable. +func decimalCompare(fact, operand any) (int, bool) { + factText, ok := fact.(string) + if !ok || !decimalPattern.MatchString(factText) { + return 0, false + } + operandText, ok := operand.(string) + if !ok || !decimalPattern.MatchString(operandText) { + return 0, false + } + factValue, factOK := new(big.Rat).SetString(factText) + operandValue, operandOK := new(big.Rat).SetString(operandText) + if !factOK || !operandOK { + return 0, false + } + return factValue.Cmp(operandValue), true +} diff --git a/internal/evaluation/condition_test.go b/internal/evaluation/condition_test.go new file mode 100644 index 0000000..97a07cf --- /dev/null +++ b/internal/evaluation/condition_test.go @@ -0,0 +1,354 @@ +package evaluation + +import ( + "encoding/json" + "reflect" + "testing" +) + +func TestDecimalCompare(t *testing.T) { + cases := []struct { + fact, operand any + want int + comparable bool + }{ + {"100", "99", 1, true}, + {"99", "100", -1, true}, + {"5000", "5000", 0, true}, + {"5000.00", "5000", 0, true}, // mathematical value, not lexical + {"-0.5", "0", -1, true}, + {json.Number("100"), "99", 0, false}, // JSON numbers are not decimals (RFC 0006) + {"0100", "99", 0, false}, // leading zero not admitted + {"1e3", "99", 0, false}, // exponent not admitted + {"+1", "1", 0, false}, // leading plus not admitted + {"1.", "1", 0, false}, // fraction needs digits + {true, "1", 0, false}, + {nil, "1", 0, false}, + } + for _, testCase := range cases { + got, comparable := decimalCompare(testCase.fact, testCase.operand) + if comparable != testCase.comparable || (comparable && got != testCase.want) { + t.Errorf("decimalCompare(%v, %v) = (%d, %v), want (%d, %v)", + testCase.fact, testCase.operand, got, comparable, testCase.want, testCase.comparable) + } + } +} + +func TestResolvePointer(t *testing.T) { + document := map[string]any{ + "a": map[string]any{"b/c": "slash", "~d": "tilde"}, + "arr": []any{"zero", "one"}, + } + cases := []struct { + pointer string + want any + resolved bool + }{ + {"", document, true}, + {"/a/b~1c", "slash", true}, + {"/a/~0d", "tilde", true}, + {"/arr/0", "zero", true}, + {"/arr/1", "one", true}, + {"/arr/2", nil, false}, // out of range + {"/arr/01", nil, false}, // leading zero + {"/arr/-", nil, false}, // past-the-end never resolves + {"/missing", nil, false}, + {"/arr/0/x", nil, false}, // traversal into a non-container + {"a", nil, false}, // no leading slash + } + for _, testCase := range cases { + got, resolved := resolvePointer(document, testCase.pointer) + if resolved != testCase.resolved { + t.Errorf("resolvePointer(%q) resolved = %v, want %v", testCase.pointer, resolved, testCase.resolved) + continue + } + if resolved && !reflect.DeepEqual(got, testCase.want) { + t.Errorf("resolvePointer(%q) = %v, want %v", testCase.pointer, got, testCase.want) + } + } +} + +func TestJSONEqualIsTypePreserving(t *testing.T) { + cases := []struct { + a, b any + want bool + determinable bool + }{ + {json.Number("5.0"), json.Number("5"), true, true}, // mathematical value + {json.Number("5"), "5", false, true}, // no coercion across types + {"5", json.Number("5"), false, true}, + {true, "true", false, true}, + {nil, nil, true, true}, + {[]any{json.Number("1"), "x"}, []any{json.Number("1.0"), "x"}, true, true}, + {[]any{"x", "y"}, []any{"y", "x"}, false, true}, // arrays compare in order + {map[string]any{"a": json.Number("1"), "b": "z"}, map[string]any{"b": "z", "a": json.Number("1.00")}, true, true}, + {map[string]any{"a": "1"}, map[string]any{"a": "1", "b": "2"}, false, true}, + // A representable-vs-unrepresentable pair is undeterminable, never a + // silent false (spec §7.4: incomparable values produce unknown)... + {json.Number("1e999999999"), json.Number("2e999999999"), false, false}, + {[]any{json.Number("1e999999999")}, []any{json.Number("2e999999999")}, false, false}, + // ...but identical tokens are equal regardless of representability. + {json.Number("1e999999999"), json.Number("1e999999999"), true, true}, + // A definite mismatch elsewhere still decides the comparison. + {[]any{json.Number("1e999999999"), "x"}, []any{json.Number("2e999999999"), "y"}, false, true}, + } + for _, testCase := range cases { + got, determinable := jsonEqual(testCase.a, testCase.b) + if determinable != testCase.determinable || (determinable && got != testCase.want) { + t.Errorf("jsonEqual(%v, %v) = (%v, %v), want (%v, %v)", testCase.a, testCase.b, got, determinable, testCase.want, testCase.determinable) + } + } +} + +func TestThreeValuedOperators(t *testing.T) { + facts := map[string]any{"known": "yes"} + unknownFact := map[string]any{"op": "fact", "path": "/missing", "operator": "equals", "value": "x"} + trueFact := map[string]any{"op": "fact", "path": "/known", "operator": "equals", "value": "yes"} + falseFact := map[string]any{"op": "fact", "path": "/known", "operator": "equals", "value": "no"} + + cases := []struct { + name string + node map[string]any + want tri + }{ + {"all-false-beats-unknown", map[string]any{"op": "all", "conditions": []any{unknownFact, falseFact}}, triFalse}, + {"all-unknown", map[string]any{"op": "all", "conditions": []any{unknownFact, trueFact}}, triUnknown}, + {"all-true", map[string]any{"op": "all", "conditions": []any{trueFact, trueFact}}, triTrue}, + {"any-true-beats-unknown", map[string]any{"op": "any", "conditions": []any{unknownFact, trueFact}}, triTrue}, + {"any-unknown", map[string]any{"op": "any", "conditions": []any{unknownFact, falseFact}}, triUnknown}, + {"any-false", map[string]any{"op": "any", "conditions": []any{falseFact, falseFact}}, triFalse}, + {"not-unknown-stays-unknown", map[string]any{"op": "not", "condition": unknownFact}, triUnknown}, + {"not-true", map[string]any{"op": "not", "condition": trueFact}, triFalse}, + {"literal", map[string]any{"op": "literal", "value": true}, triTrue}, + {"in-no-match-is-false", map[string]any{"op": "fact", "path": "/known", "operator": "in", "value": []any{"a", "b"}}, triFalse}, + {"in-match", map[string]any{"op": "fact", "path": "/known", "operator": "in", "value": []any{"yes"}}, triTrue}, + {"ordered-number-fact-unknown", map[string]any{"op": "fact", "path": "/known", "operator": "greater-than", "value": "1"}, triUnknown}, + } + for _, testCase := range cases { + if got := evalCondition(testCase.node, facts, map[string]tri{}); got != testCase.want { + t.Errorf("%s = %v, want %v", testCase.name, got, testCase.want) + } + } +} + +// Resolver edges exercised directly, without the validation gate, so states no +// conformant single pack can combine are still pinned. +func TestResolverEdges(t *testing.T) { + base := func() map[string]any { + return map[string]any{ + "outcomes": []any{ + map[string]any{"id": "a"}, map[string]any{"id": "b"}, + }, + } + } + + t.Run("direct-escalation-without-escalation-object", func(t *testing.T) { + pack := base() + pack["rules"] = []any{map[string]any{"id": "r", "when": map[string]any{"op": "literal", "value": true}, "outcome": "a", "onUnknown": "ignore"}} + pack["exceptions"] = []any{map[string]any{"id": "x", "when": map[string]any{"op": "literal", "value": true}, "effect": "escalate", "onUnknown": "ignore"}} + disposition, _ := resolve(pack, map[string]any{}, map[string]tri{}) + if disposition.Kind != "unresolved" || !reflect.DeepEqual(disposition.Reasons, []string{"exception-escalation"}) { + t.Fatalf("direct escalation must be unresolved: %+v", disposition) + } + if disposition.Handoff.State != "requested" || disposition.Handoff.Target != nil { + t.Fatalf("a direct request without an escalation object has no Core-defined destination: %+v", disposition.Handoff) + } + }) + + t.Run("conflicting-forced-outcomes", func(t *testing.T) { + pack := base() + pack["exceptions"] = []any{ + map[string]any{"id": "x1", "when": map[string]any{"op": "literal", "value": true}, "effect": "force-outcome", "outcome": "a", "onUnknown": "ignore"}, + map[string]any{"id": "x2", "when": map[string]any{"op": "literal", "value": true}, "effect": "force-outcome", "outcome": "b", "onUnknown": "ignore"}, + } + disposition, _ := resolve(pack, map[string]any{}, map[string]tri{}) + if disposition.Kind != "unresolved" || !reflect.DeepEqual(disposition.Reasons, []string{"conflict"}) { + t.Fatalf("incompatible forced outcomes must conflict: %+v", disposition) + } + }) + + t.Run("suppressed-rule-restores-fallback", func(t *testing.T) { + pack := base() + pack["rules"] = []any{map[string]any{"id": "r", "when": map[string]any{"op": "literal", "value": true}, "outcome": "a", "onUnknown": "ignore"}} + pack["exceptions"] = []any{map[string]any{"id": "x", "when": map[string]any{"op": "literal", "value": true}, "effect": "suppress-rule", "targetRule": "r", "onUnknown": "ignore"}} + pack["fallbackOutcome"] = "b" + disposition, trace := resolve(pack, map[string]any{}, map[string]tri{}) + if disposition.Kind != "outcome" || disposition.OutcomeID != "b" { + t.Fatalf("with the only rule suppressed the fallback applies: %+v", disposition) + } + found := false + for _, entry := range trace { + if entry.Stage == "rule" && entry.ID == "r" && entry.Suppressed { + found = true + } + } + if !found { + t.Fatalf("the suppressed rule must stay visible in the trace: %+v", trace) + } + }) + + t.Run("no-match-without-fallback", func(t *testing.T) { + pack := base() + pack["rules"] = []any{map[string]any{"id": "r", "when": map[string]any{"op": "literal", "value": false}, "outcome": "a", "onUnknown": "ignore"}} + disposition, _ := resolve(pack, map[string]any{}, map[string]tri{}) + if disposition.Kind != "unresolved" || !reflect.DeepEqual(disposition.Reasons, []string{"no-match"}) { + t.Fatalf("no rule and no fallback is no-match: %+v", disposition) + } + }) + + t.Run("unknown-rule-ignore-does-not-block-fallback", func(t *testing.T) { + pack := base() + pack["rules"] = []any{map[string]any{"id": "r", "when": map[string]any{"op": "fact", "path": "/missing", "operator": "equals", "value": "x"}, "outcome": "a", "onUnknown": "ignore"}} + pack["fallbackOutcome"] = "b" + disposition, trace := resolve(pack, map[string]any{}, map[string]tri{}) + if disposition.Kind != "outcome" || disposition.OutcomeID != "b" { + t.Fatalf("an ignored unknown must not block the fallback: %+v", disposition) + } + visible := false + for _, entry := range trace { + if entry.Stage == "rule" && entry.Condition == "unknown" && entry.OnUnknown == "ignore" { + visible = true + } + } + if !visible { + t.Fatalf("an ignored unknown stays visible in the trace: %+v", trace) + } + }) + + t.Run("unknown-rule-escalate-blocks-fallback", func(t *testing.T) { + pack := base() + pack["rules"] = []any{map[string]any{"id": "r", "when": map[string]any{"op": "fact", "path": "/missing", "operator": "equals", "value": "x"}, "outcome": "a", "onUnknown": "escalate"}} + pack["fallbackOutcome"] = "b" + disposition, _ := resolve(pack, map[string]any{}, map[string]tri{}) + if disposition.Kind != "unresolved" || !reflect.DeepEqual(disposition.Reasons, []string{"unknown"}) { + t.Fatalf("an escalating unknown blocks both candidate and fallback: %+v", disposition) + } + }) + + t.Run("retains-both-unknown-and-conflict", func(t *testing.T) { + pack := base() + pack["rules"] = []any{ + map[string]any{"id": "r1", "when": map[string]any{"op": "literal", "value": true}, "outcome": "a", "onUnknown": "ignore"}, + map[string]any{"id": "r2", "when": map[string]any{"op": "literal", "value": true}, "outcome": "b", "onUnknown": "ignore"}, + map[string]any{"id": "r3", "when": map[string]any{"op": "fact", "path": "/missing", "operator": "equals", "value": "x"}, "outcome": "a", "onUnknown": "escalate"}, + } + disposition, _ := resolve(pack, map[string]any{}, map[string]tri{}) + if !reflect.DeepEqual(disposition.Reasons, []string{"conflict", "unknown"}) { + t.Fatalf("neither unknown nor conflict is discarded: %+v", disposition) + } + }) +} + +// evidence-present maps the tri-state availability directly (§7.5): present is +// true, absent is false, unknown or undeclared is unknown. +func TestEvidencePresentTriState(t *testing.T) { + evidence := map[string]tri{"a": triTrue, "b": triFalse, "c": triUnknown} + cases := []struct { + name string + want tri + }{{"a", triTrue}, {"b", triFalse}, {"c", triUnknown}, {"undeclared", triUnknown}} + for _, testCase := range cases { + node := map[string]any{"op": "evidence-present", "evidenceRequirement": testCase.name} + if got := evalCondition(node, nil, evidence); got != testCase.want { + t.Errorf("evidence-present(%s) = %v, want %v", testCase.name, got, testCase.want) + } + } +} + +// Ordered comparison at the condition level: decimal strings decide, JSON +// numbers and non-grammar strings degrade to unknown, and undeterminable +// number equality degrades to unknown rather than false. +func TestOrderedAndUndeterminableConditions(t *testing.T) { + facts := map[string]any{ + "amount": "150", + "count": json.Number("150"), + "huge": json.Number("1e999999999"), + } + cases := []struct { + name string + node map[string]any + want tri + }{ + {"decimal-string-greater", map[string]any{"op": "fact", "path": "/amount", "operator": "greater-than", "value": "100"}, triTrue}, + {"decimal-string-not-less", map[string]any{"op": "fact", "path": "/amount", "operator": "less-than", "value": "100"}, triFalse}, + {"json-number-ordered-unknown", map[string]any{"op": "fact", "path": "/count", "operator": "greater-than", "value": "100"}, triUnknown}, + {"huge-number-equals-unknown", map[string]any{"op": "fact", "path": "/huge", "operator": "equals", "value": json.Number("2e999999999")}, triUnknown}, + {"huge-number-not-equals-unknown", map[string]any{"op": "fact", "path": "/huge", "operator": "not-equals", "value": json.Number("2e999999999")}, triUnknown}, + {"huge-identical-token-equal", map[string]any{"op": "fact", "path": "/huge", "operator": "equals", "value": json.Number("1e999999999")}, triTrue}, + {"in-with-undeterminable-member", map[string]any{"op": "fact", "path": "/huge", "operator": "in", "value": []any{json.Number("2e999999999"), "x"}}, triUnknown}, + } + for _, testCase := range cases { + if got := evalCondition(testCase.node, facts, map[string]tri{}); got != testCase.want { + t.Errorf("%s = %v, want %v", testCase.name, got, testCase.want) + } + } +} + +// §8.1: the configured target is requested only when a retained reason appears +// in escalation.triggers — except a direct exception escalation, which is a +// request regardless of the trigger list. +func TestHandoffTriggerMatching(t *testing.T) { + base := func() map[string]any { + return map[string]any{ + "outcomes": []any{map[string]any{"id": "a"}, map[string]any{"id": "b"}}, + "rules": []any{ + map[string]any{"id": "r", "when": map[string]any{"op": "literal", "value": false}, "outcome": "a", "onUnknown": "ignore"}, + }, + } + } + + t.Run("reason-not-in-triggers-means-no-handoff", func(t *testing.T) { + pack := base() // no fallback -> unresolved{no-match} + pack["escalation"] = map[string]any{ + "triggers": []any{"conflict"}, + "target": map[string]any{"kind": "human-role", "name": "Reviewer"}, + } + disposition, _ := resolve(pack, map[string]any{}, map[string]tri{}) + if !reflect.DeepEqual(disposition.Reasons, []string{"no-match"}) { + t.Fatalf("expected no-match: %+v", disposition) + } + if disposition.Handoff.State != "none" { + t.Fatalf("no-match is not in the trigger list, so no handoff is configured: %+v", disposition.Handoff) + } + }) + + t.Run("matching-trigger-echoes-target", func(t *testing.T) { + pack := base() + pack["escalation"] = map[string]any{ + "triggers": []any{"no-match"}, + "target": map[string]any{"kind": "team", "name": "Intake board"}, + } + disposition, _ := resolve(pack, map[string]any{}, map[string]tri{}) + if disposition.Handoff.State != "requested" || disposition.Handoff.Target == nil || + disposition.Handoff.Target.Kind != "team" || disposition.Handoff.Target.Name != "Intake board" { + t.Fatalf("a matching trigger must echo the declared target exactly: %+v", disposition.Handoff) + } + }) + + t.Run("direct-escalation-ignores-trigger-list", func(t *testing.T) { + pack := base() + pack["exceptions"] = []any{ + map[string]any{"id": "x", "when": map[string]any{"op": "literal", "value": true}, "effect": "escalate", "onUnknown": "ignore"}, + } + pack["escalation"] = map[string]any{ + "triggers": []any{"conflict"}, // exception-escalation is deliberately absent + "target": map[string]any{"kind": "human-role", "name": "Escalation desk"}, + } + disposition, _ := resolve(pack, map[string]any{}, map[string]tri{}) + if disposition.Handoff.State != "requested" || disposition.Handoff.Target == nil || disposition.Handoff.Target.Name != "Escalation desk" { + t.Fatalf("a direct request uses the configured target regardless of the trigger list: %+v", disposition.Handoff) + } + }) + + t.Run("escalate-takes-precedence-over-force", func(t *testing.T) { + pack := base() + pack["exceptions"] = []any{ + map[string]any{"id": "force", "when": map[string]any{"op": "literal", "value": true}, "effect": "force-outcome", "outcome": "b", "onUnknown": "ignore"}, + map[string]any{"id": "esc", "when": map[string]any{"op": "literal", "value": true}, "effect": "escalate", "onUnknown": "ignore"}, + } + disposition, _ := resolve(pack, map[string]any{}, map[string]tri{}) + if disposition.Kind != "unresolved" || !reflect.DeepEqual(disposition.Reasons, []string{"exception-escalation"}) { + t.Fatalf("a direct escalation takes precedence over a compatible forced outcome: %+v", disposition) + } + }) +} diff --git a/internal/evaluation/engine.go b/internal/evaluation/engine.go new file mode 100644 index 0000000..6e44e7c --- /dev/null +++ b/internal/evaluation/engine.go @@ -0,0 +1,236 @@ +// Package evaluation implements the JPS Core §§7–8 experiment, as pinned by +// the specification's RFC 0006 (Draft), behind an explicitly experimental +// surface (ADR-0007). +// +// It claims no evaluator conformance — JPS 0.1.0-draft §3.4 forbids such +// claims outright — and may change or be removed without compatibility +// promise. Evaluation is a pure function of its inputs: a conformant pack, +// one facts document, the tri-state availability of declared evidence, and +// the supported-extension set. Errors are not dispositions: a non-conformant +// pack, a malformed input, or an undeclared evidence key is refused. +package evaluation + +import ( + "fmt" + "sort" + "strings" + + "github.com/Judgment-Pack/judgment-pack-runtime/internal/artifacts" + "github.com/Judgment-Pack/judgment-pack-runtime/internal/carrier" + "github.com/Judgment-Pack/judgment-pack-runtime/internal/display" + "github.com/Judgment-Pack/judgment-pack-runtime/internal/result" + "github.com/Judgment-Pack/judgment-pack-runtime/internal/validation" +) + +// Failure reports why an evaluation could not run. It is an operational +// refusal, never a disposition. +type Failure struct { + Code string + Message string + ExitCode int +} + +// Engine evaluates conformant packs. It wraps the validation engine so a pack +// is always checked to full document conformance before a single condition is +// interpreted. +type Engine struct { + validator *validation.Engine +} + +func NewEngine(validator *validation.Engine) *Engine { + return &Engine{validator: validator} +} + +// Evaluate validates the pack (with the caller's supported extensions), then +// applies the §§7–8 experiment to the facts and evidence inputs. command names +// the reporting surface, exactly as in the describe package. +func (e *Engine) Evaluate(pack, facts, evidence []byte, supported []string, command string) (result.Evaluation, *Failure) { + // One byte-limit boundary for every surface: the MCP wire must refuse the + // same oversized input the CLI's bounded reads refuse. + for name, data := range map[string][]byte{"pack": pack, "facts": facts, "evidence": evidence} { + if int64(len(data)) > carrier.HardMaxBytes { + return result.Evaluation{}, &Failure{ + Code: "JPS-RESOURCE-INPUT-BYTE-LIMIT", + Message: fmt.Sprintf("The %s input exceeds the hard %d-byte limit.", name, carrier.HardMaxBytes), + ExitCode: result.ExitIO, + } + } + } + validated, operational := e.validator.Validate(pack, validation.Options{ + Through: "semantic", + SupportedExtensions: supported, + Limits: carrier.DefaultLimits(), + }) + if operational != nil { + return result.Evaluation{}, &Failure{Code: operational.Code, Message: operational.Message, ExitCode: operational.ExitCode} + } + if validated.Status != "valid" { + return result.Evaluation{}, packNotConformant(validated) + } + + packDocument, failure := decodeInput(pack, "pack") + if failure != nil { + return result.Evaluation{}, failure + } + packRoot, ok := packDocument.(map[string]any) + if !ok { + return result.Evaluation{}, &Failure{Code: "JPS-EVALUATION-INTERNAL", Message: "The validated pack did not decode to a JSON object.", ExitCode: result.ExitInternal} + } + + factsDocument, failure := decodeInput(facts, "facts") + if failure != nil { + return result.Evaluation{}, failure + } + + presence, failure := decodeEvidence(evidence, packRoot) + if failure != nil { + return result.Evaluation{}, failure + } + + disposition, trace := resolve(packRoot, factsDocument, presence) + if trace == nil { + trace = []result.TraceEntry{} + } + evaluation := result.Evaluation{ + OutputVersion: result.OutputVersion, + Tool: result.CurrentTool(), + Command: command, + Status: "evaluated", + Experimental: true, + ConformanceClaim: result.EvaluationClaim, + SpecVersion: validated.SpecVersion, + Disposition: disposition, + Trace: trace, + } + if set, err := artifacts.Load(validated.SpecVersion); err == nil { + evaluation.Artifact = &result.Artifact{ + SpecVersion: validated.SpecVersion, + BundleDigest: set.Lock().BundleDigest.Value, + Provenance: set.Lock().Source.Kind, + } + } + return evaluation, nil +} + +// packNotConformant refuses a pack that did not reach full document +// conformance, naming the first diagnostic so the refusal is self-sufficient. +func packNotConformant(validated result.Validation) *Failure { + exitCode := result.ExitInvalid + if validated.Status == "unsupported" { + exitCode = result.ExitUnsupported + } + detail := "" + if len(validated.Diagnostics) > 0 { + first := validated.Diagnostics[0] + location := first.InstancePath + if location == "" { + location = "" + } + detail = fmt.Sprintf(" First diagnostic: %s at %s: %s", first.Code, location, first.Message) + } + return &Failure{ + Code: "JPS-EVALUATION-PACK-NOT-CONFORMANT", + Message: fmt.Sprintf("The pack is not evaluated because its document conformance status is %q; evaluation requires a fully conformant pack. Run spec validate for the complete report.%s", validated.Status, detail), + ExitCode: exitCode, + } +} + +// decodeInput decodes one JSON input with the same strict carrier rules the +// pack itself gets: duplicate member names, depth, and size limits are all +// enforced, and the failure names the offending input. +func decodeInput(data []byte, name string) (any, *Failure) { + if len(data) == 0 { + return nil, &Failure{ + Code: "JPS-EVALUATION-INPUT-MISSING", + Message: fmt.Sprintf("The %s input is required and was empty.", name), + ExitCode: result.ExitInvocation, + } + } + document, failure := carrier.Decode(data, carrier.DefaultLimits()) + if failure != nil { + return nil, decodeFailure(name, failure) + } + return document, nil +} + +// decodeFailure maps a carrier failure on an evaluation input: a resource +// limit keeps its own JPS-RESOURCE-* code and the IO exit class, mirroring +// validation; anything else is a caller input error. +func decodeFailure(name string, failure *carrier.Failure) *Failure { + if failure.Resource { + return &Failure{ + Code: failure.Diagnostic.Code, + Message: fmt.Sprintf("The %s input exceeds a resource limit: %s", name, failure.Diagnostic.Message), + ExitCode: result.ExitIO, + } + } + return &Failure{ + Code: "JPS-EVALUATION-INPUT-JSON", + Message: fmt.Sprintf("The %s input is not acceptable JSON: %s", name, failure.Diagnostic.Message), + ExitCode: result.ExitInvocation, + } +} + +// decodeEvidence decodes the optional tri-state evidence-availability input: +// a JSON object keyed by declared evidenceRequirements[].id with values +// "present", "absent", or "unknown". An omitted key means unknown; an +// undeclared key or any other value is an input error. Every declared +// requirement is represented in the returned map. +func decodeEvidence(data []byte, packRoot map[string]any) (map[string]tri, *Failure) { + declared := map[string]tri{} + names := []string{} + for _, entry := range asArray(packRoot["evidenceRequirements"]) { + if requirement, ok := entry.(map[string]any); ok { + if id, ok := requirement["id"].(string); ok { + declared[id] = triUnknown + names = append(names, id) + } + } + } + sort.Strings(names) + if len(data) == 0 { + return declared, nil + } + document, failure := carrier.Decode(data, carrier.DefaultLimits()) + if failure != nil { + return nil, decodeFailure("evidence", failure) + } + object, ok := document.(map[string]any) + if !ok { + return nil, &Failure{ + Code: "JPS-EVALUATION-EVIDENCE-SHAPE", + Message: "The evidence input must be a JSON object mapping declared evidence-requirement ids to \"present\", \"absent\", or \"unknown\".", + ExitCode: result.ExitInvocation, + } + } + for key, value := range object { + if _, known := declared[key]; !known { + detail := "The pack declares no evidence requirements." + if len(names) > 0 { + detail = "Declared ids: " + strings.Join(names, ", ") + "." + } + return nil, &Failure{ + Code: "JPS-EVALUATION-EVIDENCE-KEY", + Message: fmt.Sprintf("Evidence key %q does not name a declared evidence requirement. %s", display.Sanitize(key), detail), + ExitCode: result.ExitInvocation, + } + } + state, ok := value.(string) + if !ok || (state != "present" && state != "absent" && state != "unknown") { + return nil, &Failure{ + Code: "JPS-EVALUATION-EVIDENCE-VALUE", + Message: fmt.Sprintf("Evidence key %q must map to \"present\", \"absent\", or \"unknown\".", display.Sanitize(key)), + ExitCode: result.ExitInvocation, + } + } + switch state { + case "present": + declared[key] = triTrue + case "absent": + declared[key] = triFalse + default: + declared[key] = triUnknown + } + } + return declared, nil +} diff --git a/internal/evaluation/engine_test.go b/internal/evaluation/engine_test.go new file mode 100644 index 0000000..12a95ee --- /dev/null +++ b/internal/evaluation/engine_test.go @@ -0,0 +1,428 @@ +package evaluation + +import ( + "encoding/json" + "os" + "path/filepath" + "reflect" + "strings" + "testing" + + "github.com/Judgment-Pack/judgment-pack-runtime/internal/artifacts" + "github.com/Judgment-Pack/judgment-pack-runtime/internal/carrier" + "github.com/Judgment-Pack/judgment-pack-runtime/internal/result" + "github.com/Judgment-Pack/judgment-pack-runtime/internal/validation" +) + +func newTestEngine(t *testing.T) *Engine { + t.Helper() + validator, err := validation.NewEngine() + if err != nil { + t.Fatal(err) + } + return NewEngine(validator) +} + +func intakePack(t *testing.T) []byte { + t.Helper() + data, err := os.ReadFile(filepath.Join("testdata", "data-request-intake-triage.json")) + if err != nil { + t.Fatal(err) + } + return data +} + +// factsJSON builds the /request facts document. An empty value omits the member. +func factsJSON(t *testing.T, requestType, completeness, appropriateness string, embargo *bool) []byte { + t.Helper() + request := map[string]any{} + if requestType != "" { + request["type"] = requestType + } + if completeness != "" { + request["completeness"] = completeness + } + if appropriateness != "" { + request["appropriateness"] = appropriateness + } + if embargo != nil { + request["embargoedInformationToUnauthorizedRecipients"] = *embargo + } + data, err := json.Marshal(map[string]any{"request": request}) + if err != nil { + t.Fatal(err) + } + return data +} + +func boolPtr(v bool) *bool { return &v } + +// TestRFC0006AppendixInstances executes the nine walked instances committed in +// spec RFC 0006's appendix, under its pinned semantics. These rows are the +// seed of the proposed evaluation corpus; every expected disposition was +// independently verified during the RFC's adversarial review. +func TestRFC0006AppendixInstances(t *testing.T) { + engine := newTestEngine(t) + pack := intakePack(t) + no := boolPtr(false) + yes := boolPtr(true) + + cases := []struct { + name string + facts []byte + evidence string + wantKind string + wantOutcome string + wantReasons []string + wantHandoff string + }{ + { + name: "1-not-applicable-type", + facts: factsJSON(t, "dataset-deletion", "", "", nil), + evidence: `{"intake-form":"present","sponsor-endorsement":"present"}`, + wantKind: "not-applicable", + wantReasons: []string{"not-applicable"}, + wantHandoff: "requested", + }, + { + name: "2-applicability-unknown", + facts: factsJSON(t, "", "complete", "pass", no), + evidence: `{"intake-form":"present","sponsor-endorsement":"present"}`, + wantKind: "unresolved", + wantReasons: []string{"unknown"}, + wantHandoff: "requested", + }, + { + name: "3-decline", + facts: factsJSON(t, "one-time-extract", "complete", "hard-fail", no), + evidence: `{"intake-form":"present","sponsor-endorsement":"present"}`, + wantKind: "outcome", + wantOutcome: "decline-redirect", + wantHandoff: "none", + }, + { + name: "4-clarify-incomplete", + facts: factsJSON(t, "data-access", "incomplete", "pass", no), + evidence: `{"intake-form":"present","sponsor-endorsement":"present"}`, + wantKind: "outcome", + wantOutcome: "clarify-return", + wantHandoff: "none", + }, + { + name: "5-clarify-pending", + facts: factsJSON(t, "data-access", "complete", "pending", no), + evidence: `{"intake-form":"present","sponsor-endorsement":"present"}`, + wantKind: "outcome", + wantOutcome: "clarify-return", + wantHandoff: "none", + }, + { + name: "6-proceed", + facts: factsJSON(t, "new-data-pipeline", "complete", "pass", no), + evidence: `{"intake-form":"present","sponsor-endorsement":"present"}`, + wantKind: "outcome", + wantOutcome: "proceed", + wantHandoff: "none", + }, + { + name: "7a-evidence-unknown", + facts: factsJSON(t, "dataset-onboarding", "complete", "pass", no), + evidence: `{"intake-form":"present","sponsor-endorsement":"unknown"}`, + wantKind: "unresolved", + wantReasons: []string{"unknown"}, + wantHandoff: "requested", + }, + { + name: "7b-evidence-absent", + facts: factsJSON(t, "dataset-onboarding", "complete", "pass", no), + evidence: `{"intake-form":"present","sponsor-endorsement":"absent"}`, + wantKind: "unresolved", + wantReasons: []string{"missing-required-evidence"}, + wantHandoff: "requested", + }, + { + name: "8-forced-outcome", + facts: factsJSON(t, "reporting-feed", "complete", "pass", yes), + evidence: `{"intake-form":"present","sponsor-endorsement":"present"}`, + wantKind: "outcome", + wantOutcome: "decline-redirect", + wantHandoff: "none", + }, + { + name: "9-conflict", + facts: factsJSON(t, "pipeline-change", "incomplete", "hard-fail", no), + evidence: `{"intake-form":"present","sponsor-endorsement":"present"}`, + wantKind: "unresolved", + wantReasons: []string{"conflict"}, + wantHandoff: "requested", + }, + } + + for _, testCase := range cases { + t.Run(testCase.name, func(t *testing.T) { + output, failure := engine.Evaluate(pack, testCase.facts, []byte(testCase.evidence), nil, "test") + if failure != nil { + t.Fatalf("evaluation failed: %s: %s", failure.Code, failure.Message) + } + if !output.Experimental || output.ConformanceClaim != result.EvaluationClaim { + t.Fatalf("evaluation must be labeled experimental with no claim: %+v", output) + } + disposition := output.Disposition + if disposition.Kind != testCase.wantKind { + t.Fatalf("kind = %q, want %q (%+v)", disposition.Kind, testCase.wantKind, disposition) + } + if disposition.OutcomeID != testCase.wantOutcome { + t.Fatalf("outcomeId = %q, want %q", disposition.OutcomeID, testCase.wantOutcome) + } + wantReasons := testCase.wantReasons + if wantReasons == nil { + wantReasons = []string{} + } + if !reflect.DeepEqual(disposition.Reasons, wantReasons) { + t.Fatalf("reasons = %v, want %v", disposition.Reasons, wantReasons) + } + if disposition.Handoff.State != testCase.wantHandoff { + t.Fatalf("handoff = %q, want %q", disposition.Handoff.State, testCase.wantHandoff) + } + if testCase.wantHandoff == "requested" { + target := disposition.Handoff.Target + if target == nil || target.Kind != "human-role" || target.Name != "Intake reviewer" { + t.Fatalf("a requested handoff must echo the declared target exactly: %+v", disposition.Handoff) + } + } + if testCase.wantKind == "outcome" && len(disposition.Reasons) != 0 { + t.Fatalf("an outcome result retains no reasons: %+v", disposition) + } + }) + } +} + +// A forced outcome must skip normal rules entirely: the proceed rule would be +// true for instance 8's facts, and the trace must show it was never evaluated. +func TestForcedOutcomeSkipsRules(t *testing.T) { + engine := newTestEngine(t) + output, failure := engine.Evaluate( + intakePack(t), + factsJSON(t, "reporting-feed", "complete", "pass", boolPtr(true)), + []byte(`{"intake-form":"present","sponsor-endorsement":"present"}`), + nil, "test") + if failure != nil { + t.Fatal(failure.Message) + } + skipped := 0 + for _, entry := range output.Trace { + if entry.Stage == "rule" { + if !entry.Skipped { + t.Fatalf("rule %s must be skipped under a forced outcome: %+v", entry.ID, entry) + } + skipped++ + } + } + if skipped != 3 { + t.Fatalf("all three rules should appear skipped in the trace, got %d", skipped) + } +} + +// Errors are not dispositions (ADR-0007): a non-conformant pack is refused +// with a self-sufficient message, never evaluated. +func TestNonConformantPackIsRefused(t *testing.T) { + engine := newTestEngine(t) + var document map[string]any + if err := json.Unmarshal(intakePack(t), &document); err != nil { + t.Fatal(err) + } + document["outcomes"] = document["outcomes"].([]any)[:1] // break references + broken, err := json.Marshal(document) + if err != nil { + t.Fatal(err) + } + _, failure := engine.Evaluate(broken, []byte(`{}`), nil, nil, "test") + if failure == nil { + t.Fatal("a non-conformant pack must be refused") + } + if failure.Code != "JPS-EVALUATION-PACK-NOT-CONFORMANT" || failure.ExitCode != result.ExitInvalid { + t.Fatalf("failure = %+v", failure) + } + if !strings.Contains(failure.Message, "JPS-") { + t.Fatalf("the refusal must name the first diagnostic: %q", failure.Message) + } +} + +func TestEvidenceInputErrors(t *testing.T) { + engine := newTestEngine(t) + pack := intakePack(t) + facts := factsJSON(t, "data-access", "complete", "pass", boolPtr(false)) + + _, failure := engine.Evaluate(pack, facts, []byte(`{"no-such-requirement":"present"}`), nil, "test") + if failure == nil || failure.Code != "JPS-EVALUATION-EVIDENCE-KEY" { + t.Fatalf("an undeclared evidence key must be an input error: %+v", failure) + } + if !strings.Contains(failure.Message, "intake-form") { + t.Fatalf("the error must list the declared ids: %q", failure.Message) + } + + _, failure = engine.Evaluate(pack, facts, []byte(`{"intake-form":"maybe"}`), nil, "test") + if failure == nil || failure.Code != "JPS-EVALUATION-EVIDENCE-VALUE" { + t.Fatalf("a bad evidence value must be an input error: %+v", failure) + } + + _, failure = engine.Evaluate(pack, nil, nil, nil, "test") + if failure == nil || failure.Code != "JPS-EVALUATION-INPUT-MISSING" { + t.Fatalf("an empty facts input must be refused: %+v", failure) + } +} + +// Omitting the evidence input entirely means every requirement is unknown, so +// the proceed rule cannot fire and RFC 0006's step-2 choice records unknown. +func TestOmittedEvidenceIsUnknown(t *testing.T) { + engine := newTestEngine(t) + output, failure := engine.Evaluate( + intakePack(t), + factsJSON(t, "data-access", "complete", "pass", boolPtr(false)), + nil, nil, "test") + if failure != nil { + t.Fatal(failure.Message) + } + if output.Disposition.Kind != "unresolved" || !reflect.DeepEqual(output.Disposition.Reasons, []string{"unknown"}) { + t.Fatalf("omitted evidence must resolve as unknown: %+v", output.Disposition) + } +} + +// RFC 0006's step-2 clause "unknown iff any presence is unknown and none is +// false": when one required requirement is absent and another unknown, only +// missing-required-evidence is recorded. +func TestStepTwoAbsentBeatsUnknown(t *testing.T) { + engine := newTestEngine(t) + output, failure := engine.Evaluate( + intakePack(t), + factsJSON(t, "data-access", "complete", "pass", boolPtr(false)), + []byte(`{"intake-form":"absent","sponsor-endorsement":"unknown"}`), + nil, "test") + if failure != nil { + t.Fatal(failure.Message) + } + if !reflect.DeepEqual(output.Disposition.Reasons, []string{"missing-required-evidence"}) { + t.Fatalf("absent evidence must record only missing-required-evidence: %+v", output.Disposition) + } +} + +// An exception whose condition is unknown with onUnknown: escalate records +// reason unknown and blocks resolution before any rule is evaluated; the +// unknown stays visible in the exception's trace entry. +func TestUnknownExceptionEscalates(t *testing.T) { + engine := newTestEngine(t) + output, failure := engine.Evaluate( + intakePack(t), + factsJSON(t, "data-access", "complete", "pass", nil), // embargo fact omitted + []byte(`{"intake-form":"present","sponsor-endorsement":"present"}`), + nil, "test") + if failure != nil { + t.Fatal(failure.Message) + } + if output.Disposition.Kind != "unresolved" || !reflect.DeepEqual(output.Disposition.Reasons, []string{"unknown"}) { + t.Fatalf("an unknown escalating exception must block resolution: %+v", output.Disposition) + } + sawException := false + for _, entry := range output.Trace { + if entry.Stage == "exception" { + if entry.Condition != "unknown" || entry.OnUnknown != "escalate" { + t.Fatalf("the exception trace must show the unknown and its policy: %+v", entry) + } + sawException = true + } + if entry.Stage == "rule" && entry.Condition != "" && !entry.Skipped { + t.Fatalf("no rule may be evaluated once step 5 blocks: %+v", entry) + } + } + if !sawException { + t.Fatalf("exception must appear in the trace: %+v", output.Trace) + } +} + +// Missing required evidence blocks a forced outcome: the exception fires, but +// step 5 produces unresolved after all exception effects were inspected. +func TestMissingEvidenceBlocksForcedOutcome(t *testing.T) { + engine := newTestEngine(t) + output, failure := engine.Evaluate( + intakePack(t), + factsJSON(t, "reporting-feed", "complete", "pass", boolPtr(true)), // embargo true -> force + []byte(`{"intake-form":"present","sponsor-endorsement":"absent"}`), + nil, "test") + if failure != nil { + t.Fatal(failure.Message) + } + if output.Disposition.Kind != "unresolved" || !reflect.DeepEqual(output.Disposition.Reasons, []string{"missing-required-evidence"}) { + t.Fatalf("missing required evidence must block the forced outcome: %+v", output.Disposition) + } + forcedSeen := false + for _, entry := range output.Trace { + if entry.Stage == "exception" && entry.Effect == "force-outcome" { + forcedSeen = true + } + } + if !forcedSeen { + t.Fatalf("the inspected forced effect must stay visible in the trace: %+v", output.Trace) + } +} + +// A pack declaring a required extension is refused without the capability and +// evaluated with it — errors are not dispositions, and the capability comes +// from the caller's supported set. +func TestRequiredExtensionCapabilityGate(t *testing.T) { + set, err := artifacts.Load(artifacts.DraftVersion) + if err != nil { + t.Fatal(err) + } + pack, err := set.Case("valid/required-extension-supported.json") + if err != nil { + t.Fatal(err) + } + engine := newTestEngine(t) + + _, failure := engine.Evaluate(pack, []byte(`{}`), nil, nil, "test") + if failure == nil || failure.Code != "JPS-EVALUATION-PACK-NOT-CONFORMANT" || failure.ExitCode != result.ExitUnsupported { + t.Fatalf("an unsupported required extension must refuse evaluation: %+v", failure) + } + + output, failure := engine.Evaluate(pack, []byte(`{}`), nil, []string{"com.example.review-policy"}, "test") + if failure != nil { + t.Fatalf("a supported required extension must evaluate: %s", failure.Message) + } + if output.Disposition.Kind == "" { + t.Fatalf("a disposition must be produced: %+v", output) + } +} + +// Strict carrier rules apply to evaluation inputs: duplicate member names are +// refused in both the evidence and facts documents. +func TestDuplicateMembersAreInputErrors(t *testing.T) { + engine := newTestEngine(t) + pack := intakePack(t) + facts := factsJSON(t, "data-access", "complete", "pass", boolPtr(false)) + + _, failure := engine.Evaluate(pack, facts, []byte(`{"intake-form":"present","intake-form":"present"}`), nil, "test") + if failure == nil || failure.Code != "JPS-EVALUATION-INPUT-JSON" { + t.Fatalf("duplicate evidence keys must be an input error: %+v", failure) + } + + _, failure = engine.Evaluate(pack, []byte(`{"request":{},"request":{}}`), nil, nil, "test") + if failure == nil || failure.Code != "JPS-EVALUATION-INPUT-JSON" { + t.Fatalf("duplicate facts members must be an input error: %+v", failure) + } + + _, failure = engine.Evaluate(pack, facts, []byte(`["present"]`), nil, "test") + if failure == nil || failure.Code != "JPS-EVALUATION-EVIDENCE-SHAPE" { + t.Fatalf("a non-object evidence document must be a shape error: %+v", failure) + } +} + +// The byte-limit boundary lives in the engine so the MCP wire refuses the same +// oversized input the CLI's bounded reads refuse. +func TestOversizedInputsAreRefused(t *testing.T) { + engine := newTestEngine(t) + huge := make([]byte, carrier.HardMaxBytes+1) + _, failure := engine.Evaluate(intakePack(t), huge, nil, nil, "test") + if failure == nil || failure.Code != "JPS-RESOURCE-INPUT-BYTE-LIMIT" || failure.ExitCode != result.ExitIO { + t.Fatalf("an oversized facts input must be refused with the byte-limit code: %+v", failure) + } +} diff --git a/internal/evaluation/resolve.go b/internal/evaluation/resolve.go new file mode 100644 index 0000000..943bf4b --- /dev/null +++ b/internal/evaluation/resolve.go @@ -0,0 +1,240 @@ +package evaluation + +import ( + "sort" + + "github.com/Judgment-Pack/judgment-pack-runtime/internal/result" +) + +// reason vocabulary of the §8 experiment. exception-escalation is a direct +// request rather than a trigger-selected one. +const ( + reasonNotApplicable = "not-applicable" + reasonMissingEvidence = "missing-required-evidence" + reasonUnknown = "unknown" + reasonConflict = "conflict" + reasonNoMatch = "no-match" + reasonExceptionEscalation = "exception-escalation" +) + +// resolver walks §8 over one conformant pack. It is a pure function of its +// inputs: the decoded pack, the decoded facts document, and the tri-state +// presence of every declared evidence requirement. +type resolver struct { + pack map[string]any + facts any + evidence map[string]tri + + reasons map[string]bool + directEscalation bool + trace []result.TraceEntry +} + +// resolve produces the RFC 0006 disposition and the informative trace. +func resolve(pack map[string]any, facts any, evidence map[string]tri) (result.Disposition, []result.TraceEntry) { + r := &resolver{pack: pack, facts: facts, evidence: evidence, reasons: map[string]bool{}} + + // Step 1: omitted applicability is the literal value true. False is a + // terminal not-applicable result; unknown produces unresolved and stops. + applicability := triTrue + if condition, present := pack["applicability"]; present { + applicability = evalCondition(condition, facts, evidence) + } + switch applicability { + case triFalse: + r.reasons[reasonNotApplicable] = true + return r.disposition("not-applicable", ""), r.trace + case triUnknown: + r.reasons[reasonUnknown] = true + return r.disposition("unresolved", ""), r.trace + } + + // Step 2, as pinned by RFC 0006: missing-required-evidence iff any + // required requirement's presence is false; unknown iff any required + // requirement's presence is unknown and none is false. + requiredFalse, requiredUnknown := false, false + for _, entry := range asArray(pack["evidenceRequirements"]) { + requirement, ok := entry.(map[string]any) + if !ok { + continue + } + if required, _ := requirement["required"].(bool); !required { + continue + } + id, _ := requirement["id"].(string) + switch r.evidence[id] { + case triFalse: + requiredFalse = true + case triUnknown: + requiredUnknown = true + } + } + if requiredFalse { + r.reasons[reasonMissingEvidence] = true + } else if requiredUnknown { + r.reasons[reasonUnknown] = true + } + + // Steps 3-4: evaluate every exception and combine true effects. An + // unknown exception with onUnknown: escalate records reason unknown; with + // onUnknown: ignore it contributes nothing but stays visible in the trace. + suppressed := map[string]bool{} + forced := map[string]bool{} + for _, entry := range asArray(pack["exceptions"]) { + exception, ok := entry.(map[string]any) + if !ok { + continue + } + id, _ := exception["id"].(string) + effect, _ := exception["effect"].(string) + verdict := evalCondition(exception["when"], facts, evidence) + entry := result.TraceEntry{Stage: "exception", ID: id, Condition: verdict.String()} + switch verdict { + case triUnknown: + onUnknown, _ := exception["onUnknown"].(string) + entry.OnUnknown = onUnknown + if onUnknown == "escalate" { + r.reasons[reasonUnknown] = true + } + case triTrue: + entry.Effect = effect + switch effect { + case "suppress-rule": + if target, ok := exception["targetRule"].(string); ok { + suppressed[target] = true + } + case "force-outcome": + if outcome, ok := exception["outcome"].(string); ok { + forced[outcome] = true + entry.Outcome = outcome + } + case "escalate": + r.directEscalation = true + r.reasons[reasonExceptionEscalation] = true + } + } + r.trace = append(r.trace, entry) + } + + // Step 5: incompatible forced outcomes are a conflict. Any blocking state + // discovered so far — after all exceptions were inspected — produces + // unresolved with every retained reason. A direct escalation takes + // precedence over suppression and forced outcomes. + if len(forced) > 1 { + r.reasons[reasonConflict] = true + } + if len(r.reasons) > 0 { + return r.disposition("unresolved", ""), r.trace + } + + // Step 6: one compatible forced outcome, no blocking state: produce it + // without evaluating normal rules. + if len(forced) == 1 { + for outcome := range forced { + for _, entry := range asArray(pack["rules"]) { + if rule, ok := entry.(map[string]any); ok { + id, _ := rule["id"].(string) + r.trace = append(r.trace, result.TraceEntry{Stage: "rule", ID: id, Condition: "not-evaluated", Skipped: true}) + } + } + return r.disposition("outcome", outcome), r.trace + } + } + + // Steps 6-8: remove suppressed rules, evaluate the rest, and collect + // candidate outcomes. An unknown rule with onUnknown: escalate records + // reason unknown and blocks both a candidate outcome and the fallback. + candidates := map[string]bool{} + for _, entry := range asArray(pack["rules"]) { + rule, ok := entry.(map[string]any) + if !ok { + continue + } + id, _ := rule["id"].(string) + if suppressed[id] { + r.trace = append(r.trace, result.TraceEntry{Stage: "rule", ID: id, Condition: "not-evaluated", Suppressed: true}) + continue + } + verdict := evalCondition(rule["when"], facts, evidence) + entry := result.TraceEntry{Stage: "rule", ID: id, Condition: verdict.String()} + switch verdict { + case triTrue: + if outcome, ok := rule["outcome"].(string); ok { + candidates[outcome] = true + entry.Outcome = outcome + } + case triUnknown: + onUnknown, _ := rule["onUnknown"].(string) + entry.OnUnknown = onUnknown + if onUnknown == "escalate" { + r.reasons[reasonUnknown] = true + } + } + r.trace = append(r.trace, entry) + } + if len(candidates) > 1 { + r.reasons[reasonConflict] = true + } + if len(r.reasons) > 0 { + return r.disposition("unresolved", ""), r.trace + } + + // Step 9: one distinct outcome and no blocking reason: produce it. + if len(candidates) == 1 { + for outcome := range candidates { + return r.disposition("outcome", outcome), r.trace + } + } + + // Step 10: no true rule contributed an outcome. Use the fallback when + // declared; otherwise unresolved with reason no-match. + if fallback, ok := r.pack["fallbackOutcome"].(string); ok { + return r.disposition("outcome", fallback), r.trace + } + r.reasons[reasonNoMatch] = true + return r.disposition("unresolved", ""), r.trace +} + +// disposition assembles the RFC 0006 result: sorted deduplicated reasons and +// the §8.1 handoff axis. An outcome result retains no reasons and requests no +// handoff. +func (r *resolver) disposition(kind, outcomeID string) result.Disposition { + disposition := result.Disposition{Kind: kind, OutcomeID: outcomeID, Reasons: []string{}, Handoff: result.Handoff{State: "none"}} + if kind == "outcome" { + return disposition + } + for reason := range r.reasons { + disposition.Reasons = append(disposition.Reasons, reason) + } + sort.Strings(disposition.Reasons) + + escalation, _ := r.pack["escalation"].(map[string]any) + triggered := false + if escalation != nil { + for _, trigger := range asArray(escalation["triggers"]) { + if name, ok := trigger.(string); ok && r.reasons[name] { + triggered = true + break + } + } + } + // §8.1: a direct exception escalation is a request regardless of the + // trigger list, using the configured target when one exists; without an + // escalation object it remains a request with no Core-defined destination. + if r.directEscalation || triggered { + disposition.Handoff.State = "requested" + if escalation != nil { + if target, ok := escalation["target"].(map[string]any); ok { + kind, _ := target["kind"].(string) + name, _ := target["name"].(string) + disposition.Handoff.Target = &result.HandoffTarget{Kind: kind, Name: name} + } + } + } + return disposition +} + +func asArray(value any) []any { + items, _ := value.([]any) + return items +} diff --git a/internal/evaluation/testdata/README.md b/internal/evaluation/testdata/README.md new file mode 100644 index 0000000..0c4f740 --- /dev/null +++ b/internal/evaluation/testdata/README.md @@ -0,0 +1,6 @@ +# Test fixtures + +data-request-intake-triage.json is copied verbatim from +Judgment-Pack/judgment-pack-spec examples/ (commit 839488c). It is the pack +that spec RFC 0006's appendix instances walk; the evaluation tests execute +those instances against this engine. diff --git a/internal/evaluation/testdata/data-request-intake-triage.json b/internal/evaluation/testdata/data-request-intake-triage.json new file mode 100644 index 0000000..116566d --- /dev/null +++ b/internal/evaluation/testdata/data-request-intake-triage.json @@ -0,0 +1,190 @@ +{ + "specVersion": "0.1.0-draft", + "id": "https://example.invalid/judgment-packs/data-request-intake-triage", + "version": "0.1.0", + "title": "Synthetic data request intake triage", + "description": "Invented intake-triage policy for a fictional data platform team: route an incoming data request to proceed, clarify/return, or decline/redirect once completeness and appropriateness have been assessed. Synthetic content for specification testing; it authorizes nothing.", + "decision": { + "intent": "Triage an incoming data request into exactly one intake outcome once the completeness of the submission and the appropriateness of the request have been assessed.", + "question": "Should this data request proceed, be returned for clarification, or be declined and redirected?" + }, + "applicability": { + "op": "fact", + "path": "/request/type", + "operator": "in", + "value": [ + "new-data-pipeline", + "pipeline-change", + "dataset-onboarding", + "one-time-extract", + "data-access", + "reporting-feed" + ] + }, + "evidenceRequirements": [ + { + "id": "intake-form", + "description": "The submitted request carrying every mandatory intake field.", + "required": true, + "kind": "document" + }, + { + "id": "sponsor-endorsement", + "description": "A named accountable business sponsor at or above the minimum seniority threshold.", + "required": true, + "kind": "attestation" + }, + { + "id": "sensitive-data-approvals", + "description": "Data-owner, privacy, and/or legal approvals where sensitive or personal data is present.", + "required": false, + "kind": "attestation" + } + ], + "sources": [ + { + "id": "intake-standard", + "title": "Example data request intake standard", + "publisher": "Example Data Governance Office", + "publishedAt": "2026-01-15", + "locator": { + "kind": "uri", + "value": "https://example.invalid/standards/data-request-intake" + }, + "citation": { + "location": "Section 6 — Triage decision and outcomes", + "excerpt": "The reviewer assigns exactly one outcome: proceed when complete and appropriate; clarify/return when substantially valid but incomplete or not yet evaluable; decline/redirect when an appropriateness criterion fails in a way more information will not fix." + }, + "rights": "Invented content for specification testing; not an operational policy." + } + ], + "outcomes": [ + { + "id": "proceed", + "label": "Proceed", + "description": "Complete and appropriate: accept into the delivery queue, record the accepted scope, and hand off to the build process." + }, + { + "id": "clarify-return", + "label": "Clarify / return", + "description": "Substantially valid but incomplete or not yet evaluable: return with the specific enumerated gaps, all at once." + }, + { + "id": "decline-redirect", + "label": "Decline / redirect", + "description": "Fails an appropriateness criterion that more information will not fix: decline with a clear reason and, where possible, a redirect or alternative." + } + ], + "rules": [ + { + "id": "decline-hard-appropriateness-failure", + "description": "Decline when the request fails an appropriateness criterion that more information cannot fix, such as no sponsored purpose, consumers not authorized for the data classification, a data-residency breach, duplication of an existing asset, or the wrong delivery mechanism.", + "when": { + "op": "fact", + "path": "/request/appropriateness", + "operator": "equals", + "value": "hard-fail" + }, + "outcome": "decline-redirect", + "onUnknown": "escalate", + "evidenceRequirementRefs": ["intake-form"], + "sourceRefs": ["intake-standard"], + "rationale": "The cited standard treats a hard appropriateness failure as a decline, not a clarification." + }, + { + "id": "clarify-incomplete-or-not-evaluable", + "description": "Return for clarification when mandatory fields are missing or ambiguous, when approvals or triggered reviews are merely pending, or when appropriateness cannot yet be judged.", + "when": { + "op": "any", + "conditions": [ + { + "op": "fact", + "path": "/request/completeness", + "operator": "equals", + "value": "incomplete" + }, + { + "op": "fact", + "path": "/request/appropriateness", + "operator": "in", + "value": ["pending", "not-evaluable"] + } + ] + }, + "outcome": "clarify-return", + "onUnknown": "escalate", + "evidenceRequirementRefs": ["intake-form"], + "sourceRefs": ["intake-standard"], + "rationale": "The cited standard returns substantially valid but not yet decidable requests with enumerated gaps." + }, + { + "id": "proceed-complete-and-appropriate", + "description": "Proceed when every mandatory intake field is complete, every appropriateness criterion passes, and the mandatory evidence is present.", + "when": { + "op": "all", + "conditions": [ + { + "op": "fact", + "path": "/request/completeness", + "operator": "equals", + "value": "complete" + }, + { + "op": "fact", + "path": "/request/appropriateness", + "operator": "equals", + "value": "pass" + }, + { + "op": "evidence-present", + "evidenceRequirement": "intake-form" + }, + { + "op": "evidence-present", + "evidenceRequirement": "sponsor-endorsement" + } + ] + }, + "outcome": "proceed", + "onUnknown": "escalate", + "evidenceRequirementRefs": ["intake-form", "sponsor-endorsement"], + "sourceRefs": ["intake-standard"], + "rationale": "The cited standard accepts complete and appropriate requests into the delivery queue." + } + ], + "exceptions": [ + { + "id": "embargoed-information-to-unauthorized", + "description": "Embargoed material non-public information reaching recipients outside the authorized set, or ahead of public release, is a hard decline: restrict to the authorized set and escalate to the disclosure office.", + "when": { + "op": "fact", + "path": "/request/embargoedInformationToUnauthorizedRecipients", + "operator": "equals", + "value": true + }, + "effect": "force-outcome", + "outcome": "decline-redirect", + "onUnknown": "escalate" + } + ], + "fallbackOutcome": "clarify-return", + "escalation": { + "triggers": [ + "not-applicable", + "missing-required-evidence", + "unknown", + "conflict", + "no-match" + ], + "target": { + "kind": "human-role", + "name": "Intake reviewer" + }, + "message": "Escalate to the intake reviewer: enumerate the missing intake fields or unresolved appropriateness criteria, note any conflicting rules, and route embargoed-information questions to the disclosure office." + }, + "metadata": { + "authors": ["Judgment Pack Specification contributors"], + "createdAt": "2026-07-27T00:00:00Z", + "license": "Apache-2.0" + } +} diff --git a/internal/mcp/doc.go b/internal/mcp/doc.go index 4d600c3..26e9ee5 100644 --- a/internal/mcp/doc.go +++ b/internal/mcp/doc.go @@ -1,10 +1,13 @@ // Package mcp exposes this runtime over the Model Context Protocol. // // It is a transport adapter only: it maps MCP tool calls onto the existing -// validation, conformance, and describe packages and returns their versioned -// results. It evaluates no condition, resolves no outcome, opens no network -// connection, and holds no credential -- an MCP client reaches exactly the same -// offline core the CLI reaches, over JSON-RPC on stdio instead of argv. +// validation, conformance, describe, and evaluation packages and returns their +// versioned results. It opens no network connection and holds no credential -- +// an MCP client reaches exactly the same offline core the CLI reaches, over +// JSON-RPC on stdio instead of argv. The validation tools evaluate no +// condition and resolve no outcome; the one exception is experimental_evaluate +// (ADR-0007), which runs the explicitly EXPERIMENTAL, non-conformance-claiming +// evaluation package and says so in every payload. // // The server speaks MCP's newline-delimited JSON-RPC 2.0 stdio framing directly, // without a third-party SDK, to keep the runtime's dependency set minimal and diff --git a/internal/mcp/server_test.go b/internal/mcp/server_test.go index 2a67920..a5eb32c 100644 --- a/internal/mcp/server_test.go +++ b/internal/mcp/server_test.go @@ -5,6 +5,8 @@ import ( "crypto/sha256" "encoding/hex" "encoding/json" + "os" + "path/filepath" "reflect" "strings" "testing" @@ -100,13 +102,13 @@ func TestServerLifecycleToolsAndValidate(t *testing.T) { tool := entry.(map[string]any) advertised[tool["name"].(string)] = tool } - for _, want := range []string{"validate", "test_conformance", "get_schema", "describe_runtime", "list_examples", "get_example"} { + for _, want := range []string{"validate", "test_conformance", "get_schema", "describe_runtime", "list_examples", "get_example", "experimental_evaluate"} { if _, ok := advertised[want]; !ok { t.Fatalf("tools/list must advertise %q; got %v", want, advertised) } } - if len(advertised) != 6 { - t.Fatalf("expected 6 distinct tools, got %d", len(advertised)) + if len(advertised) != 7 { + t.Fatalf("expected 7 distinct tools, got %d", len(advertised)) } required := advertised["get_example"]["inputSchema"].(map[string]any)["required"].([]any) if len(required) != 1 || required[0] != "name" { @@ -228,3 +230,40 @@ func TestExampleToolsSurfaceEmbeddedFixtures(t *testing.T) { t.Fatalf("an unknown example name should be an in-band tool error: %#v", unknown) } } + +// experimental_evaluate is labeled, produces a disposition for a conformant +// pack, and refuses a non-conformant pack as an in-band tool error. +func TestExperimentalEvaluateTool(t *testing.T) { + pack, err := os.ReadFile(filepath.Join("..", "evaluation", "testdata", "data-request-intake-triage.json")) + if err != nil { + t.Fatal(err) + } + facts := `{"request":{"type":"one-time-extract","completeness":"incomplete","appropriateness":"pass","embargoedInformationToUnauthorizedRecipients":false}}` + evidence := `{"intake-form":"present","sponsor-endorsement":"present"}` + input := strings.Join([]string{ + message(t, 1, "tools/call", map[string]any{"name": "experimental_evaluate", "arguments": map[string]any{"pack": string(pack), "facts": facts, "evidence": evidence}}), + message(t, 2, "tools/call", map[string]any{"name": "experimental_evaluate", "arguments": map[string]any{"pack": `{"specVersion":"0.1.0-draft"}`, "facts": `{}`}}), + }, "") + responses := runServer(t, input) + + evaluated := responses[0]["result"].(map[string]any) + if evaluated["isError"] != false { + t.Fatalf("a produced disposition is a successful call: %#v", evaluated) + } + structured := evaluated["structuredContent"].(map[string]any) + if structured["experimental"] != true || structured["conformanceClaim"] != "none" { + t.Fatalf("the payload must be labeled experimental with no claim: %#v", structured) + } + disposition := structured["disposition"].(map[string]any) + if disposition["kind"] != "outcome" || disposition["outcomeId"] != "clarify-return" { + t.Fatalf("disposition = %#v", disposition) + } + + refused := responses[1]["result"].(map[string]any) + if refused["isError"] != true { + t.Fatalf("a non-conformant pack must be an in-band tool error: %#v", refused) + } + if text := refused["content"].([]any)[0].(map[string]any)["text"].(string); !strings.Contains(text, "document conformance") { + t.Fatalf("the refusal must be self-sufficient: %q", text) + } +} diff --git a/internal/mcp/tools.go b/internal/mcp/tools.go index 888b2af..19676ec 100644 --- a/internal/mcp/tools.go +++ b/internal/mcp/tools.go @@ -1,6 +1,7 @@ package mcp import ( + "bytes" "encoding/json" "errors" "fmt" @@ -8,11 +9,13 @@ import ( "github.com/Judgment-Pack/judgment-pack-runtime/internal/artifacts" "github.com/Judgment-Pack/judgment-pack-runtime/internal/carrier" "github.com/Judgment-Pack/judgment-pack-runtime/internal/describe" + "github.com/Judgment-Pack/judgment-pack-runtime/internal/evaluation" "github.com/Judgment-Pack/judgment-pack-runtime/internal/validation" ) // toolDefinitions is the tools/list payload. Every tool wraps a read-only core -// operation and evaluates nothing. +// operation; all evaluate nothing except experimental_evaluate, the one +// explicitly EXPERIMENTAL evaluator (ADR-0007), which claims no conformance. func toolDefinitions() []map[string]any { return []map[string]any{ { @@ -81,6 +84,21 @@ func toolDefinitions() []map[string]any { }, }, }, + { + "name": "experimental_evaluate", + "description": "EXPERIMENTAL (ADR-0007): apply the JPS Core §§7-8 experiment, as pinned by spec RFC 0006 (Draft), to one conformant pack and one facts document, returning a disposition (kind, reasons, handoff) and a trace. This claims NO evaluator conformance — JPS 0.1.0-draft forbids such claims — authorizes nothing, executes nothing, and may change or be removed without compatibility promise.", + "inputSchema": map[string]any{ + "type": "object", + "additionalProperties": false, + "required": []string{"pack", "facts"}, + "properties": map[string]any{ + "pack": map[string]any{"type": "string", "description": "The JPS document to evaluate, as JSON text. It must have full document conformance; a non-conformant pack is refused."}, + "facts": map[string]any{"type": "string", "description": "One JSON facts document, as JSON text; fact.path pointers resolve against it."}, + "evidence": map[string]any{"type": "string", "description": "Optional tri-state evidence availability, as JSON text: an object mapping declared evidence-requirement ids to \"present\", \"absent\", or \"unknown\". An omitted id is unknown."}, + "supported_extensions": map[string]any{"type": "array", "items": map[string]any{"type": "string"}, "description": "Extension names this consumer supports."}, + }, + }, + }, } } @@ -105,6 +123,8 @@ func (s *Server) callTool(rawParams json.RawMessage) (any, *rpcError) { return s.toolListExamples(), nil case "get_example": return s.toolGetExample(params.Arguments), nil + case "experimental_evaluate": + return s.toolExperimentalEvaluate(params.Arguments), nil default: return nil, &rpcError{Code: codeInvalidParams, Message: "Unknown tool: " + params.Name} } @@ -235,6 +255,41 @@ func (s *Server) toolGetExample(rawArgs json.RawMessage) any { } } +// toolExperimentalEvaluate runs the experimental §§7-8 evaluator (ADR-0007). +// An evaluation failure — a non-conformant pack, a malformed input, an +// undeclared evidence key — is an in-band tool error; a produced disposition +// of any kind is a successful call. +func (s *Server) toolExperimentalEvaluate(rawArgs json.RawMessage) any { + var args struct { + Pack string `json:"pack"` + Facts string `json:"facts"` + Evidence string `json:"evidence"` + SupportedExtensions []string `json:"supported_extensions"` + } + if len(rawArgs) > 0 { + // Strict decoding honors the declared additionalProperties: false — a + // misspelled key (say "evidnce") must be an error, not a silently + // different disposition. + decoder := json.NewDecoder(bytes.NewReader(rawArgs)) + decoder.DisallowUnknownFields() + if err := decoder.Decode(&args); err != nil { + return toolError(`The "experimental_evaluate" arguments must be an object with string "pack" and "facts", optional string "evidence", and optional "supported_extensions" (an array of strings); unknown keys are rejected.`) + } + } + if args.Pack == "" { + return toolError(`The "pack" argument is required: pass the JPS document as JSON text.`) + } + if args.Facts == "" { + return toolError(`The "facts" argument is required: pass one JSON facts document as JSON text.`) + } + evaluator := evaluation.NewEngine(s.engine) + output, failure := evaluator.Evaluate([]byte(args.Pack), []byte(args.Facts), []byte(args.Evidence), args.SupportedExtensions, "mcp experimental_evaluate") + if failure != nil { + return toolError(failure.Message) + } + return toolResult(output) +} + // toolResult wraps a versioned core payload as an MCP tool result: the payload // as structured content, and its JSON as text for a client that reads only // text. A reported "invalid" document is a successful call, not a tool error. diff --git a/internal/result/result.go b/internal/result/result.go index 9761389..0fd987e 100644 --- a/internal/result/result.go +++ b/internal/result/result.go @@ -198,6 +198,70 @@ type Example struct { WrittenTo string `json:"writtenTo,omitempty"` } +// --- experimental evaluation (ADR-0007; spec RFC 0006) --- + +// EvaluationClaim is carried by every experimental-evaluation payload so the +// output can never be mistaken for a conformance claim: JPS 0.1.0-draft §3.4 +// forbids evaluator-conformance claims outright. +const EvaluationClaim = "none" + +// HandoffTarget echoes the pack's declared escalation target when a handoff is +// requested. +type HandoffTarget struct { + Kind string `json:"kind"` + Name string `json:"name"` +} + +// Handoff reports whether the evaluation requests a human handoff. A direct +// exception escalation on a pack with no escalation object is a requested +// handoff with no Core-defined destination: state "requested", target absent. +type Handoff struct { + State string `json:"state"` + Target *HandoffTarget `json:"target,omitempty"` +} + +// Disposition is the portable evaluation result proposed by spec RFC 0006: +// kind ("outcome", "not-applicable", or "unresolved"), the outcome id exactly +// when kind is "outcome", a deduplicated sorted reason set (empty exactly when +// kind is "outcome"), and the handoff axis. +type Disposition struct { + Kind string `json:"kind"` + OutcomeID string `json:"outcomeId,omitempty"` + Reasons []string `json:"reasons"` + Handoff Handoff `json:"handoff"` +} + +// TraceEntry records one exception or rule evaluation. The trace is +// informative: §8 requires an unknown that resolution ignored to remain +// visible, and permits recording contributing ids. +type TraceEntry struct { + Stage string `json:"stage"` + ID string `json:"id"` + Condition string `json:"condition"` + Effect string `json:"effect,omitempty"` + Outcome string `json:"outcome,omitempty"` + Suppressed bool `json:"suppressed,omitempty"` + OnUnknown string `json:"onUnknown,omitempty"` + Skipped bool `json:"skipped,omitempty"` +} + +// Evaluation is the experimental-evaluation envelope. Experimental and +// ConformanceClaim are always set so no consumer can read the payload as a +// standard: this surface may change or be removed without compatibility +// promise (ADR-0007). +type Evaluation struct { + OutputVersion string `json:"outputVersion"` + Tool Tool `json:"tool"` + Command string `json:"command"` + Status string `json:"status"` + Experimental bool `json:"experimental"` + ConformanceClaim string `json:"conformanceClaim"` + SpecVersion string `json:"specVersion"` + Disposition Disposition `json:"disposition"` + Trace []TraceEntry `json:"trace"` + Artifact *Artifact `json:"artifact,omitempty"` +} + type OperationalError struct { OutputVersion string `json:"outputVersion"` Tool Tool `json:"tool"`