diff --git a/CHANGELOG.md b/CHANGELOG.md index db52999..8930e51 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,17 @@ All notable changes to tagged releases are documented here. +## Unreleased + +- Add the MCP `prompts` capability (ADR-0008): three non-normative authoring-method prompts — + `author_pack`, `test_pack`, `fix_pack` — served as static, versioned text that the client's model + executes with the client's key. The server still calls no model, holds no key, and opens no + network connection; serving a prompt is read-only, like `get_schema`. Method content distills the + expressiveness studies: the resolution-model shapes that avoid conflicts, the `onUnknown` + discipline, the decimal-string rule for ordered comparisons, the prepared-facts ledger, and the + instance-matrix logic probe. Following a prompt does not make a pack conformant, and no prompt + interprets any policy. + ## 0.2.0 - 2026-07-27 - Add an EXPERIMENTAL evaluator (ADR-0007): `judgment-pack experimental evaluate` and the diff --git a/docs/adr/0003-mcp-integration-and-testing-surface.md b/docs/adr/0003-mcp-integration-and-testing-surface.md index 99bcc6b..a95cd5d 100644 --- a/docs/adr/0003-mcp-integration-and-testing-surface.md +++ b/docs/adr/0003-mcp-integration-and-testing-surface.md @@ -12,6 +12,10 @@ deciders: Brian Jin > **Amended by [0007](0007-experimental-evaluator.md):** the surface gains exactly one explicitly > experimental evaluation tool (`experimental_evaluate`); the validation tools still evaluate > nothing. +> +> **Amended by [0008](0008-mcp-prompts-authoring-method.md):** the surface gains the protocol's +> `prompts` capability — non-normative authoring-method text served statically; the client's model +> executes it. ## Context and problem statement diff --git a/docs/adr/0008-mcp-prompts-authoring-method.md b/docs/adr/0008-mcp-prompts-authoring-method.md new file mode 100644 index 0000000..8a0d999 --- /dev/null +++ b/docs/adr/0008-mcp-prompts-authoring-method.md @@ -0,0 +1,87 @@ +--- +status: proposed +date: 2026-07-27 +deciders: Brian Jin +--- + +# Serve authoring method as MCP prompts; the intelligence stays in the client + +> **Proposed.** The MCP server gains the protocol's `prompts` capability with three non-normative +> method prompts — `author_pack`, `test_pack`, `fix_pack` — served as static, versioned text that +> the client's model executes with the client's key. + +## Context and problem statement + +Authoring a judgment pack is judgment-laden translation, and by [ADR-0006](0006-authoring-lifecycle-in-the-client.md) +it lives in the client: the runtime is a stateless oracle that never authors, decides, or holds a +key. But the expressiveness studies produced hard-won *method* knowledge that currently lives only +in prose documentation a filesystem-less client cannot reach: the single-outcome/detector +architecture that §8's conflict rule forces authors to rediscover, the `onUnknown` discipline and +its fallback-blocking behavior, the decimal-string rule for ordered comparisons, and the +prepared-facts ledger that RFC 0007 makes basic authoring hygiene. Every new authoring client +currently relearns these by tripping on them. + +An "encode this policy" *tool* was considered and rejected earlier in the project: it would put a +model, a key, and nondeterminism inside the runtime, and would make the reference implementation +the canonical interpreter of what policies mean — spec-by-implementation one layer up. + +MCP has a third primitive that threads this: **prompts** are static text the server serves and the +*client's* model executes. The server never calls a model. + +## Decision drivers + +- Equip the client-side authoring loop without violating keyless / offline / stateless. +- Turn study findings into method every author inherits, instead of folklore rediscovered per pack. +- Keep interpretation plural: the runtime may teach *how to work*, never *what a policy means*. + +## Considered options + +- **A. MCP `prompts` capability** serving method guidance as versioned static text. +- **B. Documentation only** (status quo): the method stays in `docs/`, unreachable from a + filesystem-less client's tool surface. +- **C. An authoring tool** that produces packs server-side. Rejected again for the reasons above. + +## Decision outcome + +Chosen option: **A**. Serving a prompt is a read-only operation in the same class as `get_schema`: +static bytes, versioned with the binary, no model, no key, no network, no state. It *strengthens* +ADR-0006 rather than bending it — the prompt's entire content is instructions for the client-side +loop against the oracle tools. + +Settled constraints: + +- **Three prompts, method-only:** `author_pack` (the guided create → validate → evaluate loop, the + §8-shape guidance, the decimal-string and `onUnknown` rules, the prepared-facts ledger), + `test_pack` (the instance-matrix probe: per-outcome, conflict, unknown, missing-evidence, + not-applicable, forced-outcome, ordered-comparison rows), `fix_pack` (diagnostics-driven repair + in carrier → structural → semantic order). +- **Non-normative, and each prompt says so in its own text:** following the method does not make a + pack conformant; only validation decides conformance; the produced document belongs to the + client. No prompt implies the runtime blesses, stores, or interprets anything. +- **No policy interpretation:** prompts teach the format's mechanics and the loop's discipline. + They never state what any domain's policy means. Argument values (a policy text passed by the + caller) are echoed into the rendered prompt verbatim, not interpreted. +- **Versioned with the binary** like the embedded fixtures: prompt text changes ship as releases + and appear in the changelog. + +This amends [0003](0003-mcp-integration-and-testing-surface.md) a second time: the surface is now +tools + prompts. The tools' posture is unchanged. + +### Consequences + +- Good, because a filesystem-less client gets the authoring method at the same place it gets the + oracle, and study findings stop being folklore. +- Good, because the division of labor becomes explicit and inspectable: the runtime ships the + method, the client ships the mind, the tools stay the oracle. +- Bad, because the reference runtime's prompts become the de facto authoring *style*; mitigated by + the non-normative marking, by the prompts being small inspectable text rather than behavior, and + by teaching mechanics rather than interpretation. +- Bad, because client support for MCP prompts is uneven; degradation is graceful (tools are + untouched) but the guidance reaches only clients that surface prompts. + +## More information + +Method content sources: the expressiveness studies in `judgment-pack-evaluator-experiments` +(studies 001–003) and spec RFC 0007. Protocol: MCP `prompts/list` and `prompts/get`, capability +`prompts` advertised at initialize. Follows ADR-0002 (one core), ADR-0003 (MCP surface), ADR-0006 +(authoring in the client), ADR-0007 (the experimental evaluator the `test_pack` prompt drives). diff --git a/docs/adr/README.md b/docs/adr/README.md index ee9f8fa..f514d0a 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -36,3 +36,4 @@ when** it became that way. architecture.md links out to the relevant ADR rather | [0005](0005-single-jps-diagnostic-code-prefix.md) | Use a single `JPS-` prefix for diagnostic codes | accepted | | [0006](0006-authoring-lifecycle-in-the-client.md) | The authoring lifecycle lives in the client; the runtime is a stateless oracle | accepted | | [0007](0007-experimental-evaluator.md) | Ship an experimental evaluator behind an explicit experimental surface | accepted | +| [0008](0008-mcp-prompts-authoring-method.md) | Serve authoring method as MCP prompts; the intelligence stays in the client | proposed | diff --git a/docs/mcp-clients.md b/docs/mcp-clients.md index e95851c..9758431 100644 --- a/docs/mcp-clients.md +++ b/docs/mcp-clients.md @@ -27,6 +27,22 @@ gitignored in this repository. Copy a snippet, don't commit one. None of these evaluate, decide, or authorize anything, except `experimental_evaluate`, which evaluates experimentally and says so in every payload. +## The prompts + +The server also serves three **method prompts** (MCP `prompts` capability) — static, versioned, +non-normative guidance that your client's model executes with your key. In Claude Code they appear +as slash commands (`/mcp__judgment-pack__author_pack`, `…test_pack`, `…fix_pack`); other clients +surface them differently or not at all, and everything works without them. + +| Prompt | Guides | +| --- | --- | +| `author_pack` | Encoding one policy decision: the create → validate → evaluate loop, the resolution-model shapes that avoid conflicts, the decimal-string and `onUnknown` rules, the prepared-facts ledger | +| `test_pack` | Probing a pack's logic with an instance matrix (per-outcome, conflict, unknown, missing-evidence, not-applicable, forced-outcome, ordered-comparison rows) | +| `fix_pack` | Repairing a non-conformant pack from the validator's diagnostics, in carrier → structural → semantic order | + +Following a prompt does not make a pack conformant — only validation decides that — and the +documents you produce are yours; the runtime stores and interprets nothing (ADR-0008). + ## Claude Code One command, from any directory: diff --git a/internal/mcp/prompts.go b/internal/mcp/prompts.go new file mode 100644 index 0000000..8ed01cb --- /dev/null +++ b/internal/mcp/prompts.go @@ -0,0 +1,237 @@ +package mcp + +import ( + "encoding/json" + "strings" +) + +// The prompts surface (ADR-0008) serves authoring METHOD as static, versioned +// text; the client's model executes it with the client's key. The server never +// calls a model: serving a prompt is read-only, the same class of operation as +// get_schema. Everything here is non-normative guidance — following it does +// not make a pack conformant, and nothing in it is a conformance claim. + +type promptArgument struct { + Name string `json:"name"` + Description string `json:"description"` + Required bool `json:"required"` +} + +type promptDefinition struct { + name string + description string + arguments []promptArgument + // build renders the prompt text with the caller's arguments. + build func(args map[string]string) string +} + +func promptDefinitions() []promptDefinition { + return []promptDefinition{ + { + name: "author_pack", + description: "Guide an authoring session: encode one policy decision as a JPS judgment " + + "pack using the validate/evaluate loop. Non-normative method guidance; the pack it " + + "helps produce is yours, not the runtime's.", + arguments: []promptArgument{ + {Name: "policy", Description: "The policy text or a description of the decision to encode.", Required: false}, + }, + build: buildAuthorPack, + }, + { + name: "test_pack", + description: "Guide a logic-testing session: build an instance matrix for a pack you hold " + + "and probe it with the experimental evaluator. Non-normative method guidance.", + arguments: []promptArgument{ + {Name: "pack", Description: "The pack under test, as JSON text (optional; you may also reference a file you hold).", Required: false}, + }, + build: buildTestPack, + }, + { + name: "fix_pack", + description: "Guide a repair session: work a non-conformant pack back to valid using the " + + "validator's diagnostics. Non-normative method guidance.", + arguments: []promptArgument{ + {Name: "diagnostics", Description: "The diagnostics from a failed validate call (optional).", Required: false}, + }, + build: buildFixPack, + }, + } +} + +const authoringDisclaimer = `(Method guidance from the judgment-pack runtime, non-normative: following it does not make a +pack conformant, and only "spec validate" / the "validate" tool decides conformance. The document +you produce is yours; this runtime stores nothing and decides nothing.)` + +func buildAuthorPack(args map[string]string) string { + var b strings.Builder + b.WriteString("Encode ONE policy decision as a Judgment Pack (JPS 0.1.0-draft), working in a\n") + b.WriteString("validate-and-fix loop against the judgment-pack tools.\n\n") + if policy := strings.TrimSpace(args["policy"]); policy != "" { + b.WriteString("The policy to encode:\n\n---\n" + policy + "\n---\n\n") + } else { + b.WriteString("Ask for the policy text or decision description if you do not already hold it.\n\n") + } + b.WriteString(`Work in this order: + +1. ORIENT. Call get_schema for the format reference and list_examples for the bundled valid + fixtures; fetch the one closest to your decision with get_example and use its SHAPE (not its + content) as scaffolding. Fixtures are version-pinned conformance cases, not templates. + +2. SCOPE. One pack = one decision ("may X be done?"), with the outcomes the policy itself names. + Agent procedure ("confirm before acting") is not a decision; leave it out. If the policy names a + case that must go to a human, that is the escalation machinery, not an outcome. + +3. STRUCTURE for the resolution model. Two rules that fire together naming DIFFERENT outcomes are a + conflict and the result is unresolved -- there is no rule priority. Two shapes work well: + - Detector style: every rule detects one outcome (say, a violation -> deny); the opposite + outcome is reachable only as fallbackOutcome. Simultaneous detections then agree instead of + conflicting, and every fired rule is a citable reason. + - Carve-out style: encode "if A then deny, otherwise allow when B" with a force-outcome or + escalate EXCEPTION for A (exceptions outrank rules) and rules for B. + Pick one deliberately; mixing affirmative and negative rules invites unresolved conflicts. + +4. CONDITIONS. Facts are addressed by RFC 6901 JSON Pointer into one facts document. Ordered + comparisons (greater-than family) are defined only over decimal STRINGS matching + -?(0|[1-9][0-9]*)(\.[0-9]+)? -- a JSON number on either side yields unknown, silently. The + format has no arithmetic, no date/time comparison, and no quantifier over arrays: any such value + must be prepared upstream and supplied as a fact. Keep a PREPARED-FACTS ledger as you go: every + fact that is computed or concluded rather than stated by the requester, and for each, whether + producing it requires applying the policy itself (flag those loudly -- they are decision logic + living outside your pack). + +5. UNCERTAINTY is the point. onUnknown: escalate on a rule means "a missing fact here must stop + the decision"; onUnknown: ignore means "this rule simply does not fire". Gate rules with an + always-present boolean condition where you can: conjunction is strong, so a false gate beats an + unknown sibling and an inapplicable rule stays false instead of escalating. Wire escalation + triggers to the reasons you actually want handed to a human, and remember an outcome produced by + fallbackOutcome is blocked by an escalating unknown -- that blocking is usually what you want. + +6. EVIDENCE. evidenceRequirements with required: true block every evaluation that arrives without + an evidence document -- use required: true only for evidence the decision truly cannot proceed + without, and remember callers may not supply an evidence document at all. + +7. LOOP. validate the draft; every diagnostic names its location and the fix; repair and repeat to + exit 0. Then evaluate against 2-3 realistic facts documents and check the dispositions match + your intent -- including one probe with a load-bearing fact REMOVED, which should escalate, not + guess. + +8. RECORD. Keep your interpretation decisions (ambiguous policy text, the reading you chose, why) + and the prepared-facts ledger next to the pack. They are the audit trail of everything the pack + itself cannot say. + +`) + b.WriteString(authoringDisclaimer) + return b.String() +} + +func buildTestPack(args map[string]string) string { + var b strings.Builder + b.WriteString("Probe the LOGIC of a judgment pack you hold, using the experimental_evaluate tool.\n") + b.WriteString("Validation proved the pack well-formed; this session tests whether it decides what its\n") + b.WriteString("author intended. The evaluator is experimental and claims no conformance.\n\n") + if pack := strings.TrimSpace(args["pack"]); pack != "" { + b.WriteString("The pack under test:\n\n---\n" + pack + "\n---\n\n") + } + b.WriteString(`Build an instance matrix -- one facts document per row -- and evaluate each: + +1. One instance per declared outcome, engineered so exactly that outcome's rule fires. +2. A conflict probe: facts that make two rules with DIFFERENT outcomes true at once. Expect + unresolved with reason "conflict" -- if you get an outcome instead, the rules cannot actually + fire together and your probe is wrong, or the pack shape differs from what you think. +3. An unknown probe per escalating rule: omit one fact that rule needs. Expect unresolved with + reason "unknown" and a handoff if the trigger is wired. If the fallback outcome appears instead, + the rule has onUnknown: ignore -- check that is intended. +4. A missing-evidence probe if the pack declares required evidence: evaluate without an evidence + document. Expect unresolved with reason "missing-required-evidence". +5. A not-applicable probe if the pack declares applicability: facts outside scope. Expect a + not-applicable result, not an outcome. +6. A forced-outcome probe per exception: facts that satisfy the exception AND a rule it should + override. Expect the exception's outcome with the rule skipped. +7. An ordered-comparison probe if any rule compares magnitudes: supply the value as a JSON number + instead of a decimal string. Expect unknown behavior per the rule's onUnknown -- this catches + the most common silent authoring mistake. + +Read each disposition fully: kind, outcomeId, reasons, handoff, and the trace (which rules fired, +which were skipped). A divergence between expectation and disposition is either a pack bug or an +intended escalation -- decide which, fix the pack if the former, and re-run the whole matrix after +any change. Keep the matrix with the pack; it is the pack's regression suite. + +`) + b.WriteString(authoringDisclaimer) + return b.String() +} + +func buildFixPack(args map[string]string) string { + var b strings.Builder + b.WriteString("Repair a judgment pack that failed validation, using the validator's diagnostics.\n\n") + if diagnostics := strings.TrimSpace(args["diagnostics"]); diagnostics != "" { + b.WriteString("The diagnostics:\n\n---\n" + diagnostics + "\n---\n\n") + } else { + b.WriteString("Run the validate tool on the pack first; work from its diagnostics.\n\n") + } + b.WriteString(`Each diagnostic is self-sufficient: a code, the JSON Pointer location, and a message that +names the offending value and what would be accepted. Work them in order: + +1. Carrier diagnostics (malformed or duplicate-member JSON) first -- nothing else is trustworthy + until the document parses cleanly. +2. Structural diagnostics next. Common causes: an unknown member (typo or misplaced field), a + condition "op" outside the allowed set, an ordered-comparison operand that is a JSON number + instead of a decimal string, an exception whose members do not match its effect (force-outcome + needs "outcome", suppress-rule needs "targetRule", escalate needs neither). +3. Semantic diagnostics last. These are almost always dangling references: a rule outcome, + fallbackOutcome, exception target, evidenceRequirementRef, or sourceRef that names no declared + id. The message lists the declared ids -- fix the reference or add the declaration, then check + for others of the same kind before re-validating. + +Re-validate after each batch; repeat to exit 0. An "unsupported" result is different from invalid: +it means a required extension this consumer does not support, which no edit to the pack fixes. +After reaching valid, re-run whatever instance matrix exists for the pack -- a repair that changes +ids or structure can change behavior, and validity does not test logic. + +`) + b.WriteString(authoringDisclaimer) + return b.String() +} + +// listPrompts is the prompts/list payload. +func listPrompts() []map[string]any { + defs := promptDefinitions() + out := make([]map[string]any, 0, len(defs)) + for _, def := range defs { + out = append(out, map[string]any{ + "name": def.name, + "description": def.description, + "arguments": def.arguments, + }) + } + return out +} + +// getPrompt renders one prompt with the caller's arguments. +func getPrompt(rawParams json.RawMessage) (map[string]any, *rpcError) { + var params struct { + Name string `json:"name"` + Arguments map[string]string `json:"arguments"` + } + if err := json.Unmarshal(rawParams, ¶ms); err != nil { + return nil, &rpcError{Code: codeInvalidParams, Message: "Invalid prompts/get parameters."} + } + for _, def := range promptDefinitions() { + if def.name != params.Name { + continue + } + if params.Arguments == nil { + params.Arguments = map[string]string{} + } + return map[string]any{ + "description": def.description, + "messages": []map[string]any{ + { + "role": "user", + "content": map[string]any{"type": "text", "text": def.build(params.Arguments)}, + }, + }, + }, nil + } + return nil, &rpcError{Code: codeInvalidParams, Message: "Unknown prompt: " + params.Name} +} diff --git a/internal/mcp/server.go b/internal/mcp/server.go index e9b2749..9e4a0c8 100644 --- a/internal/mcp/server.go +++ b/internal/mcp/server.go @@ -104,6 +104,11 @@ func (s *Server) handle(request *rpcRequest) (map[string]any, bool) { case "tools/call": res, rerr := s.callTool(request.Params) return s.reply(request, res, rerr) + case "prompts/list": + return s.reply(request, map[string]any{"prompts": listPrompts()}, nil) + case "prompts/get": + res, rerr := getPrompt(request.Params) + return s.reply(request, res, rerr) default: if len(request.ID) == 0 { return nil, false // ignore unknown notifications @@ -139,7 +144,7 @@ func (s *Server) initializeResult(rawParams json.RawMessage) map[string]any { } return map[string]any{ "protocolVersion": version, - "capabilities": map[string]any{"tools": map[string]any{}}, + "capabilities": map[string]any{"tools": map[string]any{}, "prompts": map[string]any{}}, "serverInfo": map[string]any{ "name": result.CLIName, "version": result.CLIVersion, diff --git a/internal/mcp/server_test.go b/internal/mcp/server_test.go index a5eb32c..3b14e1f 100644 --- a/internal/mcp/server_test.go +++ b/internal/mcp/server_test.go @@ -267,3 +267,69 @@ func TestExperimentalEvaluateTool(t *testing.T) { t.Fatalf("the refusal must be self-sufficient: %q", text) } } + +// The prompts surface (ADR-0008) serves non-normative authoring method as +// static text: capability advertised, three prompts listed with their +// arguments, argument text echoed verbatim into the rendered prompt, and the +// non-normative disclaimer present in every rendering. +func TestPromptsSurface(t *testing.T) { + input := strings.Join([]string{ + message(t, 1, "initialize", map[string]any{"protocolVersion": "2025-06-18"}), + message(t, -1, "notifications/initialized", nil), + message(t, 2, "prompts/list", nil), + message(t, 3, "prompts/get", map[string]any{"name": "author_pack", "arguments": map[string]any{"policy": "Employees may expense meals under 50 dollars."}}), + message(t, 4, "prompts/get", map[string]any{"name": "no_such_prompt"}), + }, "") + responses := runServer(t, input) + + capabilities := responses[0]["result"].(map[string]any)["capabilities"].(map[string]any) + if _, ok := capabilities["prompts"]; !ok { + t.Fatalf("initialize must advertise the prompts capability: %#v", capabilities) + } + + prompts := responses[1]["result"].(map[string]any)["prompts"].([]any) + names := map[string]bool{} + for _, entry := range prompts { + names[entry.(map[string]any)["name"].(string)] = true + } + for _, want := range []string{"author_pack", "test_pack", "fix_pack"} { + if !names[want] { + t.Fatalf("prompts/list must include %q: %v", want, names) + } + } + if len(names) != 3 { + t.Fatalf("expected exactly 3 prompts, got %d", len(names)) + } + + rendered := responses[2]["result"].(map[string]any) + text := rendered["messages"].([]any)[0].(map[string]any)["content"].(map[string]any)["text"].(string) + if !strings.Contains(text, "Employees may expense meals under 50 dollars.") { + t.Fatalf("the policy argument must be echoed verbatim into the prompt") + } + for _, marker := range []string{"non-normative", "validate", "onUnknown"} { + if !strings.Contains(text, marker) { + t.Fatalf("author_pack rendering must contain %q", marker) + } + } + + if _, isError := responses[3]["error"]; !isError { + t.Fatalf("an unknown prompt must be a JSON-RPC error: %#v", responses[3]) + } +} + +// Every prompt renders with no arguments at all, and every rendering carries +// the no-claim disclaimer -- the guardrail that the method text can never be +// read as the runtime blessing a pack. +func TestEveryPromptRendersWithDisclaimer(t *testing.T) { + for _, name := range []string{"author_pack", "test_pack", "fix_pack"} { + responses := runServer(t, message(t, 1, "prompts/get", map[string]any{"name": name})) + result := responses[0]["result"].(map[string]any) + text := result["messages"].([]any)[0].(map[string]any)["content"].(map[string]any)["text"].(string) + if !strings.Contains(text, "does not make a\npack conformant") && !strings.Contains(text, "does not make a pack conformant") { + t.Fatalf("%s must carry the non-normative disclaimer", name) + } + if len(text) < 500 { + t.Fatalf("%s rendering suspiciously short: %d bytes", name, len(text)) + } + } +}