Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
.idea/

# Local AI-assistant configuration and state.
# .mcp.json is each contributor's client wiring, not project source; see docs/mcp-clients.md.
/.mcp.json
/.agent/
/.agents/
/.ai/
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@

All notable changes to tagged releases are documented here.

## Unreleased

- Add an EXPERIMENTAL evaluator (ADR-0007): `judgment-pack experimental evaluate` and the
`experimental_evaluate` MCP tool apply the JPS Core §§7–8 experiment, as pinned by the
specification's RFC 0006 (Draft), to one conformant pack, one facts document, and an optional
tri-state evidence-availability input, returning a disposition (kind, reasons, handoff) and a
trace. It claims NO evaluator conformance — JPS `0.1.0-draft` forbids such claims — authorizes
nothing, evaluates only fully conformant packs, and may change or be removed without
compatibility promise. The nine walked instances committed in RFC 0006's appendix run as this
surface's acceptance tests. Validation behavior, claims, and exit classes are unchanged.

## 0.1.0 - 2026-07-25

- Add a `NOTICE` file identifying Brian Jin as the copyright holder and carrying
Expand Down
17 changes: 12 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,13 @@ schemas, and conformance corpus are owned by the separate
[`judgment-pack-spec`](https://github.com/Judgment-Pack/judgment-pack-spec) repository, and any
independent implementation that passes the published conformance corpus is equally valid.

The runtime **validates documents**. It does not evaluate rules, choose an outcome, fetch a source,
authorize an action, or establish truth, organizational authority, safety, or operational fitness.
This matches the scope of JPS `0.1.0-draft`, which defines carrier, structural, and semantic
document conformance only and defines no evaluator conformance class.
The runtime **validates documents**. It does not fetch a source, authorize an action, or establish
truth, organizational authority, safety, or operational fitness. This matches the scope of JPS
`0.1.0-draft`, which defines carrier, structural, and semantic document conformance only and
defines no evaluator conformance class. Its one evaluation surface is explicitly EXPERIMENTAL:
`judgment-pack experimental evaluate` (and the `experimental_evaluate` MCP tool) runs the
specification's §§7–8 experiment per [ADR-0007](docs/adr/0007-experimental-evaluator.md), claims
no conformance, and may change or be removed without compatibility promise.

The command binary is `judgment-pack`; release archives also ship a `jpack` short alias for the
same program.
Expand All @@ -27,14 +30,18 @@ judgment-pack version
judgment-pack spec validate <pack-or->
judgment-pack spec test-conformance [suite]
judgment-pack spec schema <spec-version>
judgment-pack spec examples [name]
judgment-pack mcp
judgment-pack experimental evaluate <pack-or-> (EXPERIMENTAL; no conformance claim)
```

The namespace is `judgment-pack spec`, not `judgment-pack jps`. JPS remains the name of the
specification and the prefix of its provisional diagnostic codes.

The `mcp` command serves the same offline operations to a Model Context Protocol client over stdio,
so an agent can validate documents as a tool call; see [docs/agent-testing.md](docs/agent-testing.md).
so an agent can validate documents as a tool call; see
[docs/mcp-clients.md](docs/mcp-clients.md) for per-client setup and
[docs/agent-testing.md](docs/agent-testing.md) for the agent-driven testing protocol.

## Install a tagged release

Expand Down
4 changes: 4 additions & 0 deletions docs/adr/0003-mcp-integration-and-testing-surface.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ deciders: Brian Jin

> **Accepted.** Phase 0 passed and Phase 1 is built: `internal/mcp` is a hand-rolled stdio MCP
> server wired as the `judgment-pack mcp` subcommand.
>
> **Amended by [0007](0007-experimental-evaluator.md):** the surface gains exactly one explicitly
> experimental evaluation tool (`experimental_evaluate`); the validation tools still evaluate
> nothing.

## Context and problem statement

Expand Down
93 changes: 93 additions & 0 deletions docs/adr/0007-experimental-evaluator.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
status: proposed
date: 2026-07-27
deciders: Brian Jin
---

# Ship an experimental evaluator behind an explicit experimental surface

> **Proposed.** Implement JPS Core's §§7–8 experiment — as pinned by the specification's
> [RFC 0006 (Draft)](https://github.com/Judgment-Pack/judgment-pack-spec/blob/main/rfcs/0006-evaluator-conformance.md)
> — as `judgment-pack experimental evaluate` and the `experimental_evaluate` MCP tool, claiming no
> evaluator conformance.

## Context and problem statement

The specification published RFC 0006 (Draft): a proposed normative evaluator conformance class
whose acceptance is gated on evidence from two independent implementations. Core `0.1.0-draft`
§3.4 forbids any evaluator-conformance claim, and the specification's tooling-architecture doc
pre-states the guardrails for exactly this situation: an experimental evaluator must be
unmistakably labeled, produce no conformance claim, and remain outside the default validation
path. Someone has to build the first implementation to generate the RFC's evidence; this runtime
is the natural first implementer. Separately, agent-driven authoring (ADR-0006) today proves that
a pack is *well-formed* but not that its logic is *correct* — an evaluator closes that loop.

## Decision drivers

- Generate RFC 0006's acceptance evidence: real implementation experience against its pinned
semantics and its nine committed instances.
- Preserve the runtime's identity: stateless, keyless, offline `bytes -> result`; evaluation is
still a pure function over supplied inputs, with no store and no side effect.
- Never contaminate the conformance surface: validation behavior, claims, and exit classes stay
untouched; the experiment must be impossible to mistake for a standard.

## Considered options

- **A. An `experimental` command namespace in this runtime** — CLI `judgment-pack experimental
evaluate` plus one clearly-labeled MCP tool.
- **B. A separate experimental binary or repository.**
- **C. Wait for the RFC to be accepted first.**

## Decision outcome

Chosen option: **A**. Option C is circular — acceptance requires implementations. Option B
fragments distribution for no isolation gain: the guardrail is the labeled, non-default surface,
not a separate artifact, and the shared engine seam (one core, per ADR-0002) is the point.

Settled constraints:

- **Semantics:** §§7–8 exactly as pinned by RFC 0006's sketch — three-valued conditions including
the §7 preamble's `literal` and §7.5 evidence presence; required-evidence check restated as
`missing-required-evidence` iff presence is false and `unknown` iff presence is unknown with
none false; ordered comparison defined only over §2.2 grammar-conforming JSON strings (JSON
numbers yield `unknown`); conflict never tie-broken; §8's order treated as contract, not
algorithm.
- **Inputs:** a pack (validated to full document conformance first — a non-conformant pack is
refused as an error, never evaluated), one JSON facts document, an optional tri-state
evidence-availability document keyed by declared requirement ids (omitted key = `unknown`;
undeclared key = input error), and the supported-extension set.
- **Output:** RFC 0006's disposition (`kind`, `outcomeId`, deduplicated `reasons`, `handoff`)
plus a minimal trace, in an envelope that carries `"experimental": true` and
`"conformanceClaim": "none"`. Producing a disposition exits 0 — the disposition is data, not a
process verdict; operational failures use the existing exit classes.
- **Errors are not dispositions:** invalid pack, unsupported required extension, malformed or
oversized inputs, and undeclared evidence keys are explicit errors.
- **Labeling:** the command group, tool name (`experimental_evaluate`), help text, tool
description, and result envelope all carry the experimental marker and the no-claim statement.
Diagnostic codes minted for this surface (`JPS-EVALUATION-*`) stay `codeStability:
"provisional"` like every other code; the experimental marking is carried by the envelope
(`experimental`, `conformanceClaim`) and the surface naming, not by a new stability tier.
- **Scope:** single-pack evaluation only. Judgment Graph composition (spec RFC 0002) and planner
selection (spec RFC 0004) are explicitly out of scope — the graph composes the dispositions
this surface produces and cannot be implemented before their semantics exist.

This amends [0003](0003-mcp-integration-and-testing-surface.md) narrowly: the existing validation
and description tools still evaluate nothing, and the MCP surface as a whole no longer "evaluates nothing" — it
gains exactly one tool that evaluates *experimentally and says so*.

### Consequences

- Good, because RFC 0006 gains its first implementation and executable evidence (the nine
instances become this surface's acceptance tests).
- Good, because agents can close the logic-correctness loop (author → validate → evaluate against
sample facts) entirely offline and keyless.
- Bad, because an experimental surface invites misuse as if it were conformant; mitigated by
labeling in every artifact and by §3.4's permanent prohibition for `0.1.0-draft`.
- Bad, because the surface may change or be removed without compatibility promise as RFC 0006
evolves; that is the meaning of experimental.

## More information

Semantics source: JPS Core §§7–8 and spec RFC 0006's Specification sketch and appendix. Guardrails:
the specification's tooling-architecture document. Follows ADR-0002 (one core), ADR-0003 (MCP
surface), and ADR-0006 (client-held lifecycle; evaluation inputs cross the wire as text).
1 change: 1 addition & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,4 @@ when** it became that way. architecture.md links out to the relevant ADR rather
| [0004](0004-decline-http-api.md) | Permanently decline an in-runtime HTTP API | accepted |
| [0005](0005-single-jps-diagnostic-code-prefix.md) | Use a single `JPS-` prefix for diagnostic codes | accepted |
| [0006](0006-authoring-lifecycle-in-the-client.md) | The authoring lifecycle lives in the client; the runtime is a stateless oracle | accepted |
| [0007](0007-experimental-evaluator.md) | Ship an experimental evaluator behind an explicit experimental surface | proposed |
5 changes: 3 additions & 2 deletions docs/agent-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,9 @@ CLI loop here needs neither an SDK nor a running server.

- A `judgment-pack` binary (installed or built below).
- Any agent client that can run a shell command and read and write files, configured with **your
own** model API key. The runtime holds no key, opens no network connection, and evaluates nothing,
so the key lives entirely in the client. Any of the many such clients works — for example Claude
own** model API key. The runtime holds no key and opens no network connection — the key lives
entirely in the client. (Its only evaluation surface is the explicitly experimental,
non-conformance-claiming evaluator of ADR-0007.) Any of the many such clients works — for example Claude
Code, Cline, Cursor, or Continue.

## Get the validator
Expand Down
7 changes: 5 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,10 @@ required-extension capability check
one result model ──► human or versioned JSON renderer
```

Each layer stops when the preceding layer fails. No layer evaluates a condition, resolves a
decision outcome, fetches a locator, loads an extension, or authorizes an action.
Each layer stops when the preceding layer fails. No validation layer evaluates a condition,
resolves a decision outcome, fetches a locator, loads an extension, or authorizes an action;
evaluation exists only behind the explicitly experimental surface of
[ADR-0007](adr/0007-experimental-evaluator.md).

## Packages

Expand All @@ -50,6 +52,7 @@ decision outcome, fetches a locator, loads an extension, or authorizes an action
- `internal/fssecure` opens selected local files defensively and enforces bounded regular-file reads.
- `internal/result` defines machine output version 1 and exit classes.
- `internal/describe` composes the machine descriptions the CLI and MCP server share, so neither drifts.
- `internal/evaluation` implements the EXPERIMENTAL, non-conformance-claiming JPS §§7–8 evaluator (ADR-0007).
- `internal/cli` owns commands, streams, and human/JSON rendering.
- `internal/mcp` adapts the offline core onto Model Context Protocol tools over stdio.
- `tools/sync-spec-artifacts` is an explicit maintainer-only snapshot importer.
Expand Down
8 changes: 5 additions & 3 deletions docs/authoring-lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ The runtime is a **stateless oracle**: `bytes -> result`. Every step below consu
> *Is what you now hold conformant?*

It cannot distinguish Create from Update, because both arrive as "validate this document." It keeps
no store, holds no credential, opens no network connection, and evaluates nothing. So the authoring
no store, holds no credential, and opens no network connection; its one evaluation surface is the
explicitly EXPERIMENTAL `experimental_evaluate` (ADR-0007), which claims no conformance. So the authoring
loop — the part that holds a document, edits it, saves it, and deletes it — lives entirely in the
**client**. That is the whole of ADR-0006, and everything here follows from it.

Expand Down Expand Up @@ -167,8 +168,9 @@ remains.
- No store: it never saves, names, lists, overwrites, or deletes user content.
- No credential and no network: the client owns the key; the server opens no connection.
- No path over the wire: MCP documents are values (text), never references to the server's filesystem.
- No evaluation: it validates conformance; it never resolves a condition, chooses an outcome, or
authorizes anything.
- No evaluation in the authoring loop: validation never resolves a condition, chooses an outcome,
or authorizes anything. The separate `experimental_evaluate` tool evaluates experimentally, says
so in every payload, and still authorizes nothing.

## See also

Expand Down
111 changes: 111 additions & 0 deletions docs/mcp-clients.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Connecting MCP clients

`judgment-pack mcp` serves the offline validator to any Model Context Protocol client over stdio:
no port, no credential, no network. The client supplies its own model and API key; the runtime
never sees either (see [ADR-0003](adr/0003-mcp-integration-and-testing-surface.md) and
[ADR-0006](adr/0006-authoring-lifecycle-in-the-client.md)).

Every client below launches the same binary. Install a released `judgment-pack` (see the
[README](../README.md)) and make sure the command the client launches resolves — if `judgment-pack`
is not on the PATH the client's launcher inherits, use the absolute path instead; a wrong path is
the most common failure and most clients report it poorly.

Client configuration is personal wiring, not project source, so the local config files below are
gitignored in this repository. Copy a snippet, don't commit one.

## The tools

| Tool | Purpose |
| --- | --- |
| `validate` | Carrier, structural, and semantic conformance for one document |
| `test_conformance` | Run a version-pinned conformance corpus |
| `get_schema` | The exact bundled JSON Schema, with digest |
| `describe_runtime` | Versions and artifact provenance |
| `list_examples` / `get_example` | The embedded valid fixtures, read-only |
| `experimental_evaluate` | EXPERIMENTAL (ADR-0007): the §§7–8 experiment; claims no conformance |

None of these evaluate, decide, or authorize anything, except `experimental_evaluate`, which
evaluates experimentally and says so in every payload.

## Claude Code

One command, from any directory:

```bash
claude mcp add --scope user judgment-pack -- judgment-pack mcp
```

Or per project, in `.mcp.json` at the project root (gitignored here):

```json
{
"mcpServers": {
"judgment-pack": {
"type": "stdio",
"command": "judgment-pack",
"args": ["mcp"]
}
}
}
```

Restart the session, approve the server when prompted, and confirm with `/mcp`.

## OpenAI Codex CLI

In `~/.codex/config.toml`:

```toml
[mcp_servers.judgment-pack]
command = "judgment-pack"
args = ["mcp"]
```

Or equivalently: `codex mcp add judgment-pack -- judgment-pack mcp`. Start `codex` and confirm
with `/mcp`. All tools are read-only except the experimental evaluator, so an automatic approval
mode is reasonable.

## GitHub Copilot (VS Code)

In `.vscode/mcp.json` at the workspace root (gitignored here) — note the root key is `servers`,
not `mcpServers`:

```json
{
"servers": {
"judgment-pack": {
"type": "stdio",
"command": "judgment-pack",
"args": ["mcp"]
}
}
}
```

For a user-level server instead, run **MCP: Open User Configuration** from the Command Palette.
Copilot uses MCP tools in agent mode; see the
[VS Code MCP documentation](https://code.visualstudio.com/docs/agents/reference/mcp-configuration)
and [GitHub's Copilot MCP guide](https://docs.github.com/copilot/customizing-copilot/using-model-context-protocol/extending-copilot-chat-with-mcp)
for the current UI flow. The Copilot CLI has its own
[MCP configuration](https://docs.github.com/en/copilot/how-tos/copilot-cli/customize-copilot/add-mcp-servers).

## Remote-only clients

Some clients (the ChatGPT desktop app's connectors, Microsoft Copilot Studio) cannot launch a
local process and require a public HTTPS MCP endpoint. This runtime deliberately does not serve
HTTP ([ADR-0004](adr/0004-decline-http-api.md)); reaching those clients means an external
stdio-to-HTTP bridge plus a tunnel, which is developer tooling outside this repository, or a
separately governed hosted distribution.

## Troubleshooting

- **Server fails to start:** run `judgment-pack mcp` in a terminal yourself; if it starts and
waits on stdin, the binary is fine and the problem is the client's command path or config file.
- **Config edits not picked up:** clients read MCP configuration at session start — restart the
session.
- **Old binary:** `judgment-pack version` must report a release that has the `mcp` command
(v0.1.0 or later); a stale build on the PATH is easy to pick up by mistake.

The end-to-end authoring loop these tools support — create, read, update, delete, with the
runtime as a stateless oracle — is described in [authoring-lifecycle.md](authoring-lifecycle.md),
and the agent-driven testing protocol in [agent-testing.md](agent-testing.md).
Loading