diff --git a/skills/README.md b/skills/README.md index aca394a..4c62a7b 100644 --- a/skills/README.md +++ b/skills/README.md @@ -1,8 +1,7 @@ # Agent skills The ten skills shipped in this batch: the five funnel skills and two -pre-1 skills authored against the v0.0.26 DSL contract and the 23-tool -tenant MCP surface, plus three post-funnel and cross-cutting skills +pre-1 skills authored against the v0.0.26 DSL contract and the 23-tool tenant MCP surface; validated by the eval suite, plus three post-funnel and cross-cutting skills (`verify-connector-output`, `update-and-rollback`, `diagnose-authoring-failure`) over the tenant connector and authoring tool surface. Each skill's `SOURCES.md` names the pinned sources with @@ -24,3 +23,19 @@ their SHAs. The eval bundle (`evals/skills-bundle/bundle.json`) is a manifest pointing into this directory; the skill bodies live here as the single source of truth. + +## Eval evidence + +The measurable evidence for these skills is the deterministic, committed, +executable eval harness in `evals/`: + +- **Tier-0 scored replay** — `npm run eval:run -- --scenario evals/scenarios/tier1-directory.json --driver tier0` replays a committed scenario end-to-end against the local fixture and produces a scored JSONL record of the full S0–S11 funnel (no credentials, no network beyond localhost). +- **Unit smokes** — `npm run eval:test` runs the committed scorer/parser/stages/record/scenario/driver tests. +- **Scorer + fixture harness** — `evals/runner/score.ts` and `evals/fixture/` are the deterministic scoring and fixture layers the replay exercises. + +The Tier-1 baseline E2E (baseline vs with-skills pass rates on a real tenant) +is structurally blocked: the public repo ships only the Tier-0 canned driver +(no real-tenant driver), and the c1-side MCP surface on eval environments +exposes no `c1_connector_authoring_*` tools. Per the carry-forward rule +established in the merged PRs #10/#11, the baseline blocker is carried +forward; no pass-rate numbers are reported because none exist. diff --git a/skills/author-in-app-connector/SOURCES.md b/skills/author-in-app-connector/SOURCES.md index 1eb8a19..94abb54 100644 --- a/skills/author-in-app-connector/SOURCES.md +++ b/skills/author-in-app-connector/SOURCES.md @@ -11,3 +11,5 @@ model memory). Source-of-truth precedence: (a) MCP-served guide, (b) | Authoring proto | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | RPC titles for every funnel step (`connector_authoring_create_draft`, `connector_authoring_create_draft_source_upload`, `connector_authoring_finalize_draft_source_upload`, `connector_authoring_build_bundle`, `connector_authoring_get_run`, `connector_authoring_run_draft_test_sync`, `connector_authoring_get_test_run_evidence`, `connector_authoring_provision_connector`, `connector_authoring_deploy_connector_instance`, `connector_authoring_mint_approval_token`); `MintApprovalTokenResponse.activation_url`. | | Lifecycle doc | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | The 12-step order and stop-if gates; the handoff-table discipline (fill every row from the tool response of the step that produces it); step 11's human-OWNER handoff. | | In-repo SDK declarations | connector-authoring `b7e8a616cbbb1e336b788f807a3810b08ae00bc7` | The `.d.ts` module surface the funnel's source files are authored against. | +| SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | +| baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | diff --git a/skills/build-and-test/SOURCES.md b/skills/build-and-test/SOURCES.md index 5d0662e..4d19d80 100644 --- a/skills/build-and-test/SOURCES.md +++ b/skills/build-and-test/SOURCES.md @@ -11,3 +11,5 @@ model memory). Source-of-truth precedence: (a) MCP-served guide, (b) | Authoring proto | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | `DraftSourceUploadTarget` (`url` + `required_headers`); `CreateDraftSourceUpload` / `FinalizeDraftSourceUpload` (declared files with `size_bytes`, caps); `GetRun` (build runs only); `RunDraftTestSync` / `GetTestRunEvidence` (durable PASS/FAIL row, `NotFound` while pending). | | Lifecycle doc | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | Step 2 (upload dance with `wc -c`), step 4–5 (build + poll), step 9–10 (draft test + PASS evidence; fresh `test_run_id` on re-run). | | In-repo SDK declarations | connector-authoring `b7e8a616cbbb1e336b788f807a3810b08ae00bc7` | The `.d.ts` module surface the uploaded source is authored against. | +| SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | +| baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | diff --git a/skills/deploy-and-activate/SOURCES.md b/skills/deploy-and-activate/SOURCES.md index dd25352..1d02e66 100644 --- a/skills/deploy-and-activate/SOURCES.md +++ b/skills/deploy-and-activate/SOURCES.md @@ -11,3 +11,5 @@ model memory). Source-of-truth precedence: (a) MCP-served guide, (b) | Authoring proto | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | `MintApprovalTokenResponse` (`token_id`, `expires_at`, `activation_url`; `expires_in_seconds` gt 0, lte 14400); `RevisionStatus` enum (`REVISION_STATUS_ACTIVE` = 1, `activation_epoch` on the ACTIVE row); `ProvisionConnector` / `DeployConnectorInstance` RPC titles. | | Lifecycle doc | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | Step 8's `c1_connector_service_get` + `c1_connector_service_update` with `updateMask: "config"` and the `EnvConfig` `configuration` JSON (keys must match `config("field-name")`); the empty-`stringValue`-deletes-the-secret warning; step 11a–11d (deploy → mint → hand off to human OWNER → poll `list_revision_summaries` until `REVISION_STATUS_ACTIVE` + record `activation_epoch` → `c1_connector_service_force_sync`); "What success looks like" (`SYNC_STATUS_DONE` via `c1_connector_service_get`; a subsequent `sync_disabled` is normal); the common-failures rows (`credential re-entry required`, `activation evidence is unsatisfied`). | | In-repo SDK declarations | connector-authoring `b7e8a616cbbb1e336b788f807a3810b08ae00bc7` | The `config("field-name")` names the `configuration` keys must match (see `examples/http/connector.ts`: `base-url`, `account-email`, `api-token`). | +| SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | +| baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | diff --git a/skills/design-access-model/SOURCES.md b/skills/design-access-model/SOURCES.md index d500510..e5ac149 100644 --- a/skills/design-access-model/SOURCES.md +++ b/skills/design-access-model/SOURCES.md @@ -11,3 +11,8 @@ decisions 2-3); the port source below is what this skill quotes. | claude-marketplace `source-openapi-spec` | `0cc5ac2a2dbe60b430444c59e53016da2c72b3d1` | The IAM go/no-go + park-with-evidence contract this skill's handoff to `source-openapi-spec` carries. | | c1 `docs/in-app-connector-authoring.md` | `16e0e0fbf0c999e3942f1ea0f8aef95c65e6fbc3` | The bundle caps (262144-byte source, 1048576-byte bundle) and the lifecycle contract the pre-1 judgment phase feeds. | | In-repo SDK declarations + examples | connector-authoring `7dc673ac86616503acee3d45f98348370dc155cd` | The `baton/types.d.ts` TRAIT_* consts (`TRAIT_USER`, `TRAIT_GROUP`, `TRAIT_ROLE`, `TRAIT_APP`, `TRAIT_SECRET`) and the `examples/` access-model shapes. | +| SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | +| baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | +| Worked build — access-model shape | baton-axiomatic-openrouter#1 | The access-model shape (resource types, traits, entitlements, grants) this worked build demonstrates. | +| Worked build — access-model shape | baton-axiomatic-litellm#1 | The access-model shape (resource types, traits, entitlements, grants) this worked build demonstrates. | +| Worked build — access-model shape | baton-axiomatic-shopify | The access-model shape (resource types, traits, entitlements, grants) this worked build demonstrates. | diff --git a/skills/diagnose-authoring-failure/SOURCES.md b/skills/diagnose-authoring-failure/SOURCES.md index 8871c61..2d6085c 100644 --- a/skills/diagnose-authoring-failure/SOURCES.md +++ b/skills/diagnose-authoring-failure/SOURCES.md @@ -11,3 +11,5 @@ model memory). Source-of-truth precedence: (a) MCP-served guide, (b) | Authoring proto | c1 `2e5f53eb441a93087d9754085ca17a5061e125ea` | `GetTestRunEvidence` (poll `(catalog_id, revision_id, test_run_id)`; `result` PASS/FAIL + `error` on the evidence row). | | Lifecycle doc | c1 `2e5f53eb441a93087d9754085ca17a5061e125ea` | The Debugging section: the common-failures table (all nine rows), the draft-test FAIL reading (evidence row authoritative; PASS requires `ConnectionOK`, `HostCallOK`, no read error and no write attempt, config version handle match, runtime image digest match), and where logs live (product connector activity and sync logs; the status row `c1_connector_service_get` -> `status.status`, `status.lastError`). | | baton-admin `diagnose-connector-failure` | `6fe6886f607ed0d2e48a616c30e7ce4bffc32489` | The taxonomy approach: start from the symptom, classify the failure surface, collect the smallest evidence, keep the diagnosis focused on symptom/evidence/owner/next-fix/rerun-target, and the output contract (failure summary, classified surface, evidence used, likely owner, one concrete next fix, smallest verification command). | +| SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | +| baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | diff --git a/skills/read-authoring-contract/SOURCES.md b/skills/read-authoring-contract/SOURCES.md index 278a331..9a19191 100644 --- a/skills/read-authoring-contract/SOURCES.md +++ b/skills/read-authoring-contract/SOURCES.md @@ -11,3 +11,5 @@ model memory). Source-of-truth precedence: (a) MCP-served guide, (b) | Authoring proto | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | RPC titles `connector_authoring_get_authoring_guide`, `connector_authoring_list_sdk_types_versions`, `connector_authoring_get_sdk_types`, `connector_authoring_list_authored_catalog_entries`, `connector_authoring_list_drafts`, `connector_authoring_create_draft`; `ListSDKTypesVersionsResponse` fields `default_tag` / `runtime_pin_matched`. | | Lifecycle doc | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | Step 0 (guide read) and the handoff-table discipline (record every returned ID). | | In-repo SDK declarations | connector-authoring `b7e8a616cbbb1e336b788f807a3810b08ae00bc7` | The `.d.ts` module surface (`@baton/runtime`, `@baton/helpers`, `@baton/types`) the tagged declarations cover. | +| SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | +| baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | diff --git a/skills/source-openapi-spec/SOURCES.md b/skills/source-openapi-spec/SOURCES.md index 58811d4..f76bb1c 100644 --- a/skills/source-openapi-spec/SOURCES.md +++ b/skills/source-openapi-spec/SOURCES.md @@ -10,3 +10,8 @@ gate, and the park-with-evidence contract are the locked intent of this batch | claude-marketplace `source-openapi-spec` | `0cc5ac2a2dbe60b430444c59e53016da2c72b3d1` | The sourcing port source: the 5-rung authority ladder, the programmatic IAM go/no-go gate (member/user LISTING at minimum), and park-with-evidence as a success outcome. | | c1 `docs/in-app-connector-authoring.md` | `16e0e0fbf0c999e3942f1ea0f8aef95c65e6fbc3` | The bundle caps — the 262144-byte source limit and the 1048576-byte bundle limit (lines 727-728) — and the lifecycle contract the pre-1 judgment phase feeds. | | In-repo SDK declarations + examples | connector-authoring `7dc673ac86616503acee3d45f98348370dc155cd` | The `baton/*.d.ts` module surface and the `examples/` spec-driven connector shapes the sourced spec feeds. | +| SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | +| baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | +| OpenRouter official spec (evidence) | OpenRouter official published spec, vendored verbatim | The authoritative spec evidence for a SaaS provider with an official published spec. | +| LiteLLM runtime-generated spec (evidence) | `pip install "litellm[proxy]==1.92.0"` → `app.openapi()` | The runtime-generated spec evidence for a self-hosted OSS provider. | +| Shopify vendored minimal spec (evidence) | `allengrant/shopify_openapi` @ master, discoveredAt 2026-06-04 | The vendored minimal spec evidence for a provider without an official published spec. | diff --git a/skills/update-and-rollback/SOURCES.md b/skills/update-and-rollback/SOURCES.md index 24a4c42..619fdba 100644 --- a/skills/update-and-rollback/SOURCES.md +++ b/skills/update-and-rollback/SOURCES.md @@ -10,3 +10,5 @@ model memory). Source-of-truth precedence: (a) MCP-served guide, (b) | Authoring proto | c1 `2e5f53eb441a93087d9754085ca17a5061e125ea` | `MintApprovalToken` (`expires_in_seconds` 1-14400, `activation_url`); `ListRevisionSummaries` + `RevisionStatus` enum (`REVISION_STATUS_ACTIVE` = 1, `activation_epoch` on the ACTIVE row). The proto exposes no rollback RPC - its service comment notes rollback is the REST-only endpoint; the rollback contract lives in the lifecycle doc row below. | | Lifecycle doc | c1 `2e5f53eb441a93087d9754085ca17a5061e125ea` | The "Updating and rolling back a live connector" section: the image-digest reuse rule; the rotation STOP verbatim (`serve image does not match the revision-pinned runtime image`; do not clear runtime fields, call the provisioner directly, or mutate the deployment or AWS resources; record tenant/catalog/app/connector/target-revision IDs; escalate to Connector Authoring / managed-runtime engineering); the REST-only OWNER-gated rollback body and the strictly-greater-activation-epoch pointer move. | | c1 Go source (same pin) | c1 `2e5f53eb441a93087d9754085ca17a5061e125ea` | The `SYNC_STATUS_ERROR` / `SYNC_STATUS_DISABLED` terminal-state semantics: `ConnectorStatusToAPI` derives DISABLED only from an ERROR-classified sync; `sync_disabled_reason` distinguishes the data-anomaly auto-pause from deliberate pauses. | +| SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | +| baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | diff --git a/skills/verify-connector-output/SOURCES.md b/skills/verify-connector-output/SOURCES.md index 1dc5aec..97acf11 100644 --- a/skills/verify-connector-output/SOURCES.md +++ b/skills/verify-connector-output/SOURCES.md @@ -11,3 +11,5 @@ model memory). Source-of-truth precedence: (a) MCP-served guide, (b) | Lifecycle doc | c1 `2e5f53eb441a93087d9754085ca17a5061e125ea` | "What success looks like": `SYNC_STATUS_DONE` via `c1_connector_service_get` (a subsequent `sync_disabled` is normal); the UI spot-check path `/admin/connector///`. | | Marketplace `probe-contracts.md` | claude-marketplace `0cc5ac2a2dbe60b430444c59e53016da2c72b3d1` | The assertion-inventory adaptations: per-resource-type sync counts; count parity against the live API, not the seed list; ID stability across re-sync. | | c1 Go source (same pin) | c1 `2e5f53eb441a93087d9754085ca17a5061e125ea` | The `SYNC_STATUS_DISABLED` semantics: `ConnectorStatusToAPI` derives DISABLED only from an ERROR-classified sync; `sync_disabled_reason` distinguishes the data-anomaly auto-pause (`Sync paused due to significant drop in sync data` prefix) from deliberate pauses (`system`, `system-customer-opt-out`). | +| SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | +| baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | diff --git a/skills/write-connector-source/SOURCES.md b/skills/write-connector-source/SOURCES.md index f827755..d904218 100644 --- a/skills/write-connector-source/SOURCES.md +++ b/skills/write-connector-source/SOURCES.md @@ -18,3 +18,7 @@ at the pinned SHA still carry the presence-only capabilities wording. | `runtime-gotchas.md` | claude-marketplace `0cc5ac2a2dbe60b430444c59e53016da2c72b3d1` | The traps this skill quotes: opaque config refs (`ref \|\| default` never fires for transports - trap 1), ES5 bundle target (no `u` regex flag, goja quirks - trap 3), offset-pagination `totalPath` termination (trap 6), list-call scoping (trap 7). | | shopify `Makefile` | baton-axiomatic-shopify `6ea2834043832f7bb21ca59111de2cf021c17518` | The `capabilities` target: `env $(CAPABILITIES_ENV) BATON_CONFIG= baton-axiomatic capabilities > baton_capabilities.json` with `CAPABILITIES_ENV ?= BATON_BASE_URL=https://placeholder.myshopify.com BATON_TOKEN=placeholder`; capability generation never makes network calls. | | Investigation (internal artifact, not a fetchable pin) | `/shared/src-hv96/investigation.md` | §3.1 source-file contract, §5 skill 5, §8 build order step 4. | +| SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | +| baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | +| baton-admin DSL skill | baton-admin `author-js-dsl-connector` @ `6fe6886f607ed0d2e48a616c30e7ce4bffc32489` | The JS DSL authoring rules this skill's source-file contract aligns with. | +| baton-admin DSL skill | baton-admin `author-auth-config-surface` @ `6fe6886f607ed0d2e48a616c30e7ce4bffc32489` | The auth-config surface rules this skill's config-schema contract aligns with. |