-
Notifications
You must be signed in to change notification settings - Fork 0
CXF-222: skills README + SOURCES.md provenance delta #16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,7 @@ | ||
| # Agent skills | ||
|
|
||
| The ten skills shipped in this batch: the five funnel skills and two | ||
| pre-1 skills authored against the v0.0.26 DSL contract and the 23-tool | ||
| tenant MCP surface, plus three post-funnel and cross-cutting skills | ||
| pre-1 skills authored against the v0.0.26 DSL contract and the 23-tool tenant MCP surface; validated by the eval suite, plus three post-funnel and cross-cutting skills | ||
| (`verify-connector-output`, `update-and-rollback`, | ||
| `diagnose-authoring-failure`) over the tenant connector and authoring | ||
| tool surface. Each skill's `SOURCES.md` names the pinned sources with | ||
|
|
@@ -24,3 +23,19 @@ their SHAs. | |
| The eval bundle (`evals/skills-bundle/bundle.json`) is a manifest pointing | ||
| into this directory; the skill bodies live here as the single source of | ||
| truth. | ||
|
|
||
| ## Eval evidence | ||
|
|
||
| The measurable evidence for these skills is the deterministic, committed, | ||
| executable eval harness in `evals/`: | ||
|
|
||
| - **Tier-0 scored replay** — `npm run eval:run -- --scenario evals/scenarios/tier1-directory.json --driver tier0` replays a committed scenario end-to-end against the local fixture and produces a scored JSONL record of the full S0–S11 funnel (no credentials, no network beyond localhost). | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Suggestion: the scenario cited as the skills' "measurable evidence" is the no-skills arm — |
||
| - **Unit smokes** — `npm run eval:test` runs the committed scorer/parser/stages/record/scenario/driver tests. | ||
| - **Scorer + fixture harness** — `evals/runner/score.ts` and `evals/fixture/` are the deterministic scoring and fixture layers the replay exercises. | ||
|
|
||
| The Tier-1 baseline E2E (baseline vs with-skills pass rates on a real tenant) | ||
| is structurally blocked: the public repo ships only the Tier-0 canned driver | ||
| (no real-tenant driver), and the c1-side MCP surface on eval environments | ||
| exposes no `c1_connector_authoring_*` tools. Per the carry-forward rule | ||
| established in the merged PRs #10/#11, the baseline blocker is carried | ||
| forward; no pass-rate numbers are reported because none exist. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,3 +11,8 @@ decisions 2-3); the port source below is what this skill quotes. | |
| | claude-marketplace `source-openapi-spec` | `0cc5ac2a2dbe60b430444c59e53016da2c72b3d1` | The IAM go/no-go + park-with-evidence contract this skill's handoff to `source-openapi-spec` carries. | | ||
| | c1 `docs/in-app-connector-authoring.md` | `16e0e0fbf0c999e3942f1ea0f8aef95c65e6fbc3` | The bundle caps (262144-byte source, 1048576-byte bundle) and the lifecycle contract the pre-1 judgment phase feeds. | | ||
| | In-repo SDK declarations + examples | connector-authoring `7dc673ac86616503acee3d45f98348370dc155cd` | The `baton/types.d.ts` TRAIT_* consts (`TRAIT_USER`, `TRAIT_GROUP`, `TRAIT_ROLE`, `TRAIT_APP`, `TRAIT_SECRET`) and the `examples/` access-model shapes. | | ||
| | SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | | ||
| | baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | | ||
| | Worked build — access-model shape | baton-axiomatic-openrouter#1 | The access-model shape (resource types, traits, entitlements, grants) this worked build demonstrates. | | ||
| | Worked build — access-model shape | baton-axiomatic-litellm#1 | The access-model shape (resource types, traits, entitlements, grants) this worked build demonstrates. | | ||
| | Worked build — access-model shape | baton-axiomatic-shopify | The access-model shape (resource types, traits, entitlements, grants) this worked build demonstrates. | | ||
|
Comment on lines
+16
to
+18
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Suggestion: these three rows carry no pin in the "Pin / SHA" column — |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,3 +11,5 @@ model memory). Source-of-truth precedence: (a) MCP-served guide, (b) | |
| | Authoring proto | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | RPC titles `connector_authoring_get_authoring_guide`, `connector_authoring_list_sdk_types_versions`, `connector_authoring_get_sdk_types`, `connector_authoring_list_authored_catalog_entries`, `connector_authoring_list_drafts`, `connector_authoring_create_draft`; `ListSDKTypesVersionsResponse` fields `default_tag` / `runtime_pin_matched`. | | ||
| | Lifecycle doc | c1 `2502b4cd8f59bf6614616013010ec4f0bf72f9ae` | Step 0 (guide read) and the handoff-table discipline (record every returned ID). | | ||
| | In-repo SDK declarations | connector-authoring `b7e8a616cbbb1e336b788f807a3810b08ae00bc7` | The `.d.ts` module surface (`@baton/runtime`, `@baton/helpers`, `@baton/types`) the tagged declarations cover. | | ||
| | SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Suggestion: this row is the same boilerplate applied to all ten skills, so in Also note this row overlaps the existing "In-repo SDK declarations" row (line 13, pinned at |
||
| | baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,3 +10,8 @@ gate, and the park-with-evidence contract are the locked intent of this batch | |
| | claude-marketplace `source-openapi-spec` | `0cc5ac2a2dbe60b430444c59e53016da2c72b3d1` | The sourcing port source: the 5-rung authority ladder, the programmatic IAM go/no-go gate (member/user LISTING at minimum), and park-with-evidence as a success outcome. | | ||
| | c1 `docs/in-app-connector-authoring.md` | `16e0e0fbf0c999e3942f1ea0f8aef95c65e6fbc3` | The bundle caps — the 262144-byte source limit and the 1048576-byte bundle limit (lines 727-728) — and the lifecycle contract the pre-1 judgment phase feeds. | | ||
| | In-repo SDK declarations + examples | connector-authoring `7dc673ac86616503acee3d45f98348370dc155cd` | The `baton/*.d.ts` module surface and the `examples/` spec-driven connector shapes the sourced spec feeds. | | ||
| | SDK contract (`baton/*.d.ts`) | connector-authoring git tag `v0.0.26` (sync commit `01a69d8d` "Sync baton runtime types for v0.0.26") | The `.d.ts` module surface this skill is authored against. `runtime_pin_matched`: not verifiable offline — no tenant MCP surface is reachable from the authoring env; repo tag v0.0.26 is the served default_tag when the tenant runtime pin matches (see `read-authoring-contract` for the runtime check). | | ||
| | baton-axiomatic DSL contract | baton-axiomatic `docs/DSL.md` + `runtime/baton/*.d.ts` @ v0.0.26 (`825e5516…`) | The DSL semantics ground truth the in-repo `.d.ts` are synced from. | | ||
| | OpenRouter official spec (evidence) | OpenRouter official published spec, vendored verbatim | The authoritative spec evidence for a SaaS provider with an official published spec. | | ||
| | LiteLLM runtime-generated spec (evidence) | `pip install "litellm[proxy]==1.92.0"` → `app.openapi()` | The runtime-generated spec evidence for a self-hosted OSS provider. | | ||
| | Shopify vendored minimal spec (evidence) | `allengrant/shopify_openapi` @ master, discoveredAt 2026-06-04 | The vendored minimal spec evidence for a provider without an official published spec. | | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Suggestion: |
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 Suggestion: the inserted
; validated by the eval suite,splits the "five funnel skills and two pre-1 skills … plus three post-funnel …" list mid-sentence, soplus three post-funnel …now dangles after a semicolon clause. The line also runs unwrapped while the rest of the paragraph wraps at ~70 chars. Confidence: high.Consider moving "validated by the eval suite" into its own sentence after the list, so the claim isn't attached only to the funnel/pre-1 subset and reads consistently with the
## Eval evidencesection below (which states the Tier-1 baseline is blocked).