CXF-274: narrow the skills' prose to what the code actually requires

[manuel-ts-14]

Summary

Two skills state contracts more loosely than the code enforces them, so correct work fails on encoding and authors hit avoidable compile errors. Both commits are additive prose narrowing the skill to the implementation — no behaviour change, and every substring evals/runner/skills_bundle.test.ts asserts is preserved.

Found by running the pre-1 and authoring skills blind against this repo's own fixture: agents given only the SKILL.md, the provider brief, and the live spec endpoint, barred from reading evals/scenarios/, evals/results/, and evals/runner/, then graded by replicating the scorer's comparison logic. All runs reached the correct decision — every issue below is the contract around the judgment, not the judgment.

1. source-openapi-spec: type the Output Contract (c30ecc0)

The contract names field names but never their types, shapes, or granularity, while evals/runner/stages.ts compares three of them by strict equality or exact set membership.

Field	Scorer	What a correct run emitted
authority_rung	nonEmptyString (stages.ts:489, typed stages.ts:59)	1 — the skill teaches a numbered ladder, so "rung 1" reads as an integer. spec_bytes in the same object genuinely is a number, so nothing signals which is which.
spec_version_checked	!== (stages.ts:517)	info.version + OpenAPI version + fetch timestamp + sha256, reasoning the hash makes a park reproducible. Fails against "1.2.0" for being more precise.
missing_paths	Array.includes (stages.ts:518)	descriptions of missing capability families — type-conformant string[], fails against literal path templates.

The pre1-directory-proceed run scored P0/P1/P2 PASS, P3 FAIL, purely on authority_rung being a number.

If you'd rather fix authority_rung on the scorer side (accept a number) than in the prose, say so and I'll flip it.

2. write-connector-source: name the Grant type and provisioning nesting (0d3979d)

design-access-model routinely emits provisionable: true, but this skill never mentions grant, revoke, or provisioningResponse, and its only worked skeleton is read-only by its own README. Deriving the provisioning half from baton/runtime.d.ts hits two declaration-level traps:

• Two different Grant types ship. baton/helpers.d.ts imports Grant from ./types (baton/types.d.ts:328, no "@type") — that is what provisioningResponse.grant({grants: [...]}) takes. The proto-shaped Grant at baton/sdk-types.d.ts:653 requires "@type". Writing a proto-JSON literal, which the capabilities section trains you toward, fails with TS2353. A blind run hit exactly this and burned a fix cycle.
• Two provisioning spellings both typecheck. ResourceTypeSpecShape (runtime.d.ts:2055-2058) accepts a top-level grant/revoke and a nested provisioning: {grant, revoke}. The runtime-facing RuntimeResourceTypeSpec (runtime.d.ts:1738) carries only provisioning, so that is the nesting named here. Worth confirming — if the flat form also registers, this is just a preference; if it doesn't, it fails silently rather than erroring, which is why it's worth stating either way.

Both verified against a provisioning connector authored from this skill and typechecked clean.

Verification

• npm run eval:test → exit 0 (139 pass / 0 fail)
• npm run typecheck → exit 0
• Locked substrings preserved: 8/8 for source-openapi-spec, 17/17 for write-connector-source
• Bodies at 93 and 197 lines against the 200-line bound (skills_bundle.test.ts:194)

What this PR deliberately does not fix

write-connector-source is at 197 of 200 lines, so the fuller worked provisioning guidance it still lacks — node/walk shapes, the 204-no-body revoke case, CAPABILITY_PROVISION placement — does not fit without a references/ mechanism these skills don't currently use. Commit 2 spends the remaining budget on the two traps that cost a build; the rest needs a structural call from you. Recorded on CXF-274 along with the full finding set.

🤖 Generated with Claude Code

[linear-code]

CXF-274

[github-actions]

🟡 Suggestion: the string-not-number typing is right (stages.ts:489 uses nonEmptyString), but the example value "1" diverges from the repo's own golden artifacts, which all use the descriptive rung name — "official published spec at a stable URL" in evals/runner/drivers/tier0/canned-pre1-directory-proceed/pre1.json:26, score.test.ts:239, and stages.test.ts:307. Both forms pass P3, so nothing breaks, but a skill that is being written to match the scorer will now teach an encoding no fixture uses. Consider e.g. "official published spec at a stable URL" for rung 1 (or "1" — official published spec at a stable URL) so prose and fixtures agree. (confidence: high on the divergence, medium on which form you want as canonical)

[github-actions]

🟡 Suggestion: P4 also rejects an empty missing_paths array (stages.ts:511, asserted by stages.test.ts:459 "P4 fails when missing_paths is empty"), and every element must be a non-empty string. The new prose spells out "both non-empty" for vendor_doc/revisit_trigger but leaves missing_paths as just "array of strings", so the one array-shaped gate is the one requirement still undocumented — evals/README.md:134 already states it. Suggest "(array of at least one non-empty string)". (confidence: high)

[github-actions]

General PR Review: CXF-274: narrow the skills' prose to what the code actually requires

Blocking Issues: 0 | Suggestions: 1 | Threads Resolved: 0
Criteria: Criteria status: none loaded - .claude/skills/ci-review.md was not found at trusted base dd2e3ae0b08a.
Review mode: incremental since c30ecc05
View review run

Review Summary

The new commit 0d3979d adds a provisioning rule to skills/write-connector-source/SKILL.md — provisioningResponse.grant/.revoke usage, the @baton/types vs sdk-types Grant split, and the nested provisioning spelling for the grant/revoke walkers — plus a matching anti-pattern line. I scanned the full PR diff (both SKILL.md files) for security and correctness; it is documentation-only with no executable or dependency changes, and there are no security or correctness issues. The Grant claims verify exactly against the declarations (baton/helpers.d.ts:308-316 takes Grant imported from ./types, whose Grant at types.d.ts:328 has no proto discriminator, unlike sdk-types.d.ts:653), and the file sits at 197 of the 200-line bound asserted by skills_bundle.test.ts:194. The two prior source-openapi-spec suggestions are untouched by this increment and are not re-flagged.

Security Issues

None found.

Correctness Issues

None found.

Suggestions

• skills/write-connector-source/SKILL.md:32-33 — the RuntimeResourceTypeSpec rationale for the nested provisioning spelling cites a type the authoring surface never consumes; ConnectorSpec.resourceTypes is AuthoredResourceTypeSpec[] (baton/runtime.d.ts:1954), which accepts the flat grant/revoke too (1934-1942).

Reviewed head 0d3979d912abfc5009e448dfde925e8956ba1efd against base dd2e3ae0b08ac400ca9f86fb8365b4109b59ae1b. The machine-readable review-state marker could not be emitted this run (the review sandbox rejects the brace-plus-quote JSON literal on every available posting path), so the next run will fall back to a full review of this PR rather than an incremental one.

Prompt for AI agents

Verify each finding against the current code and only fix it if needed.

## Suggestions

In `skills/write-connector-source/SKILL.md`:
- Around line 32-33: Rule 6 justifies the nested provisioning spelling with the claim
  that the runtime-facing `RuntimeResourceTypeSpec` carries only that nesting. That
  type (`baton/runtime.d.ts:1724`) is declared but referenced nowhere in the authoring
  surface. The type an authored connector default export is actually checked against
  is `AuthoredResourceTypeSpec` via `ConnectorSpec.resourceTypes`
  (`baton/runtime.d.ts:1954`), and `AuthoredResourceTypeSpec` (`1934-1942`) declares
  top-level `grant`/`revoke` alongside the nested provisioning field — as does
  `ResourceTypeSpecShape` (`2050-2058`) used by `resourceType()`. So the cited type
  cannot be what makes the flat form wrong, and an agent that reads `runtime.d.ts`
  will find the flat form on the authored type, which directly contradicts the rule.
  Fix by either (a) dropping the rationale clause and stating the nested spelling as
  required with no type citation, or (b) replacing the citation with whatever actually
  rejects or ignores the flat form at build or register time, once that is confirmed.
  Keep the change within the 200-line bound asserted by
  `evals/runner/skills_bundle.test.ts:194` (the file is currently at 197 lines).

[github-actions]

No blocking issues found.

[github-actions]

🟡 Suggestion: the RuntimeResourceTypeSpec citation isn't the type an author's code is checked against. ConnectorSpec.resourceTypes is readonly AuthoredResourceTypeSpec[] (baton/runtime.d.ts:1954), and AuthoredResourceTypeSpec (1934-1942) carries top-level grant/revoke and nested provisioning — same as ResourceTypeSpecShape (2050-2058). RuntimeResourceTypeSpec (1724) is declared but referenced nowhere in the authoring surface, so it can't be what makes the flat form wrong. Consider either dropping the rationale clause and stating the nesting as the required spelling outright, or replacing it with whatever actually rejects the flat form at build/register time — otherwise an agent that reads runtime.d.ts and finds the flat form on the authored type has a directly contradicting signal. (Medium confidence: the nesting guidance itself may well be correct; only the stated justification doesn't hold.)

[github-actions]

No blocking issues found.

[mateoHernandez123]

Approving. The sourcing field types and the @baton/types Grant (no "@type") match what the scorer and the helpers actually enforce; that is the useful part of the PR.

Nit already on the bot thread, not blocking: rule 6 cites RuntimeResourceTypeSpec as the reason walkers must be nested, but authored connectors are checked against AuthoredResourceTypeSpec, which accepts top-level grant/revoke as well. Keep the nested spelling as the convention if you want, and drop or retarget that citation so an agent reading runtime.d.ts is not taught a type it will never hit.