Skip to content

feat(runtime): add error details, method-safety mutation, search terms - #171

Merged
samzong merged 7 commits into
mainfrom
fix/runtime-ux-round2
Aug 30, 2026
Merged

feat(runtime): add error details, method-safety mutation, search terms#171
samzong merged 7 commits into
mainfrom
fix/runtime-ux-round2

Conversation

@samzong

@samzong samzong commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Round 2 of runtime UX contracts for agent-facing generated CLIs, 7 commits:

  • feat(runtime): usage errors carry an optional detail built only from spec metadata (enum sets, required flag/field names, output formats) via an explicit constructor; never echoes user input; cobra unknown command/flag paths stay detail-free.
  • feat(runtime): API errors surface a layered detail: declared JSON fields (message, error, error.message, detail, depth <= 2, sanitized, <= 240 runes, bodies > 32KB skipped) -> known_errors cause matching the status -> unchanged status + hint. Raw response bodies are never emitted.
  • feat(runtime): catalog mutation follows RFC 9110 method safety (GET/HEAD/OPTIONS/TRACE = read, other declared methods = write, unknown only for undeterminable); overlay mutation: read|write overrides every inference.
  • feat(overlay): search_terms flow overlay -> CommandSpec -> generated literal -> catalog -> search index (identity synonym at summary weight) -> generated Skill docs.
  • feat(codegen): body.flags skips nested object properties instead of rejecting the whole body; skipped fields land in body.set_only_fields, --set/--set-str help, and keep local required validation.
  • fix(runtime): detail sanitization strips all non-graphic runes (C1 controls, bidi overrides); required set-only body fields are validated locally.

Verification

  • make check (fmt-check, vet, golangci-lint, tests) — pass
  • go test -race -count=1 ./... — pass
  • go vet ./... — pass
  • Isolated real-behavior verification: scratch copy of a downstream generated CLI (tokener) with replace to this branch, regenerated via lathe bootstrap; __lathe verify --json 20/20 checks ok; verified enum detail output, declared server message + known_errors fallback against a local mock upstream, mutation values, search_terms hit, and --name demo --dry-run producing {"name":"demo"} with nested limits skipped.

Compatibility

  • SchemaVersion 13 -> 16 and CatalogSchemaVersion 20 -> 23: downstream CLIs must regenerate after bumping (enforced explicitly by AssertSchema).
  • Catalog/envelope additions are all optional fields (error.detail, search_terms, body.set_only_fields).
  • mutation value semantics: operations previously reported unknown for REST write methods now report write; consumer handling is unchanged, classification is more precise (documented in docs/contracts.md).
  • Docs updated: docs/contracts.md, docs/cli-usage.md.

Checklist

  • Tests or focused verification cover the changed surface.
  • User-facing behavior changes are documented.
  • Generated output under internal/generated/, .cache/, and ad-hoc skills/<cli-name>/ directories is not committed.
  • Commits are signed off when this is ready to merge.

Old contract: usage errors carried only code, message, and hint; agents
could not learn which values a flag accepts or which required field was
missing without re-reading the catalog.

New contract: LatheError gains an optional detail field, filled only by
validation call sites via the explicit WithUsageDetail constructor and
built exclusively from spec metadata (flag names, declared value sets,
required field names). Detail never echoes user-provided values, is
sanitized to a single line, and is bounded to 240 runes. Human output
prints a Detail: line between Error: and Hint:; JSON/YAML envelopes add
detail (omitted when empty). Covered call sites: enum violations
(--range accepts: 7, 30), missing required flags/params/body fields
(missing required: name), and unsupported output formats. Cobra unknown
command and unknown flag paths produce no detail because their causes
contain user input; the machine error contract test now asserts detail
stays empty there.

Reason: agents should correct invalid invocations from the error alone
without guessing, while keeping the no-echo leakage boundary intact.

Signed-off-by: samzong <samzong.lu@gmail.com>
Old contract: every HTTP failure surfaced only as API request failed
plus a status-based hint; the server's declared explanation was
discarded and agents could not distinguish a revoked key from a generic
403.

New contract: api_error detail is filled from the first available
layer. Layer 1: when the response Content-Type is JSON and a string is
extractable from a declared field (message; error as a string or
error.message; detail; depth <= 2), detail is that message, stripped of
control characters and bounded to 240 runes; bodies over 32KB are never
parsed. Layer 2: when extraction fails and the command catalog declares
a known_errors entry matching the status, detail is its cause; layers
never stack. Layer 3: unchanged status plus hint. The error message
stays API request failed, HTTPError now records the response
Content-Type, and the raw body is never emitted in any layer; the
upstream-secret negative cases keep proving non-JSON bodies do not
leak.

Reason: agents acting on failed API calls need the server's declared
cause without lathe leaking arbitrary response bodies.

Signed-off-by: samzong <samzong.lu@gmail.com>
Old contract: catalog mutation was read for GET/HEAD, GraphQL templates
classified query vs mutation, and every other method surfaced as
unknown, forcing agents to treat ordinary POST/PUT/PATCH/DELETE
operations as undecidable.

New contract: RFC 9110 safe methods (GET, HEAD, OPTIONS, TRACE) are
read; every other declared method defaults to write; unknown is
reserved for operations with no determinable method (and workflows with
no steps). A new overlay field mutation: read|write explicitly
overrides every inference, flowing overlay -> CommandSpec.Mutation ->
generated literal -> catalog; invalid values are rejected at overlay
merge and validation. GraphQL template classification and workflow
heaviest-step aggregation are unchanged.

SchemaVersion bumps 13 -> 14 and CatalogSchemaVersion bumps 20 -> 21
following the repository convention that a new CommandSpec field and a
catalog behavior change require regeneration; consumers that treated
unknown as dangerous now see write for the same operations and should
handle it identically with more precise information.

Reason: agents need an accurate read/write signal instead of a
defensive unknown for every REST mutation.

Signed-off-by: samzong <samzong.lu@gmail.com>
Old contract: search matched only spec-derived fields (operation id,
use, aliases, shortcuts, paths, summary, description, flags); domain
vocabulary like spend could not reach usage-summary without editing
upstream summaries.

New contract: overlay operations accept search_terms: ["..."], carried
end to end: overlay parser -> CommandSpec.SearchTerms -> generated
literal -> catalog search_terms field -> search index -> generated
Skill module reference (Search terms: ... per operation). Search treats
each curated term as an identifying synonym weighted like summary text
(weight 30), so a single term surfaces the command while exact
command-name and operation-id matches still outrank it.

SchemaVersion bumps 14 -> 15 and CatalogSchemaVersion bumps 21 -> 22
per the repository convention for new CommandSpec and catalog fields;
downstream CLIs must be regenerated to pick up search_terms, recorded
in docs/contracts.md.

Reason: agents search by user intent vocabulary; curated synonyms close
the gap between spec wording and how operators actually ask.

Signed-off-by: samzong <samzong.lu@gmail.com>
Old contract: enabling overlay body.flags on a request body containing
any nested object property rejected the entire body, so commands like
keys create lost every typed flag (including --name) because of one
nested limits field.

New contract: nested object properties (including arrays of objects)
are skipped instead of fatal. Top-level scalar and scalar-array
properties still expand into typed flags; skipped fields are recorded
as RequestBody.SetOnlyFields, flow into the generated literal, surface
in the catalog as body.set_only_fields, and are named in the
--set/--set-str flag help so callers know those fields use --set,
--set-str, or --file. A body whose properties are all nested objects
still fails codegen with an explicit message, and the other rejection
paths (maps, oneOf/anyOf/allOf, multipart, GraphQL templates,
unsupported root types, flag collisions) are unchanged. Full nested
flattening (e.g. --limits-max-budget-usd) is documented as a possible
future extension only.

SchemaVersion bumps 15 -> 16 and CatalogSchemaVersion bumps 22 -> 23
for the new RequestBody and catalog body fields.

Reason: one nested field should not cost a command every typed flag
agents rely on for safe body construction.

Signed-off-by: samzong <samzong.lu@gmail.com>
Old contract: sanitizeErrorDetail replaced only C0 controls and DEL, so
C1 control characters (CSI U+009B, OSC U+009D) and Unicode bidi
overrides (U+202E) from a hostile server's declared error message
passed through to the human Detail: line and the JSON/YAML envelope,
contradicting the documented control-characters-stripped guarantee.

New contract: every non-graphic rune (unicode.IsGraphic == false,
covering C0, C1, DEL, and format characters including bidi overrides)
is replaced with a space before collapsing and bounding the detail.

Reason: detail is the one channel that intentionally carries
server-influenced text; it must not be able to carry terminal escape
or display-spoofing sequences.

Signed-off-by: samzong <samzong.lu@gmail.com>
Old contract: after nested object properties were skipped into
SetOnlyFields, a schema-required nested field lost local required
validation entirely: typed scalar required fields in the same body were
checked before the request, while an omitted required nested field was
only caught by the server round-trip.

New contract: validateRequiredBodyParams also fails with the same
missing required: <name> usage detail when a schema-required field
listed in SetOnlyFields is absent from the resolved body; providing it
via --set, --set-str, or --file passes as before.

Reason: required enforcement must be consistent across typed and
set-only fields, or agents are back to guessing part of the body
shape.

Signed-off-by: samzong <samzong.lu@gmail.com>
@codspeed-hq

codspeed-hq Bot commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will degrade performance by 49.98%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 21 regressed benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
large 996.3 µs 2,255.3 µs -55.83%
BenchmarkFormatTableInferredColumns 471.3 µs 986 µs -52.2%
yaml 2.6 ms 5.4 ms -52%
BenchmarkCatalogJSON 1.5 ms 3.1 ms -51.82%
large 1.2 ms 2.6 ms -51.41%
json 658.3 µs 1,354.4 µs -51.4%
BenchmarkFindCatalogCommand 4.4 µs 9 µs -51.02%
small 99.9 µs 203.8 µs -50.97%
large 808.5 µs 1,639.7 µs -50.7%
miss 1.9 ms 3.8 ms -50.58%
table 379.9 µs 767.5 µs -50.5%
hit 1.9 ms 3.9 ms -50.19%
json-large 1.7 ms 3.4 ms -49.96%
json-small 129.6 µs 255.4 µs -49.24%
small 74.6 µs 144.9 µs -48.49%
BenchmarkParseNormalize 3.1 ms 6 ms -48.08%
large 707.2 µs 1,360.1 µs -48%
yaml-large 8.1 ms 15.5 ms -47.9%
small 76.7 µs 143.3 µs -46.51%
BenchmarkBuildFlat 1.1 ms 2 ms -45.88%
... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing fix/runtime-ux-round2 (edfa5f1) with main (7ec8e13)

Open in CodSpeed

@samzong
samzong merged commit 4677bfa into main Aug 30, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant