Skip to content

ci: add spec drift check script and workflow - #111

Merged
hewliyang merged 4 commits into
hly/chore/cleanupfrom
hly/chore/spec-drift
Sep 8, 2026
Merged

hewliyang merged 4 commits into
hly/chore/cleanupfrom
hly/chore/spec-drift

Conversation

@hewliyang

@hewliyang hewliyang commented Sep 8, 2026 •

Copy link
Copy Markdown
Member

Stacked on #110.

Summary

Adds scripts/spec_drift.py, a structural comparison between the SDK types and https://reducto.ai/openapi.json, and a workflow that runs it on pull requests.

How it works

Schema names in the spec do not match SDK type names, so nothing is matched by name.

  1. Anchor on endpoints. The script parses src/reducto/resources/*.py and _client.py with ast and finds each self._post("/path", body=maybe_transform(..., ParamsType), cast_to=ResponseType, query=...). New spec endpoints with no SDK method are reported too.
  2. Normalize both sides into one shape model (object / enum / primitive / array / map / union). Spec: $ref, anyOf/oneOf/allOf, const, format: binary. SDK: runtime introspection of TypedDicts and pydantic models, honoring Required[], PropertyInfo(alias=...), Literal, Optional, Iterable, SequenceNotStr, FileTypes.
  3. Walk in parallel from request, query, and response of each endpoint. Union members are paired by structural similarity, then compared.

Noise controls: response required drift is skipped when the spec field is nullable (SDK models those as Optional = None); each (spec schema, SDK class) pair is compared once, in spec path order; /upload merges body and query since one params type feeds both.

Spec snapshot

spec/openapi.json is a committed copy of the public spec. The PR check runs against it, so it is reproducible, needs no network, and an upstream API change cannot turn an unrelated PR red.

Workflows

  • .github/workflows/spec-drift.yml replaces sdksync from reductoai/sdk-utils. It runs the script against the snapshot on PR open, reopen, and push. The full report goes to the job summary. The job fails when drift is found.
  • Refreshing the snapshot is manual: uv run python scripts/spec_drift.py --update-snapshot, sync the SDK, commit both together.

Usage

uv run python scripts/spec_drift.py                    # committed snapshot
uv run python scripts/spec_drift.py --live              # https://reducto.ai/openapi.json
uv run python scripts/spec_drift.py --spec other.json   # any URL or file
uv run python scripts/spec_drift.py --update-snapshot   # refresh spec/openapi.json, then check
uv run python scripts/spec_drift.py --json
uv run python scripts/spec_drift.py --ignore extra --warn-only

Current result against the snapshot (v1.12.12-114)

57 drift items: endpoint=2, enum=4, extra=1, missing=50. Highlights: missing DELETE /job/{job_id} and DELETE /upload/{file_id}; parse settings embed_pdf_metadata_dpi, extract_document_properties, hybrid_vpc, tenant_throttling, TableAgentic.mode; extract queue_priority, citations.parent_block; split deep_split, auto_partition; classify category_groups, model; queue_priority enum lacks standard; response_type discriminator missing on every response model.

Because of this, the check will be red on this PR until #113 (the SDK sync) lands on top. If you prefer a soft check in the meantime, change the run step to --warn-only.

Verification

  • ruff, pyright (strict), mypy clean on the script
  • ./scripts/lint and ./scripts/test pass

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

- Add scripts/spec_drift.py. It anchors SDK request/response types to
  endpoints discovered from the resource modules, normalizes both the
  OpenAPI spec and the SDK types into one shape model, and compares
  properties, types, enum values, and required-ness structurally.
- Replace the sdksync-based spec-drift workflow with one that runs the
  script on pull requests and writes the report to the job summary.
- Document the script in CONTRIBUTING.md.
The spec has one URL and no version history, so the pin records the
info.version the SDK was last synced against. spec_drift.py reports a
differing live version as 'version' drift and --update-pin writes the
new version once no structural drift remains. CI ignores the version
kind (it bumps on every backend deploy) and prints the gap as a notice.
Replaces the version pin. spec/openapi.json is a committed copy of the
public spec; the PR check runs against it, so it is reproducible and an
upstream change cannot fail an unrelated PR. A scheduled spec-snapshot
workflow refreshes the file and opens a PR when the live spec changed.
@hewliyang
hewliyang force-pushed the hly/chore/spec-drift branch from dc67424 to 0bcd768 Compare September 8, 2026 20:34
@hewliyang
hewliyang requested a review from kirthi20 September 8, 2026 21:38
@hewliyang
hewliyang merged commit e2222c0 into main Sep 8, 2026
7 of 8 checks passed
@hewliyang hewliyang mentioned this pull request Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants