Skip to content

fix(generate client): generate every language before writing the library (#1163) - #1169

Merged
antoinetoussaint-byte merged 1 commit into
mainfrom
fix/1163-python-client-buf-validate
Oct 26, 2026
Merged

antoinetoussaint-byte merged 1 commit into
mainfrom
fix/1163-python-client-buf-validate

Conversation

@antoinetoussaint-byte

Copy link
Copy Markdown
Contributor

Fixes #1163.

What was wrong

Two defects, each with its own owner.

1. A failed generation deleted the library's generated tree. GenerateClientLibrary rewrote contract/ first. Then, per language, it wiped the generated content (gen/, the TypeScript src/, the Python _gen/ and facades) and only afterwards asked the companion to generate. When the companion failed, the library was left without bindings. Builds before #1128 also removed every *.py at the Python package root except __init__.py. That is how authority_client/work_context.py was lost in the issue: the CLI that reported it predates #1128.

2. buf/validate did not resolve for a Python client from sources. The proto companion's Python buf.yaml template listed googleapis alone, while Go, TypeScript and Rust also list protovalidate. This is already fixed at its owner: codefly-dev/core 15188366 ("a Python client from sources resolves the shared protos every other language does"). CLI main carries it since #1128 moved core to 88f4434c. The CLI in the issue was an earlier build, the one that picked companion 0.0.16. No second mechanism is added here. This PR only adds a test that holds the CLI's Python sources path to the fix.

The fix

pkg/generators now separates generating from writing:

  • stageLanguage runs the companion for every entry of every requested language into scratch directories. Nothing under --output is touched.
  • Only once every language has succeeded does the command write contract/. It then calls stagedLanguage.apply per language, which runs the existing wipe* / merge* / finalize* functions unchanged. The rules for what survives a successful run (hand-written modules at the Python package root, scaffold files written once) are the same as before. Staging only decides when they run.
  • OpenAPI clients stage the same way. The Go generator derives import paths from go.mod, so the staged copy generates against a copy of the library's own go.mod, or the scaffold's when the library has none.

Reused: the existing ownership rules (wipePythonLibrary's notice-based facade detection, writeFileIfMissing, ensurePythonDependencies), the per-entry scratch directory the protobuf path already had, and the staging model generate proto uses. Not built: no copy of the whole library, no directory swap, no new ownership list.

Also fixed, pre-existing, found while running the gated suite:

  • assertPythonImports ran the image as proto:<tag> without its registry, so every Python import check failed with docker.io/library/proto not found.
  • TestGenerateClientPythonMapsUpstreamProtoModules still expected from buf.validate import ..., from google.api import ... and a "protovalidate" dependency. The Python plugin of companion 0.0.17 (core 844a0e09) drops imports used only for an option. The CLI's own comment on googleapisPyPI already says so. The test now asserts that.
  • make lint is clean.

Tests and evidence

Test Gate Result
TestAFailedGenerationLeavesTheLibraryUntouched (pkg/generators) none pass; fails on main (contract rewritten, _gen/ and facade wiped)
TestRegeneratingAPythonLibraryKeepsWhatItDidNotWrite none pass, now driven through stagedLanguage.apply
TestGenerateClientFailureLeavesLibraryUntouched CODEFLY_GENERATE_QUALIFY=1 pass; fails on main. A first Python run, then hand-written work_context.py and tests/ added, then a run whose sources import a missing file: the library hashes byte for byte the same
TestGenerateClientPythonFromSourcesResolvesProtovalidate CODEFLY_GENERATE_QUALIFY=1 pass: the binding generates, has no _gen/buf, and imports in the companion. Against a core copy with the pre-15188366 Python template it fails with the issue's exact error: imported file does not exist … runBuf: cannot update buf: process exited with code 100
TestGenerateClientPythonMapsUpstreamProtoModules CODEFLY_GENERATE_QUALIFY=1 pass (fails on main; stale assertions)
Other gated TestGenerateClient* CODEFLY_GENERATE_QUALIFY=1 pass when each runs in its own process

The gated tests ran locally on macOS arm64 with Docker (OrbStack) and companion ghcr.io/codefly-dev/proto:0.0.17.

End to end against codefly-dev/authority feat/installation-scoped-role-grants with this build: codefly generate client --from authority/local/grpc --language python --no-facade --name authority-client --force succeeds. It regenerates _gen/ and leaves authority_client/work_context.py and tests/ in place.

Classification

Fix, at the place that owns each behaviour. When the library is written is decided in pkg/generators. The Python buf.yaml dependencies belong to core's proto companion and were fixed there (15188366).

Not verified

  • Gated suite in one process: when the gated suite runs in one process, every go-grpc fixture after the first fails while scaffolding (codefly add service, before generate client). The error is container recovery marker does not belong to this process or its launching parent. Main fails identically (checked on TestGenerateClientFromContractsDir and TestGenerateClientIsIdempotent). Each passes alone on this branch. That fixture problem is not addressed here. No CI job provisions the companion (pkg/conformance/matrix.json row linux-amd64-docker-generate), so CI does not run the gated tests.
  • Go build failure: a Go library that generates but fails go build is still left written. The compile check needs the library's own module and hand-written packages, so it runs after apply. User files are never removed by it; the docs say so.
  • End-to-end failure on authority: an injected failure there stopped while building the descriptor set for --from, before generate client reaches the library. The in-companion failure path is covered by the gated test, not by an authority run.
  • Linux: not run on Linux or under CI's Docker.

🤖 Generated with Claude Code

…ary (#1163)

A failed run deleted the library's generated tree before the companion ran:
GenerateClientLibrary rewrote contract/ first, then each language wiped its
gen/ (_gen/ and the facades for Python) and only afterwards asked the companion
to generate. When the companion failed, the library was left without bindings.
Builds older than #1128 also wiped every module at the Python package root,
including hand-written ones.

Every language now runs the companion into scratch directories first
(stageLanguage). contract/ and each language directory are written only once
all of them have succeeded (stagedLanguage.apply), with the existing ownership
rules unchanged: the wipe still spares hand-written files and scaffolds. This
is how `generate proto` already stages. OpenAPI clients stage the same way,
generating against a copy of the library's go.mod so the import paths match.

The buf/validate half of #1163 is already fixed at its owner: core 15188366
gives Python's buf.yaml the dependencies every other language has, and main
carries it since #1128 moved core to 88f4434c. The CLI that reported the issue
predated that pin. A new companion-gated test holds the Python sources path to
it: a proto importing buf/validate generates and its bindings import.

Also fixes two pre-existing test problems: assertPythonImports ran
"proto:<tag>" without the registry, and
TestGenerateClientPythonMapsUpstreamProtoModules still expected option-only
imports that companion 0.0.17's Python plugin drops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@antoinetoussaint-byte
antoinetoussaint-byte added this pull request to the merge queue Oct 26, 2026
Merged via the queue into main with commit 6e959cd Oct 26, 2026
14 checks passed
antoinetoussaint-byte added a commit that referenced this pull request Oct 26, 2026
Resolves pkg/generators/client.go against #1169's staged generation: the
version stamp runs after each staged language is applied, and a library
with no version is refused before anything is staged, so the library is
untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

generate client --language python: buf/validate unresolved, and a failed run deletes the library's python tree

1 participant