Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion companions/imagepolicy_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@ var unpinnedUpstreamBases = []string{
"docker:29-dind",
"golang:1.27-alpine",
"node:26.3.0-alpine3.22",
"rust:1.99-alpine",
}

// corePolicy is what core requires of its own images. It is an ordinary
Expand Down
7 changes: 0 additions & 7 deletions companions/proto/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,6 @@ RUN python3 -m venv /venv && /venv/bin/pip install --no-cache-dir grpcio-tools

# --- Runtime stage: minimal alpine + the codefly base ---------------

FROM rust:1.99-alpine AS rust-builder
RUN apk add --no-cache musl-dev && \
cargo install --locked protoc-gen-prost --version 0.5.0 && \
cargo install --locked protoc-gen-tonic --version 0.5.0

# Keep the interpreter compatible with the pinned generator's wheels and
# gencode contract. Alpine's python3 moves independently; grpcio-tools 1.71
# has Python 3.13 wheels, not 3.14 wheels. As in the Python companion, copy
Expand Down Expand Up @@ -124,8 +119,6 @@ COPY --from=builder /go/bin/swagger /usr/local/bin/
# the plugins — a formatter that moves with @latest moves the committed shape.
COPY --from=builder /go/bin/goimports /usr/local/bin/
COPY --from=builder /usr/bin/grpc_* /usr/local/bin/
COPY --from=rust-builder /usr/local/cargo/bin/protoc-gen-prost /usr/local/bin/
COPY --from=rust-builder /usr/local/cargo/bin/protoc-gen-tonic /usr/local/bin/

# Facade plugin sources (Python scripts + the TS npm package).
COPY companions/proto/facades/python /opt/codefly-facades/python
Expand Down
47 changes: 14 additions & 33 deletions companions/proto/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,13 @@

The codefly proto companion bundles `buf`, `protoc`, `protoc-gen-*`,
`swagger`, `grpcio-tools`, and the JS/TS generators into one OCI
image. Used by `codefly generate proto` and the
`codefly generate openAPI` pipeline.
image. All proto generation runs through Codefly (`codefly generate proto`
and `codefly generate client`), which owns the pinned companion invocation.
Supported client languages are Go, Python, and TypeScript. The image and its
build contain no Rust toolchain or Rust generation plugins; unsupported
language requests fail before writing output.
The generation runner keeps the companion alive while executing its tools,
including Swagger client generation; it does not rely on the base image's default command.

## Build

Expand Down Expand Up @@ -92,32 +97,16 @@ may also contain handwritten Go, so only files carrying Go's standard

## Versioning

`info.codefly.yaml` carries the image version (`0.0.10` at time of
writing). Bump via `scripts/tag.sh` — same convention as every other
companion. Both the Dockerfile build and the Nix build read this
value so the tag stays consistent.
## Rust client imports
`info.codefly.yaml` carries the image version. Version 0.0.21 removes Rust
generation and its build dependencies. Publish through `codefly companion
publish` using the [publication procedure](../../docs/runbooks/publish-companions.md);
changing the manifest alone does not publish an image. Core 0.19.13 selects
this companion.

The proto companion supplies local `protoc-gen-prost` and `protoc-gen-tonic`
0.5.0, targeting prost/tonic 0.14. Rust clients retain googleapis and
protovalidate message bindings beside the application's bindings. Standard
`google.protobuf` types continue to resolve through `prost-types`.

Client generation validates descriptor-set type references before invoking the
plugins. Both source and descriptor-set generation are covered by a Cargo
compile and encode/decode regression containing `google.rpc.Status` and
`buf.validate.Violations`:

```sh
go test ./companions/proto -tags=proto_companion_required -run TestRustClientRetainsImportedMessageFields -timeout=180s
```

This requires the built proto companion and Cargo. The updated plugins ship in
companion version 0.0.16; bumping the manifest alone does not publish that image.
Use the [companion publication procedure](../../docs/runbooks/publish-companions.md).
## Python generation

Python bindings come from the companion's own plugin, `protoc-gen-codefly-python`
(`facades/python/protoc_gen_codefly_python.py`), as Go's and Rust's come from
(`facades/python/protoc_gen_codefly_python.py`), as Go's come from
local plugins: it compiles the `*_pb2` messages and `*_pb2_grpc` stubs with the
`grpc_tools` pinned in the image's venv (`grpcio-tools==1.71.0`,
`protobuf==5.29.3`), so the gencode is versioned with the image. The Docker
Expand All @@ -139,11 +128,3 @@ A change to any of a companion's image inputs (its Dockerfile and every
build-context `COPY` source, `BuildSpec.Inputs`) must bump its
`info.codefly.yaml`: `internal/companionbump`, the required `Companion version`
check on every pull request, refuses one that does not.

Rust generation stages output before publishing it. `.codefly-rust-output.json`
records the generated files and their hashes; only unchanged, tracked files may
be replaced or removed. Unrelated files are preserved. A collision with an
untracked file or an edited binding is an error, including when adopting output
created before ownership tracking. Generate into an empty output directory in
that case; do not discard local edits. Failed generation leaves existing output
untouched, and publication errors roll back changed files.
16 changes: 7 additions & 9 deletions companions/proto/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,12 @@ type clientSpec struct {
func generateClient(ctx context.Context, spec clientSpec) error {
w := wool.Get(ctx).In("generateClient", wool.DirField(spec.destination))

switch spec.language {
case languages.GO, languages.PYTHON, languages.TYPESCRIPT:
default:
return w.NewError("unsupported proto generation language %q; supported languages: go, python, typescript", spec.language)
}

image, err := CompanionImage(ctx)
if err != nil {
return w.Wrapf(err, "cannot get companion image")
Expand Down Expand Up @@ -152,9 +158,7 @@ func generateClient(ctx context.Context, spec clientSpec) error {
// Sources only: a descriptor set carries its foreign files already marked as
// buf image imports, and re-targeting them would put a second Timestamp next
// to the consumer's runtime one.
// Rust also needs imported bindings; prost keeps google.protobuf external
// through its default prost-types mapping.
includeImports := len(spec.sources) > 0 && (spec.language == languages.TYPESCRIPT || spec.language == languages.RUST)
includeImports := len(spec.sources) > 0 && spec.language == languages.TYPESCRIPT

if err = CreateBufConfiguration(ctx, tmpDir, spec.service, spec.language, spec.facade,
WithGoPackageOverrides(goPackageOverrides), WithIncludeImports(includeImports)); err != nil {
Expand Down Expand Up @@ -210,12 +214,6 @@ func generateClient(ctx context.Context, spec clientSpec) error {
if _, err = shared.CheckDirectoryOrCreate(ctx, spec.destination); err != nil {
return w.Wrapf(err, "cannot create destination")
}
if spec.language == languages.RUST {
return generateRustOutput(ctx, spec.destination, func(output string) error {
name := fmt.Sprintf("proto-%s-%d-%s", spec.service, time.Now().UnixMilli(), spec.language)
return runBuf(ctx, name, image, tmpDir, output, false, depUpdate, generateArgs, before)
})
}

if includeImports {
// The run writes a tree for every namespace the sources import, which is
Expand Down
156 changes: 16 additions & 140 deletions companions/proto/client_internal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,154 +2,14 @@ package proto

import (
"context"
"fmt"
"os"
"path/filepath"
"testing"
"time"

"github.com/codefly-dev/core/languages"
"github.com/gofrs/flock"
"github.com/stretchr/testify/require"
)

func writeRustOutput(t *testing.T, dir, name, content string) {
t.Helper()
file := filepath.Join(dir, name)
require.NoError(t, os.MkdirAll(filepath.Dir(file), 0o750))
require.NoError(t, os.WriteFile(file, []byte(content), 0o600))
}

func TestCanceledRustGenerationPreservesDestination(t *testing.T) {
dest := t.TempDir()
for _, name := range []string{"api/api.rs", "src/user.rs", "google/rpc/google.rpc.rs"} {
writeRustOutput(t, dest, name, "existing content")
}
ctx, cancel := context.WithCancel(t.Context())
cancel()
err := GenerateClient(ctx, ClientRequest{Language: languages.RUST, Destination: dest,
Sources: []Source{{Path: "api.proto", Content: []byte(`syntax = "proto3"; package api; message Response { string value = 1; }`)}}})
require.Error(t, err)
for _, name := range []string{"api/api.rs", "src/user.rs", "google/rpc/google.rpc.rs"} {
data, err := os.ReadFile(filepath.Join(dest, name))
require.NoError(t, err)
require.Equal(t, "existing content", string(data))
}
}

func TestRustPublicationOwnsFilesNotDirectories(t *testing.T) {
dest := t.TempDir()
writeRustOutput(t, dest, "src/user.rs", "user content")
first := t.TempDir()
writeRustOutput(t, first, "api/api.rs", "old generated")
writeRustOutput(t, first, "google/rpc/google.rpc.rs", "old import")
require.NoError(t, publishRustOutput(dest, first))
writeRustOutput(t, dest, "google/rpc/user.rs", "user content")
second := t.TempDir()
writeRustOutput(t, second, "api/api.rs", "new generated")
require.NoError(t, publishRustOutput(dest, second))
require.NoFileExists(t, filepath.Join(dest, "google/rpc/google.rpc.rs"))
for _, name := range []string{"src/user.rs", "google/rpc/user.rs"} {
data, err := os.ReadFile(filepath.Join(dest, name))
require.NoError(t, err)
require.Equal(t, "user content", string(data))
}
data, err := os.ReadFile(filepath.Join(dest, "api/api.rs"))
require.NoError(t, err)
require.Equal(t, "new generated", string(data))
}

func TestRustPublicationRejectsEditsAndUnownedCollisions(t *testing.T) {
for _, owned := range []bool{false, true} {
t.Run(fmt.Sprint(owned), func(t *testing.T) {
dest := t.TempDir()
if owned {
first := t.TempDir()
writeRustOutput(t, first, "api/api.rs", "generated")
require.NoError(t, publishRustOutput(dest, first))
}
writeRustOutput(t, dest, "api/api.rs", "user edit")
stage := t.TempDir()
writeRustOutput(t, stage, "api/api.rs", "replacement")
writeRustOutput(t, stage, "aaa/new.rs", "new output")
require.ErrorContains(t, publishRustOutput(dest, stage), "unowned or edited")
data, err := os.ReadFile(filepath.Join(dest, "api/api.rs"))
require.NoError(t, err)
require.Equal(t, "user edit", string(data))
require.NoFileExists(t, filepath.Join(dest, "aaa/new.rs"))
})
}
}

func TestRustPublicationDoesNotPruneEditedImports(t *testing.T) {
dest, first := t.TempDir(), t.TempDir()
writeRustOutput(t, first, "google/rpc/google.rpc.rs", "generated import")
require.NoError(t, publishRustOutput(dest, first))
writeRustOutput(t, dest, "google/rpc/google.rpc.rs", "edited import")
require.ErrorContains(t, publishRustOutput(dest, t.TempDir()), "unowned or edited")
data, err := os.ReadFile(filepath.Join(dest, "google/rpc/google.rpc.rs"))
require.NoError(t, err)
require.Equal(t, "edited import", string(data))
}

func TestRustPublicationRollsBackPartialFailure(t *testing.T) {
dest := t.TempDir()
first := t.TempDir()
writeRustOutput(t, first, "aaa/old.rs", "old output")
require.NoError(t, publishRustOutput(dest, first))
manifest, err := os.ReadFile(filepath.Join(dest, rustOutputManifest))
require.NoError(t, err)
require.NoError(t, os.Symlink("missing-target", filepath.Join(dest, "zzz")))
stage := t.TempDir()
writeRustOutput(t, stage, "aaa/old.rs", "replacement")
writeRustOutput(t, stage, "zzz/new.rs", "cannot publish")
require.Error(t, publishRustOutput(dest, stage))
data, err := os.ReadFile(filepath.Join(dest, "aaa/old.rs"))
require.NoError(t, err)
require.Equal(t, "old output", string(data))
data, err = os.ReadFile(filepath.Join(dest, rustOutputManifest))
require.NoError(t, err)
require.Equal(t, manifest, data)
require.NoFileExists(t, filepath.Join(dest, "zzz/new.rs"))
}

func TestRustPublicationRejectsEscapingSymlink(t *testing.T) {
dest, stage, outside := t.TempDir(), t.TempDir(), t.TempDir()
require.NoError(t, os.Symlink(outside, filepath.Join(dest, "api")))
writeRustOutput(t, stage, "api/api.rs", "generated")
require.Error(t, publishRustOutput(dest, stage))
require.NoFileExists(t, filepath.Join(outside, "api.rs"))
}

func TestRustGenerationHonorsDestinationLockThroughSymlink(t *testing.T) {
dest, err := filepath.EvalSymlinks(t.TempDir())
require.NoError(t, err)
writeRustOutput(t, dest, "api/api.rs", "existing output")
alias := filepath.Join(t.TempDir(), "alias")
require.NoError(t, os.Symlink(dest, alias))
lock := flock.New(filepath.Join(dest, rustOutputLock))
require.NoError(t, lock.Lock())
t.Cleanup(func() { require.NoError(t, lock.Close()) })
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel()
err = GenerateClient(ctx, ClientRequest{Language: languages.RUST, Destination: alias,
Sources: []Source{{Path: "api.proto", Content: []byte(`syntax = "proto3"; package api; message Response {}`)}}})
require.ErrorIs(t, err, context.DeadlineExceeded)
data, err := os.ReadFile(filepath.Join(dest, "api/api.rs"))
require.NoError(t, err)
require.Equal(t, "existing output", string(data))
}

func TestRustPublicationRejectsInvalidOwnership(t *testing.T) {
for _, body := range []string{`{`, `{"../outside":"hash"}`, `{"/outside":"hash"}`, `{".codefly-rust-output.json":"hash"}`, `{".codefly-rust-output.lock":"hash"}`} {
dest, stage := t.TempDir(), t.TempDir()
writeRustOutput(t, dest, rustOutputManifest, body)
writeRustOutput(t, stage, "api/api.rs", "generated")
require.Error(t, publishRustOutput(dest, stage))
require.NoFileExists(t, filepath.Join(dest, "api/api.rs"))
}
}

// TestRemoveUnownedOutput pins what a TypeScript run is allowed to keep in its
// destination. Asking buf for the imports means the run writes a tree per
// imported namespace, which is not a set known ahead of time — so the
Expand Down Expand Up @@ -213,3 +73,19 @@ func TestRemoveUnownedOutputKeepsRootLevelOutput(t *testing.T) {
_, err := os.Stat(filepath.Join(dest, "google"))
require.True(t, os.IsNotExist(err))
}

func TestUnsupportedClientLanguagePreservesDestination(t *testing.T) {
dest := t.TempDir()
file := filepath.Join(dest, "existing.txt")
require.NoError(t, os.WriteFile(file, []byte("keep"), 0600))
err := GenerateClient(t.Context(), ClientRequest{Language: languages.RUST, Destination: dest,
Sources: []Source{{Path: "api.proto", Content: []byte(`syntax = "proto3"; package api; message Request {}`)}}})
require.ErrorContains(t, err, "unsupported proto generation language")
entries, err := os.ReadDir(dest)
require.NoError(t, err)
require.Len(t, entries, 1)
data, err := os.ReadFile(file)
require.NoError(t, err)
require.Equal(t, "keep", string(data))
require.ErrorContains(t, CreateBufConfiguration(t.Context(), dest, "api", languages.RUST, FacadeOptions{}), "unsupported proto generation language")
}
37 changes: 19 additions & 18 deletions companions/proto/companion_plugins_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,19 +21,6 @@ import (
// content hashes — fail the Nix build rather than any test here.
const protocGenEsRuntimeVersion = "2.11.0"

func TestCompanionInstallsLocalRustPlugins(t *testing.T) {
for _, file := range []string{"Dockerfile", "flake.nix", "templates/rust/buf.gen.yaml.tmpl"} {
assertMentionsAll(t, file, []string{"protoc-gen-prost", "protoc-gen-tonic"})
}
content, err := os.ReadFile("templates/rust/buf.gen.yaml.tmpl")
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(content), "buf.build/") {
t.Fatal("Rust generation must use the companion's local plugins")
}
}

// bumpRunbook records why the pin sits where it does and what moving it costs.
// The guards below keep it honest, because a stale runbook is worse than none:
// it still reads as current.
Expand Down Expand Up @@ -326,10 +313,10 @@ func TestTsFacadeLockfileMatchesTheRuntime(t *testing.T) {
// neighbour — the reader then checks the wrong line, finds something plausible,
// and moves on.
var runbookCitations = map[string]string{
"Dockerfile:104": "@bufbuild/protoc-gen-es@",
"flake.nix:51": "protocGenEsVersion =",
"flake.nix:59": "hash =",
"flake.nix:61": "npmDepsHash =",
"Dockerfile:99": "@bufbuild/protoc-gen-es@",
"flake.nix:51": "protocGenEsVersion =",
"flake.nix:59": "hash =",
"flake.nix:61": "npmDepsHash =",
}

func TestRunbookCitationsPointAtWhatTheyName(t *testing.T) {
Expand Down Expand Up @@ -402,7 +389,7 @@ func TestTypeScriptTemplateUsesTheBakedProtocGenEs(t *testing.T) {
}
}

// Python's bindings come from the companion's own plugin, as Go's and Rust's do:
// Python's bindings come from the companion's own plugin, as Go's do:
// buf's remote python plugins moved on their own, and their gencode outran the
// protobuf runtime a generated library declares. Both build definitions ship
// the plugin, and the image pins the grpc_tools it runs.
Expand All @@ -419,3 +406,17 @@ func TestCompanionGeneratesPythonWithItsLocalPlugin(t *testing.T) {
}
assertMentionsAll(t, "Dockerfile", []string{"protobuf==5.29.3", "grpcio-tools==1.71.0"})
}

func TestCompanionBuildsWithoutRust(t *testing.T) {
for _, file := range []string{"Dockerfile", "flake.nix"} {
data, err := os.ReadFile(file)
if err != nil {
t.Fatal(err)
}
for _, tool := range []string{"rust:", "rust-builder", "cargo", "protoc-gen-prost", "protoc-gen-tonic"} {
if strings.Contains(string(data), tool) {
t.Errorf("%s includes removed Rust tooling %q", file, tool)
}
}
}
}
Loading
Loading