Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@ jobs:
- uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2
- run: cargo test --workspace

# The generated code in sdk/src/generated is checked in. Verify it matches
# The generated code in sdk/src/generated and the reference the tests
# compare it with, in sdk/tests/generated, are checked in. Verify they match
# what the codegen tool produces from the vendored proto. Only the .rs
# outputs are compared; the encoded descriptor set may differ byte-wise
# between protoc versions.
Expand All @@ -50,4 +51,4 @@ jobs:
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
- run: cargo run -p codegen
- run: git diff --exit-code -- 'sdk/src/generated/*.rs'
- run: git diff --exit-code -- 'sdk/src/generated/*.rs' 'sdk/tests/generated/*.rs'
13 changes: 7 additions & 6 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,16 +18,17 @@ Run all of these before you call a change done. CI requires them.
cargo fmt --all --check
cargo clippy --workspace --tests -- -D warnings
cargo test --workspace
cargo run -p codegen # needs protoc; must leave sdk/src/generated unchanged
cargo run -p codegen # needs protoc; must leave the generated files unchanged
```

## Rules

- Never edit `sdk/src/generated` by hand. Change the proto or `codegen/` and
regenerate.
- `State` and `Resource` are written by hand in `sdk/src/proto/state.rs`.
When the proto changes either message, update that file and
`sdk/tests/unknown_fields.rs` as CONTRIBUTING.md describes.
- Never edit `sdk/src/generated` or `sdk/tests/generated` by hand. Change the
proto or `codegen/` and regenerate.
- Every message keeps unknown fields: `codegen/src/messages.rs` writes the
`prost::Message` impls in place of prost's derive. When codegen refuses a
field kind, or a test in `sdk/tests/generated_messages.rs` fails, follow
CONTRIBUTING.md.
- A new public field on a message struct is a breaking change. Say so in the
commit and the pull request.
- Don't change the crate version outside a release pull request.
Expand Down
43 changes: 26 additions & 17 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,18 +33,22 @@ cargo run -p codegen

CI runs four checks on every pull request, and all four must pass before it
can merge: `fmt`, `clippy`, `test` and `codegen-drift`. The last one runs
codegen and fails when the `.rs` files in `sdk/src/generated` differ from
what's checked in.
codegen and fails when the `.rs` files in `sdk/src/generated` or
`sdk/tests/generated` differ from what's checked in.

### Generated and hand-written protocol code
### Generated protocol code

Don't edit `sdk/src/generated` by hand. Change the proto or `codegen/`, then
run codegen and commit the result.
Don't edit `sdk/src/generated` or `sdk/tests/generated` by hand. Change the
proto or `codegen/`, then run codegen and commit the result.

Two messages aren't generated. `State` and `Resource` are written by hand in
`sdk/src/proto/state.rs` so that they keep fields this SDK doesn't know,
which prost-generated messages discard. Codegen skips both and adds
`unknown_fields` to their generated serde constructors.
prost discards a field it doesn't know, so codegen replaces prost's `Message`
derive. `codegen/src/messages.rs` gives every message an `unknown_fields`
field and writes the impls the derive would have written, with the unknown
fields kept and encoded again. Codegen also writes the messages as prost
derives them to `sdk/tests/generated/prost_v1.rs`.
`sdk/tests/generated_messages.rs` compares the two for every message: the
bytes, the errors on malformed input, `Debug`, the defaults and the
accessors.

### Updating the vendored proto

Expand All @@ -58,14 +62,15 @@ Finish it on its branch:
1. Replace the file with the one from the release tag, and put the
`Vendored from crossplane/crossplane` comment back above `syntax` with the
new version.
2. Run `cargo run -p codegen` and commit the changes in `sdk/src/generated`.
New fields on generated messages need nothing else.
3. If the update changes `State` or `Resource`, the build fails in the
generated serde code and a test in `sdk/tests/unknown_fields.rs` fails.
Update `sdk/src/proto/state.rs` to match: the struct field, `encode_raw`,
`merge_field`, `encoded_len`, and `Debug` for `Resource`. Then update the
prost-derived copies and the expected field list in that test file.
4. Run the checks above.
2. Run `cargo run -p codegen` and commit the changes in `sdk/src/generated`
and `sdk/tests/generated`. A new field needs nothing else.
3. If codegen stops at a field kind it doesn't write yet, its error gives
the field and the `#[prost(...)]` attribute. Add the kind to
`codegen/src/messages.rs`, following what prost's derive generates for it.
4. If the update adds a message, `the_checks_cover_every_message_of_the_proto`
fails. Add the message to `for_every_message` and to `sample` in
`sdk/tests/generated_messages.rs`, and set new fields in the samples.
5. Run the checks above.

A new field on any message is a breaking change for this crate, because the
message structs are public and can be built with a struct literal. Say so in
Expand All @@ -81,6 +86,10 @@ change the generated code, and its pull request then fails `codegen-drift`.
Check out the branch, run `cargo run -p codegen`, and push the result to the
same branch.

A new version of `prost` can change what its `Message` derive does. Its pull
request then fails in `sdk/tests/generated_messages.rs`, and
`codegen/src/messages.rs` has to follow the derive before it merges.

## Submitting changes

* Open an issue first for anything beyond a small fix, so we can agree on the
Expand Down
4 changes: 4 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 11 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,10 @@ for working with requests and responses.
encoded file descriptor set used for gRPC server reflection.
- `src/{server,request,response,resource,logging}.rs` - the hand-written
SDK: runtime and helpers.
- `codegen/` - maintainer tool that regenerates `sdk/src/generated`.
- `tests/generated/` - the messages as prost derives them, which the tests
compare the generated ones with.
- `codegen/` - maintainer tool that regenerates `sdk/src/generated` and
`sdk/tests/generated`.
- `example/` - an example function, the starting point for new functions.

## Writing a function
Expand Down Expand Up @@ -105,12 +108,13 @@ with a few deliberate differences:
`Server::ready`: until it completes, the server reports NOT_SERVING and
answers RunFunction with UNAVAILABLE, and if it fails, the server stops
with `Error::NotReady`.
- **Unknown fields are kept on `State` and `Resource` only.** protobuf-go
keeps fields it does not know on every message. This SDK keeps them on the
two messages `response::to` copies from the request, so a field a newer
Crossplane adds to them passes through a function unchanged.
As in Go, replacing a desired resource instead of modifying it in place
drops that resource's unknown fields.
- **Every message keeps unknown fields.** Like protobuf-go, this SDK keeps
the fields it doesn't know, in `unknown_fields`, and writes them back when
it encodes the message. A field a newer Crossplane adds to a request passes
through a function unchanged, and so does a field on a response the
function forwards. A function can't read or change such a field, and it has
no JSON form. As in Go, replacing a desired resource instead of modifying
it in place drops that resource's unknown fields.
- **Graceful shutdown is built in.** The server drains in-flight requests on
SIGTERM and SIGINT.
- **Metrics are Go-compatible, served as OpenMetrics first.** Like
Expand Down
6 changes: 6 additions & 0 deletions codegen/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,9 @@ prost-build = "0.14"
tonic-prost-build = "0.14"
pbjson-build = "0.9"
tempfile = "3"
# The rewrite of the generated messages (src/messages.rs): prost-build's own
# parser and formatter.
syn = { version = "3", features = ["full"] }
quote = "1"
proc-macro2 = "1"
prettyplease = "0.3"
130 changes: 99 additions & 31 deletions codegen/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,23 @@
//!
//! Maintainer tool: requires `protoc` on PATH. SDK users never run this
//! because the generated code is committed.
//!
//! prost-build, tonic and pbjson generate the code. `messages` then rewrites
//! the messages so that they keep unknown fields, and what prost wrote for
//! them is kept beside the tests as the reference the rewrite is held to.

mod messages;

use std::error::Error;
use std::fs;
use std::path::PathBuf;
use std::path::{Path, PathBuf};

const PROTO: &str = "sdk/proto/v1/run_function.proto";
const INCLUDE: &str = "sdk/proto";
const OUT: &str = "sdk/src/generated";

/// Messages written by hand in `sdk/src/proto/state.rs` so that they keep
/// unknown fields, which prost-generated messages discard.
const HAND_WRITTEN: [&str; 2] = ["State", "Resource"];
/// The messages as prost-build derives them, which discard unknown fields:
/// what `sdk/tests/generated_messages.rs` compares the rewritten ones with.
const REFERENCE: &str = "sdk/tests/generated/prost_v1.rs";
const UNKNOWN_FIELDS_INIT: &str = "unknown_fields: Default::default(),";

fn main() -> Result<(), Box<dyn Error>> {
Expand All @@ -26,12 +31,6 @@ fn main() -> Result<(), Box<dyn Error>> {
.file_descriptor_set_path(&descriptor)
.compile_well_known_types()
.extern_path(".google.protobuf", "::pbjson_types");
for message in HAND_WRITTEN {
config.extern_path(
format!(".apiextensions.fn.proto.v1.{message}"),
format!("crate::proto::state::{message}"),
);
}

tonic_prost_build::configure()
.build_client(true)
Expand All @@ -44,40 +43,92 @@ fn main() -> Result<(), Box<dyn Error>> {
.out_dir(tmp.path())
.build(&[".apiextensions"])?;

let out = root.join(OUT);
fs::create_dir_all(&out)?;
// prost-build names its file after the proto package, a keyword in it
// escaped, and pbjson adds `.serde` to that name.
let (mut prost, mut serde) = (None, None);
for entry in fs::read_dir(tmp.path())? {
let path = entry?.path();
let name = path
.file_name()
.unwrap_or_default()
.to_string_lossy()
.into_owned();
let target = if name.ends_with(".serde.rs") {
"v1.serde.rs"
if name.ends_with(".serde.rs") {
serde = Some(fs::read_to_string(&path)?);
} else if name.ends_with(".rs") {
"v1.rs"
} else {
continue;
};
if target == "v1.serde.rs" {
let serde = init_unknown_fields(&fs::read_to_string(&path)?)?;
fs::write(out.join(target), serde)?;
} else {
fs::copy(&path, out.join(target))?;
prost = Some(fs::read_to_string(&path)?);
}
println!("wrote {}", out.join(target).display());
}
let prost = prost.ok_or("prost-build wrote no .rs file")?;
let serde = serde.ok_or("pbjson-build wrote no .serde.rs file")?;

let (header, types, services) = split(&prost);
let rewritten = messages::rewrite(types)?;

let out = root.join(OUT);
fs::create_dir_all(&out)?;
write(
&out.join("v1.rs"),
&format!("{header}{}{services}", rewritten.source),
)?;
write(
&out.join("v1.serde.rs"),
&init_unknown_fields(&serde, &rewritten.messages)?,
)?;
let reference = root.join(REFERENCE);
fs::create_dir_all(reference.parent().expect("the reference is in a directory"))?;
write(&reference, &format!("{header}{types}"))?;
fs::copy(&descriptor, out.join("fileset.bin"))?;
println!("wrote {}", out.join("fileset.bin").display());
Ok(())
}

/// Makes the generated serde constructors of the hand-written messages
/// initialize `unknown_fields`, a field the descriptor does not have.
fn init_unknown_fields(serde: &str) -> Result<String, Box<dyn Error>> {
fn write(path: &Path, content: &str) -> Result<(), Box<dyn Error>> {
fs::write(path, content)?;
println!("wrote {}", path.display());
Ok(())
}

/// Splits what prost-build wrote into its leading comment, the messages and
/// enumerations, and the gRPC client and server modules tonic appends. Only
/// the middle part is rewritten: a parser keeps no plain comment, and the
/// service code has no message in it.
fn split(prost: &str) -> (&str, &str, &str) {
let header: usize = prost
.split_inclusive('\n')
.take_while(|line| line.starts_with("//") && !line.starts_with("///"))
.map(str::len)
.sum();
let mut services = prost.len();
let mut offset = 0;
let mut docs = None;
for line in prost.split_inclusive('\n') {
if line.starts_with("///") {
docs.get_or_insert(offset);
} else {
let service = line.starts_with("pub mod ")
&& (line.trim_end().ends_with("_client {")
|| line.trim_end().ends_with("_server {"));
if service {
services = docs.unwrap_or(offset);
break;
}
docs = None;
}
offset += line.len();
}
(
&prost[..header],
&prost[header..services],
&prost[services..],
)
}

/// Makes the generated serde constructor of every message initialize
/// `unknown_fields`, a field the descriptor does not have.
fn init_unknown_fields(serde: &str, messages: &[String]) -> Result<String, Box<dyn Error>> {
let mut patched = serde.to_owned();
for message in HAND_WRITTEN {
for message in messages {
let constructor = format!("Ok({message} {{\n");
if patched.matches(&constructor).count() != 1 {
return Err(format!("expected one serde constructor for {message}").into());
Expand Down Expand Up @@ -115,6 +166,20 @@ mod tests {
})
";

fn both() -> Vec<String> {
vec!["State".to_string(), "Resource".to_string()]
}

#[test]
fn split_cuts_at_the_comment_and_at_the_services() {
let prost = "// This file is @generated by prost-build.\n/// A message.\npub struct M {}\n/// Generated client implementations.\npub mod m_client {\n}\n/// Generated server implementations.\npub mod m_server {\n}\n";
let (header, types, services) = split(prost);
assert_eq!(header, "// This file is @generated by prost-build.\n");
assert_eq!(types, "/// A message.\npub struct M {}\n");
assert!(services.starts_with("/// Generated client implementations.\npub mod m_client {"));
assert_eq!(split("pub struct M {}\n"), ("", "pub struct M {}\n", ""));
}

#[test]
fn init_unknown_fields_extends_both_constructors() {
let want = "\
Expand All @@ -127,11 +192,14 @@ mod tests {
unknown_fields: Default::default(),
})
";
assert_eq!(init_unknown_fields(GENERATED).expect("patches"), want);
assert_eq!(
init_unknown_fields(GENERATED, &both()).expect("patches"),
want
);
}

#[test]
fn init_unknown_fields_fails_without_a_constructor() {
assert!(init_unknown_fields("Ok(State {\n})").is_err());
assert!(init_unknown_fields("Ok(State {\n})", &both()).is_err());
}
}
Loading
Loading