Add a Java runtime built on generated models - #444
Open
Seth Juarez (sethjuarez) wants to merge 29 commits into
Open
Add a Java runtime built on generated models#444Seth Juarez (sethjuarez) wants to merge 29 commits into
Seth Juarez (sethjuarez) wants to merge 29 commits into
Conversation
Introduces `runtime/java/` and wires a Java emit target into the shared TypeSpec pipeline so the Java runtime is built on the same canonical Typra-emitted model layer as the C#, Go, Python, Rust and TypeScript runtimes. No hand-written duplicate model code is introduced. The @typra/emitter@0.4.2 Java backend is immature: its raw output does not compile and diverges from the other backends. `normalize-java-output.mjs` applies a fixed, deterministic set of structural rewrites from the sanctioned post-emit seam, covering seventeen catalogued emitter defects (J1-J4, J6-J17) - reserved-word field names, abstract-model instantiation, enum typing and visibility, base-property population in derived loaders, named-collection normalization, `collectionFormat` support, double-run save hooks, eagerly materialized optional defaults, and null-initialized required enums. Behaviour is matched against the generated C# and Rust runtimes rather than guessed. Object-format collection eligibility is resolved from the TypeSpec declared type - a named-collection alias, or an element model that declares `name` - which reproduces exactly the eleven object-format save sites the C# backend emits, and correctly keeps `UnionProperty.anyOf`/`oneOf` array-only. The shim is temporary and self-policing: each rewrite asserts it matched, and generation is idempotent, so an upstream emitter fix surfaces as a loud failure rather than silent drift. Verified: 334 generated files, clean build with zero warnings, 174 tests passing (152 generated example round-trips plus 22 normalization regression tests), and byte-identical output across repeated generation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Implements the front half of the Prompty pipeline in Java, built on the
Typra-emitted model package rather than a second hand-written model layer.
Rust is the behavioural reference throughout.
Loader/Frontmatter/References .prompty parsing, ${env:} and ${file:}
resolution, legacy migration
Nonces/Threads rich-input markers and thread expansion
Registry/PromptyExtension ServiceLoader-based invoker discovery
Renderer/Parser/Executor/ the four extension protocols
Processor
Pipeline validate, render, parse, prepare, process,
run, invoke
Tracer/Streams/StructuredResult/CancellationToken
Conformance is proven against the shared cross-runtime vectors: 25 load,
23 render and 15 parse cases all pass, alongside the generated-model
example suite. 264 tests, 0 failures.
Four deliberate divergences from Rust, each strictly more correct:
* parse() dispatches through the registered parser instead of
short-circuiting on "prompty", which in Rust skips nonce validation.
* Role markers resolve case-insensitively; Rust silently reads "User:"
as system.
* Thread history accepts both string and part-list content.
* Only thread markers are expanded during parsing. Image, file and audio
markers survive to wire conversion as spec section 5.2 requires; Rust
swallows them.
References inside lists are now resolved too, which the spec requires and
Rust misses. Environment adds an override layer because a JVM cannot
mutate its own environment the way the Rust harness does.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Prompty speaks to a model through three separable pieces: a wire layer that turns the loaded prompt into a request body, an executor that sends it, and a processor that turns whatever comes back into a result. Splitting them this way means the two interesting halves - request shaping and response interpretation - are pure functions that can be graded against the shared spec vectors without a network, and the executor is left holding only the parts that genuinely need one. The wire layer covers all four API shapes (chat, responses, embedding, image), tool declarations, and structured output. Model options are not remapped here: the generated model already knows how each option is named per provider, so provider code asks it rather than keeping a second copy of that table. The processor handles both single responses and streams. Streaming is where the shape of the API matters most - text is forwarded as it arrives, tool calls are accumulated across deltas and emitted only once whole, and usage comes last, so a caller that stops at the first tool call has still seen everything that matters. Two deliberate departures from the Rust reference, both tested: - Streamed function-call arguments are routed by call_id, then item_id, then output_index. Rust matches on call_id alone, but the live Responses API sends item_id and output_index and no call_id, so streamed arguments are dropped there and only recovered because the terminal event repeats them. - Every tool call gets a result message even when the caller supplies fewer results than calls. OpenAI rejects a conversation with an unanswered call, so an empty answer beats an absent one. Transport lives in the core module rather than here, because Anthropic and Foundry need the same failure classification: whether a request definitely never reached the provider decides whether retrying it is safe or risks acting twice. Streams are closeable through every wrapper, so cancellation and refusal release the connection instead of leaving it checked out of the pool. The spec-vector harness moved to test fixtures so provider modules share one comparison implementation. 22 wire vectors and 17 process vectors now run against Java, matching the Rust suites case for case; 335 tests pass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Anthropic's Messages API differs from OpenAI's in ways that rule out sharing the conversion, so this is a sibling module rather than an adapter: system messages leave the conversation and become a top-level string, content is always an array of typed blocks, tools are flat rather than nested under a function wrapper, max_tokens is required, and a tool round replays as exactly two messages instead of one per result. Assistant turns carry their raw content blocks in metadata. Thinking blocks are signed, and a signature only validates against the exact bytes Anthropic produced, so a replay rebuilt from message text would be rejected. Two deliberate divergences from the Rust reference, both documented in place: buildUrl honours ANTHROPIC_BASE_URL and collapses a duplicate /v1, matching the official SDKs and Rust's own OpenAI executor; and a mid-stream `error` event surfaces an ErrorChunk and releases the transport rather than being skipped, since Anthropic emits these for overload and ignoring one hands the caller a truncated answer that looks complete. Also strengthens the shared vector harness, which stands behind every suite in the runtime. It compared only the keys a vector mentioned, so a runtime could have added a field to every request without a test noticing, and it accepted an absent key wherever a vector stated an explicit null. Both now fail, matching the reference comparison but for its one deliberate relaxation: numbers are compared to single precision, because a 32-bit temperature cannot hold a value like 0.7 that a vector states in full. SpecVectorsTest grades the harness itself, and the vector rebuild logic the provider suites had begun to duplicate moves into a shared VectorAgents fixture. Endpoint normalization moves to Connections.trimTrailingSlashes, which strips every trailing slash rather than one; both providers left `host//v1` behind for an endpoint typed with two. 410 tests, 0 failures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The turn engine is the part of the runtime that survives a crash. It drives the model/tool loop, and after every effect it writes a checkpoint plus a journal entry so a host can resume mid-turn instead of replaying a turn that already spent money and ran side effects. Ported from the Rust engine, which is the behavioural reference. The five shared vectors in spec/vectors/engine/turn_vectors.json now run against Java, asserting the same statuses, snapshots, tool ordering, portability, and the exact ordered event stream that Rust asserts. Two things worth calling out: Resume arithmetic. Whether a resumed run continues the current iteration or starts the next one depends on whether anything is still outstanding. Rust clears the reconciliation flag on a cloned checkpoint before computing that, so the cleared flag participates in the decision; a direct reading of the public API would compute it from the stale flag and land an iteration early. Java mirrors the Rust ordering through a private overload. The flag only changes the answer for a checkpoint that holds no model response, so the two tests that guard it construct exactly that shape - an end-to-end assertion passes for an unrelated reason. Run identity. Rust takes the request by value, so an engine-assigned run id can never be seen by the caller. Java passes by reference, so the id is assigned onto the run state instead of written back; otherwise reusing one request for a second run would silently inherit the first run's identity. Every durable path is covered by a gapless-sequence invariant: a hole in the journal means an effect was recorded that a resumed run would never replay. Each new guard was proven non-vacuous by perturbing the behaviour it claims to protect and confirming it fails. 446 tests, 0 failures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Strict-mode parsing stamps a random hex nonce onto every role marker and rejects any marker that cannot reproduce it, so text injected through a template variable cannot pretend to be a new turn. Attribute values were being coerced to numbers wherever they parsed as one. Roughly one nonce in eighteen thousand comes out as digits only with a leading zero, and coercing that to a Long drops the zero, so the nonce no longer matched the one that had been stamped and a perfectly ordinary render was rejected as a prompt injection. It surfaced as an agent vector failing once every few runs and passing on a rerun. The nonce is an opaque token rather than data, so it is now captured as written. Other attributes still coerce, which is what the parse vectors expect. The Rust reference converts the coerced number back to text, which does not restore the lost zero, so it still carries this fault. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
FunctionTool.parameters is a flat list of properties, or a name-keyed
dict of them. The agent vectors instead wrapped every tool's parameters
in a {properties: [...]} object, which the schema does not define. It
reads plausibly by analogy with inputs and outputs, which genuinely are
schemas with a properties field, so the mistake went unnoticed.
It went unnoticed because Rust tolerates it. Its loader skips an
array-valued entry when reading a dict, so the malformed shape silently
became an empty parameter list and no assertion covered it. C# rejects
it, and so does the Java runtime being added here, which is how it came
to light. All thirty-four occurrences are unwrapped to the flat form,
matching the load and wire vectors, which were already correct.
Rust stays green on the corrected fixture and now actually loads the
parameters rather than discarding them. The generated Java loader gains
a note explaining why it rejects the shape rather than following Rust,
via the normalizer that authors it.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The turn engine added previously owns the loop: iterations, tool rounds, checkpoints and resume. It reaches the outside world only through ports. This supplies those ports and exposes the result as Pipeline.turn. Following Rust, the adapter holds no loop logic of its own. It binds the model, tool, permission, policy, durability and observation ports onto the existing renderer, parser, executor and processor, so there is one implementation of the agent loop rather than two that must be kept in step. Around that sit the pieces a turn needs but the engine does not define: tool dispatch with binding injection and four argument shapes, context trimming, a steering queue, guardrails over input, output and tool calls, and the event stream a caller observes. A stream that fails with an unknown outcome now keeps that distinction. It was matching the plain error branch, because the failure type extends the error chunk, and arriving as an ordinary execute error. The engine would then retry a request the provider may already have completed, repeating the model call and any tools it had triggered. Reverting the fix makes the new test report three attempts instead of one. All twenty-eight agent vectors pass. The context and steering tests assert against the conversation the model actually received, since the mock replays by index and would otherwise pass whatever the adapter did with the messages. Steering is checked for delivery and ordering only: the vectors name an iteration to inject before, but no runtime models that, and the reference drains its queue on every iteration including the first. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Each provider answers a model-list call in its own shape, and none of them answers it completely: OpenAI reports an id and an owner and nothing else, Anthropic omits the owner entirely, and Foundry splits the same facts across a data-plane and an ARM shape depending on which endpoint replied. A caller choosing a model needs the same facts from all three, so the wire mapping is per provider and a shared dataset fills what the endpoint left out. The fill is deliberately timid. Anything the provider stated wins, including an empty list, because a provider that says a model has no output modality knows something the dataset does not. The dataset only reaches fields the payload left null, and it matches model families by longest prefix at token boundaries, so gpt-4 claims gpt-4-0613 and gpt-4.1 but never gpt-45. The dataset itself is shared across runtimes and lives in spec/data, which a published artifact cannot reach, so each runtime vendors a copy. A test fails if the vendored copy drifts from the canonical one. Foundry's mappers are the whole of its contribution for now; listing needs OAuth and ARM, which arrive with the rest of that provider. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Ports two remaining pieces of the Rust runtime so the Java side can
replay a turn end-to-end and be graded against the shared vectors.
Memory is a static-helper class over the generated MemoryStore, since
Java cannot add inherent methods to generated types. Scoring follows
memory.rs exactly: +2.0 per distinct query token found in content,
+3.0 per token found in a tag, and +1.0 once for a CORE entry that
matched at all. Ranking is score-descending with insertion-order ties,
which List.sort gives us for free because it is contractually stable.
The harness package mirrors harness.rs: a collecting event sink, an
NDJSON journal writer, an in-memory checkpoint store, allow/deny
permission resolvers, a function-backed host tool executor, and
ReferenceTurnRunner, which adapts the canonical TurnEngine onto the
host-facing protocol types.
ReplayVectorsTest drives all five scenarios in
spec/vectors/harness/replay_vectors.json and reproduces the normalized
journal exactly. Ten behavioural rules were perturbation-proven and
reverted; two of those perturbations came back vacuous and were closed
with real changes rather than accepted:
- The tool-round arm of the messages_updated rule is structurally
unreachable, because CONVERSATION_UPDATED is emitted in the same
batch as TOOL_RESULT_COMMITTED in both runtimes. Rather than delete
a rule the shared contract states, it is extracted into a named
predicate, shouldRecordMessagesUpdated, and tested directly.
- The max_iterations turn_end and error payload rewrites were
unobserved, because the normalized vectors compare only :status.
Added a journal payload reader and assertions on both.
Journal records are written with an explicit LF rather than the
platform separator, matching Rust's writeln!, and that is now asserted
on the raw bytes since line-based normalization cannot see it. The
engine tool-request id fallback was also corrected to skip only null,
not empty, matching Option::or_else.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Foundry speaks the OpenAI wire format, so the executor and processor
subclass the OpenAI ones rather than duplicating the wire layer, which
mirrors how the Rust crate depends on prompty-openai.
Routing hooks the protected buildUrl and re-maps the OpenAI path onto the
Azure surface: a foundry-kind connection hits {endpoint}/{operation} with
no deployment or api-version, and every other kind hits the classic
{endpoint}/openai/deployments/{id}/{operation}?api-version={v}. Responses
is refused because Azure has no equivalent surface, matching Rust.
stripProjectPath cuts at /api/projects and rewrites *.services.ai.azure.com
to *.openai.azure.com, guarding against mistaking a non-numeric colon
suffix for a port.
Auth sends api-key for key connections and Authorization: Bearer for
foundry, falling back to AZURE_OPENAI_API_KEY and AZURE_INFERENCE_CREDENTIAL
respectively. The two are deliberately not cross-used. Java has no
DefaultAzureCredential equivalent, so its absence is reported rather than
silently degraded.
Java connections are typed, so an inline foundry credential that Rust would
read from raw JSON is dropped at load; the divergence is documented and
asserted.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The offline suite proves the runtime agrees with the shared spec vectors, but vectors are frozen fixtures: they cannot catch a request shape the services stopped accepting, or a response field a provider renamed. These tests close that gap by driving the real OpenAI, Anthropic and Foundry endpoints. They are excluded from the normal build by the `live` tag and run with `-PliveTests`. Credentials come from a gitignored `.env` at the Gradle root, installed only for keys the process environment does not already supply, so CI variables win over the file. Each test declares the variables it needs and skips -- honestly, via JUnit assumptions rather than a silent pass -- when they are absent, so a machine holding only some credentials still exercises the paths it can reach. Coverage: chat, option plumbing, streaming, embeddings, image generation, model listing, structured output, tool calls and the agent loop, plus the authentication failure path. Two things this uncovered: Streaming has two distinct surfaces and they are easy to conflate. `Pipeline.invoke` deliberately collapses a stream into accumulated text, so asserting on chunks through `invoke` only ever measures the collapse. Driving the executor and processor directly is what actually exercises the SSE parser, so both are covered separately. Anthropic sends a single `content_block_delta` for a short answer. That looked like a parser defect until the raw endpoint was queried directly and confirmed to behave the same way, so the test asks for a long enough response to span deltas rather than asserting something untrue about the service. Foundry method order is pinned. One test removes a credential to prove the absence is reported clearly rather than sent as an anonymous request, and it now restores it afterwards: without that, a later test skipped silently, looking exactly like "no credential configured" when the real cause was a previous test taking it away. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A Foundry connection needs an endpoint and a token, and a user who has neither has no way to get them from inside the runtime. This adds the two halves of that: the OAuth flows that produce a token, and the ARM queries that turn it into a list of endpoints to choose from. FoundryOAuth implements the device authorization grant and the authorization-code-with-PKCE flow against Entra ID, ported from the Rust runtime so both agree on scopes, error codes, and backoff. The poll loop gets an injectable token endpoint, sleeper, and clock, which is a departure from Rust: Rust hardwires its HTTP client and timer, and as a result none of its poll state machine is covered. Here the pending, slow_down, expiry, and timeout branches are all exercised directly, with no network and no elapsed wall time. FoundryArm enumerates subscriptions, AI resources, and projects. Projects have two representations depending on when they were created, so it probes both and merges. Each probe fails soft to an empty list, because a tenant that denies one resource provider is ordinary and should not take the whole listing down with it. Control-plane calls are bounded at thirty seconds, which Http.getJson now supports per call; model calls stay unbounded, since a slow answer there is still an answer. Http.postForm returns non-2xx responses to the caller instead of throwing. That looks lax until you consider that the device-code grant reports "the user has not signed in yet" as an HTTP 400 with a machine-readable body, so the ordinary path of a sign-in has to be readable, not exceptional. Every rule in both files was checked by reverting it and confirming a test fails. Two of those reverts found real gaps rather than confirming coverage: a failure path that parsed cleanly but carried no error code had no test, and would have polled until the deadline against a service that had already answered. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The last provider still missing a ModelLister. Foundry needs two, because its two connection kinds answer different questions: a project connection lists deployments, since a deployment name is what a user actually writes in model.id, while an Azure OpenAI key connection has no deployment sub-resource and lists the lower-level model catalog instead. Routing the wrong one at either would return names that cannot be invoked. An unsupported connection kind is refused before any request goes out, so a configuration mistake reads as one rather than as a transport failure from whichever service was guessed. The wire mapping itself already existed and is covered by the shared discovery vectors; this is the transport and dispatch around it. Credential lookup gains a raw-map overload, which lets listing see the undeclared snake_case aliases a host may have written. The typed executor path still cannot, because Java drops undeclared fields at load — but listing takes its connection as raw JSON, so on this path Java and Rust now accept the same connections. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The hygiene workflow rejects blank lines at end of file, and seven Java sources had one. Strip them. While here, close two related gaps the new runtime opened. gradlew was committed non-executable, so `./gradlew` fails on Linux and macOS; mark it 100755. And .gitattributes lists an explicit LF entry for every other source language but had none for Java or Gradle, so those files were relying on the `*` default. Add them, along with `*.jar binary` to pin what was already auto-detected. These entries match what the files already contain, so nothing is renormalized. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Every other runtime has a build-and-test workflow, but the Java one landed without any, so its 661 tests would never run on a pull request. Add prompty-java-check.yml, modelled on the existing per-runtime checks: ubuntu and windows, JDK 21 to match the Gradle toolchain, and the offline suite only. Live provider tests carry the "live" tag and stay excluded unless -PliveTests is passed, so the job needs no secrets and is safe on pull requests from forks. Also add Java to CodeQL, which scans every other language in the repo. The matrix now states a build mode per language, replacing the conditional Autobuild step. Java and Rust use none; C# and Go keep autobuild. Rust is listed as none because CodeQL rejects autobuild for it outright, which means the old Autobuild step was never doing anything for Rust -- init had already built the database in the default buildless mode. Making the mode explicit records what actually runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Seth Juarez (sethjuarez)
force-pushed
the
sejuare-microsoft/java-runtime-implementation
branch
from
August 4, 2026 12:58
785d645 to
4594f56
Compare
The attribute block in the role-marker pattern was written as a repeated `\w+\s*=\s*"?[^"]*"?\s*,?\s*`. The value class there matches commas and the closing bracket as well as the value itself, so it overlaps the start of the next iteration: an unterminated block such as `user[a= a= a= ...` can be divided among the iterations in exponentially many ways, and the missing bracket forces the engine to try all of them. Measured on JDK 21, rejecting that line cost 1.6s at ten repetitions, 12s at eleven and 52s at twelve, and did not finish inside five minutes at fourteen. That line is reachable. Rendered text embeds template variable values, so a marker-shaped line is attacker-influenced -- which is the same reason strict mode exists. Rewrite the loop so a value is either a quoted run or a run containing no quote, comma or bracket. The two branches cannot straddle an iteration boundary, so the division is forced rather than searched, and possessive quantifiers stop the engine attempting one anyway. Every attribute form the sibling runtimes exercise still parses, including a quoted value containing a comma, which a narrower value class would have broken. Three malformed forms are no longer accepted: an unquoted value containing a bracket or a comma, and an unbalanced quote. Only content can result, and the sole attribute marker in the repository is unaffected, as is the `role[nonce="..."]:` form the renderer stamps. Rust carries the same formulation and is safe only because its engine is a non-backtracking automaton; Python and TypeScript carry it on backtracking engines and appear to share the exposure. C# is unaffected, matching the block as a single lazy group. The pathological input is sized so the two outcomes stay distinguishable and so a regression leaks its uninterruptible thread for seconds rather than for the life of the test JVM. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Emitter 0.4.3 lands the J1-J8/J10/J18/J19 fixes reported upstream, so the
post-emit normalization no longer has to synthesize them. Pin the emitter
exactly, regenerate, and delete every rewrite that 0.4.3 makes redundant,
keeping only the rewrites whose defect signature is still present in raw 0.4.3
output.
Retained: J9 (named collections), J11 (double-escaped test expectations),
J12 (structured assertions on object-valued fields, reported as J20),
J13 (unreachable integer shorthand branch), J14 (derived save re-running
postSave), J15 (missing SaveContext knobs), J16 (eager optional defaults),
J17 (required enums initialized to null) and J21 (enum assertions comparing
against the constant name).
Removed: J1, J2, J3, J4, J6, J7, J8, J10, J18, J19, and the synthesized
generated-example registry - 0.4.3 emits its own TypraGeneratedTests. Each
retained rule keeps a floor in EXPECTED_MINIMUMS, so if a later emitter fixes
one the guard fails loudly rather than leaving a silent no-op behind. J13 and
J17 produce no test failure, so those floors are the only thing standing
between a silent upstream fix and an undetected divergence.
Regenerating forces two companion changes into the same commit, because the
tree does not compile without them.
The emitter creates a ${TypeName}Methods helper for each @method declaration
only when the file is missing, and never rewrites it, so the seam is the
designated home for those bodies - and until now both were the emitter's
throwing stubs. Implement them against the Rust reference in model_ext.rs and
the contract in schema/model: text() concatenates TextPart values joined by
newline, and toTextContent() returns that string when every part is text and
the saved wire form of each part otherwise, so providers receive a bare string
for single-modality content and the structured form once a message carries an
image, audio or file part.
Replace the retired registry with GeneratedExamplesTest, which discovers the
emitted example classes reflectively. Without it the suite silently dropped
from 692 to 545 while Gradle still reported success.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Message.text(), Message.toTextContent() and ToolResult.text() delegate to hand-written seam classes, and the emitter creates those classes as throwing stubs when they are missing. The generated suites never call them, so a stubbed seam compiles, passes every test, and throws only in front of a provider - which is exactly how both seams stayed unimplemented. Pin the behaviour directly against the Rust reference, including the two cases Rust cannot express or does not exercise: an empty part list, which vacuously satisfies "every part is text" and must stay a string rather than becoming an empty array, and a null part list, which a Rust Vec cannot represent but a Java field can. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The cleaner deletes stale generated Java before each emit. It decided what to delete from the auto-generated marker, but nothing tested that decision, and the extension seams it must preserve are exactly the files whose loss would be silent - the emitter recreates them as throwing stubs. Extract cleanJavaOutput() so it can be driven directly and cover the branches that matter: seams are preserved, an unmarked hand-written file is refused rather than removed, and the pass is idempotent. One case is deliberately awkward - Typra's toPascalCase keeps the underscore before a digit, so a seam for type Foo_1 lands in Foo_1Methods.java and any file-name pattern would miss it. Marker-based matching is what makes that safe. Wire two jobs into the Java workflow. generation-scripts runs those tests. generation-drift regenerates and asserts runtime/java is unchanged, which catches a hand-edited generated file, a stale checked-in tree, and a non-deterministic emit. It is also the only place the shim's EXPECTED_MINIMUMS floors execute, so an upstream fix that turns a retained rewrite into a no-op now fails a build instead of rotting. It diffs only runtime/java on purpose: npm run generate omits the Rust and TypeSpec formatters that npm run build applies, so the other runtimes come out unformatted and would report a difference that means nothing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
OpenAI's strict JSON Schema dialect wants every property named in `required` at every nesting level, with semantic optionality carried by a null branch rather than by omission. Java widened `required` at the top level but not inside nested objects, so any prompt declaring a nested optional field produced a request the API rejects outright. The asymmetry was inherited from Rust, the behavioural reference, which has the same top-level-only widening. Python, TypeScript and C# already widen at every depth, and C# pins it with a dedicated test, so the majority and the API agree that recursion is correct. Rust still needs the mirror-image fix; this commit covers the Java slice only. Nested optional properties were already being given their null branch, so they were emitted as ["string", "null"] while still missing from `required` — an internally inconsistent shape. Listing them completes it rather than changing how optionality is expressed. One line does the whole job because all four strict-schema entry points — chat tool parameters, chat structured output, and the Responses API tool and text.format builders — funnel nested objects through the same recursive property walk. Objects reached through array items and union branches recurse back into it too. Non-strict output is untouched: the new condition collapses to the old one when strict is false, and a guard test pins that. Shared vectors needed no change; they cover only flat schemas, so the nested case was previously ungraded. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The offline tests assert what the wire builder emits; only a real request
proves the service accepts it. Structured output was covered live, but
with a flat schema, so the nested rule the previous commit implemented
was ungraded end to end.
This sends a prompt whose optional nested field is genuinely absent, so
a faithful schema has to let the model answer null instead of inventing
a value. It comes back as {city=London, postcode=null}.
Reverting the widening turns this test into an HTTP 400 in which OpenAI
states the rule itself: "In context=('properties', 'address'), 'required'
is required to be supplied and to be an array including every key in
properties. Missing 'postcode'." That is the clearest available evidence
that the omission was a real defect rather than a stylistic difference.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The shared vectors write `null` to mean "this field carries nothing", and the Java harness read that as "the saved value must be null exactly". That made Java the strict outlier: the generated models materialize optional collections, so a `tools` the wire never supplied arrives as an empty list and saves as `[]` rather than disappearing, and both reference runtimes emit it that way -- Rust inserts the saved collection unconditionally, and C# guards only on non-null, which an empty list passes. The other runtimes reconcile the two spellings at the comparison seam rather than in the model. Rust's vector test asserts `as_tools().is_none()`, where that helper reports `None` for an empty vector; Python checks length rather than identity, and says so in its failure message. Do the same here. The allowance is deliberately narrow. A collection carrying entries is still a real difference, an empty string is a value rather than an absence, and the key-set check is untouched, so a vector that omits a key entirely still rejects a runtime that emits it as an empty collection. Only an explicit `null` in a vector opts in. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A JVM cannot remove a name from its own environment, so `Environment` could
only ever add a value on top of the ambient one. That left a whole family of
tests -- "a missing credential is reported rather than guessed" -- silently at
the mercy of the machine running them: green on a clean checkout, red the
moment a developer or CI exported the same variable. Exporting a real Azure
token to run the live Foundry suite was enough to turn one of them red, and
`FoundryLiveTest` already carried a comment admitting a green result there was
not evidence of anything.
`Environment.mask` closes the gap. It is production API rather than test
scaffolding because a host has the same need: running a prompt without
inheriting an ambient key was previously inexpressible.
Overrides and masks share one map so that each of set/mask/clear is a single
mutation, and a concurrent lookup can never observe a gap in which the ambient
value shows through.
Seven tests that asserted absence now say so explicitly, including the load
vectors, which mask any `${env:NAME}` they reference but do not supply. The
live Foundry test drops its caveat and asserts unconditionally.
clean env ./gradlew build 703 tests, 0 failures
hostile env ./gradlew test 703 tests, 0 failures
(hostile = AZURE_INFERENCE_CREDENTIAL, AZURE_OPENAI_{ENDPOINT,
API_KEY,DEPLOYMENT}, ANTHROPIC_API_KEY, OPENAI_API_KEY,
PROMPTY_TEST_VALUE, MISSING_VAR, NONEXISTENT all exported)
Before this change the hostile run failed.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The shim header explained why each rewrite exists but never said what has to happen for the file to go away, so a reviewer could not tell whether it was a permanent fixture or a tracked temporary one. Retirement is not deleting one file. The shim is wired into normalize-typra-output.mjs and referenced from clean-java-output.mjs, the Java README, ModelNormalizationTest and the workflow path filter, so record the whole set together with the regeneration and full-suite run that has to pass afterwards. Also record the two emitter defects that currently block that evaluation, confirmed against emitter commit 7595113. Both are stated as behavioural requirements rather than as a prescribed implementation. The enumValues note calls out that only the field default is wrong and that suppressing empty lists on save would be incorrect, since that would stop an explicitly supplied [] from round-tripping. Comment-only: regeneration output is byte-identical and the Java suite still passes 703 tests across 77 files with no generated-file drift. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…contract Recursive strict widening was already implemented for both schema paths, but only the tool-parameter test graded the whole contract. The structured-output test checked the required list and the nullable branch while leaving nested and top-level additionalProperties unasserted, so that path could have regressed independently of the tools path and still gone green. Assert the remaining terms the canonical vector pins down, and add the one the schema alone cannot express: the response_format envelope has to declare strict. Widening every nested key into required is only correct under strict mode, so if the envelope ever stopped declaring it the request would be wrong while every schema assertion carried on passing. Behaviour is unchanged; this is assertion coverage only. Both mutants are killed: dropping the nested `strict || isRequired` fails the tool-parameter and structured-output tests together, and flipping the envelope to `strict: false` fails the new assertion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
`cleanJavaOutput` is fail-closed: a file under the generated roots that carries neither the generated marker nor a seam marker aborts the build rather than being deleted. That is the right default, but it made the seam marker a single hard-coded string, so the first emitter release that changes the marker would stop the build with "refusing to clean" instead of a useful diagnostic -- and the pinned 0.4.3 prose marker, the `<typra-extension-seam>` tag, and the `<typra-editable-seam>` tag are all in play across in-flight emitter work. Accept all three as an explicit allow-list so hand-written seam bodies survive that transition. The exposure is bounded deliberately: an unrecognised marker still aborts, generated files open with the generated marker and are deleted before the seam check runs, and the tag spellings must match the opening line exactly so a marker cannot be smuggled in as a prefix of unrelated content. Only the pinned prose marker matches by prefix, because its documented form continues with a trailing sentence. Verified byte-neutral: `npm run generate` at the current 0.4.3 pin leaves every generated Java file unchanged, and both real seam files (MessageMethods, ToolResultMethods) are still preserved. schema/scripts: 11/11 node tests pass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
PR #447 (head 31ff17d) settled the two cross-runtime contracts this runtime had left open. Both are now graded directly. `recursive-array-valued-entry-rejection` requires an immediate array value inside a name-keyed object to be rejected at every recursive named-collection boundary, with a diagnostic carrying the path and the category. Probing the built classes shows Java already conforms at all three reachable boundaries, so this is pinned rather than fixed: * top level inputs.firstName * inside a list element tools[].parameters.toolArg * through a subclass inputs.cfg (ObjectProperty) .properties.nestedField Each assertion names the collection, the offending entry, and the word `array`, using a distinct entry name per boundary so a degraded generic diagnostic cannot satisfy all three. The pre-existing top-level test only checked the path, so it gains the category assertion the contract wants. `named-collection-lossless-fallback` requires object encoding only when every name is non-empty and exactly unique, and the whole ordered array otherwise, with duplicate detection before the map is built. `duplicateNamesCollapseOnSave` records that Java does not yet do this: two duplicate-named entries load as two entries but save as a single key, silently discarding the earlier payload. The collapse happens in emitter-generated save code, so it cannot be fixed from this PR without hand-editing generated output; the test is a characterization that fails loudly at its `assertInstanceOf(Map.class, ...)` the moment the lossless array fallback ships, at which point it should be inverted. runtime/java: 724 tests, 0 failures, 0 errors, 2 known skips, 80 files (live providers enabled). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prompty has runtimes in C#, Go, Python, Rust, and TypeScript. This adds the sixth: a Java runtime built on canonically generated models, matching the Rust runtime's behaviour and graded against the same shared spec vectors.
What is here
27 commits. The Java runtime is four Gradle modules —
prompty,prompty-openai,prompty-anthropic,prompty-foundry— covering load, render, parse, prepare, the extension protocols and registry, three providers with execution and processing, tool turns, structured output, streaming, the turn engine with durability and resume, memory scoring, and a reference harness with replay verification.722 tests, 0 failures. 703 offline, 19 live against real provider APIs (2 skipped without Azure/Foundry credentials). The offline suite runs in CI on Linux and Windows.
The model layer is generated, not hand-written
The prerequisite:
schema/tspconfig.yamlhad no Java Typra target, so there were no Java models to build on.I did not hand-write a model layer. Adding Java emission is the correct fix, and I raised it with the coordinator as a Typra blocker along with a concrete emitter contract. Typra has since shipped that emission: this PR pins
@typra/emitter0.4.3 exactly.0.4.3 does not yet emit compiling Java for this schema, so the pipeline still runs a deterministic normalization pass (
schema/scripts/normalize-java-output.mjs) over the output. It is a build step, not a fork — the models stay generated and regenerable, and no generated file is edited by hand.That pass began at 17 catalogued defects. Against 0.4.3 it is down to 9, because I verified each fix by probing raw shim-free output for the defect's own signature rather than assuming a green suite meant a rule was dead. J1, J2, J3, J4, J6, J7, J8, J10, J18 and J19 are gone, along with the generated-example registry the shim used to synthesize — 0.4.3 emits its own.
It is not a place to quietly diverge:
Two of the nine — an unreachable integer-shorthand branch, and required enums initialized to
null— produce no test failure at all. I found them by reading raw output. The count floors are the only thing that will notice when they are fixed upstream.335 of the 460 changed Java files are generated. No generated file was edited by hand.
One seam the generated code cannot fill
Typra emits a
\Methodshelper for each@methoddeclaration, creates it only when missing, and never rewrites it. That is the right design — it is where hand-written bodies belong — but it has a sharp edge worth naming.Message.text(),Message.toTextContent()andToolResult.text()shipped as the emitter's throwing stubs, and the 681-test suite was green the whole time. Generated tests never call@methodmembers, so nothing noticed. Worse, on a checkout missing those files the emitter recreates the stubs, so a drift check stays clean and the failure surfaces only in front of a live provider.They are implemented here against the Rust reference and pinned by a dedicated test, including the two cases Rust cannot express or does not exercise: an empty part list, which vacuously satisfies "every part is text" and must stay a string rather than becoming an empty array, and a null part list, which a Rust
Veccannot represent but a Java field can.While checking that parity I found the spec says
text()joins with a newline (message.tsp:30), which Rust and Java do — but Python and TypeScript join with the empty string, and TypeScript returns[]where the others return"". Pre-existing, unrelated to this PR, reported to the runtime-port owner.Parity method
Rust is the behavioural reference. Every provider and pipeline stage is graded by the shared JSON fixtures in
spec/vectors/: load (25), render (23), parse (15), wire (27 × 2 providers), process (21 × 2), engine turns (5), agent turns (28), discovery (7), enrichment (9), and 5 replay scenarios.Where Java deliberately differs from Rust, the reason is stated in a comment at the site. Two examples: Java's engine is synchronous where Rust is
async, and Java's Foundry OAuth poll loop takes an injectable endpoint, sleeper, and clock so its state machine is testable at all — Rust hardwires reqwest and tokio and covers none of that loop.Defects this work surfaced
Working against a reference implementation found bugs in more than the port.
A shared fixture was wrong.
agent_vectors.jsonusedparameters: {properties: [...]}, a shape the TypeSpec does not define. Rust silently dropped it; Java and C# reject it. Corrected in 34 places and re-verified green against Rust — so this fixture now actually grades what it claims to.A nonce bug present in Rust too.
parseAttributescoerced all-digit nonces to an integer, dropping leading zeros and failing validation roughly 1 render in 18,600. Fixed here; reported for Rust, not fixed in this PR.An indeterminate stream failure was being demoted to determinate, because
StreamFailureextendsErrorChunkand so matched the wrong arm first. That is the difference between "safe to retry" and "may have already had an effect".A dead branch in both runtimes. The second arm of the
messages_updatedrule is structurally unreachable. Left as-is to preserve parity, noted here.A regular expression shared with three other runtimes can be made to hang. The role-marker pattern matched an attribute block with a repeated
\w+\s*=\s*"?[^"]*"?\s*,?\s*, whose value class also matches the separator and the closing bracket. An unterminated block such asuser[a= a= a= ...therefore divides among the iterations exponentially many ways: on JDK 21 that cost 12s at eleven repetitions, 52s at twelve, and over five minutes at fourteen. Rendered text embeds template variable values, so the line is attacker-influenced. Fixed here by making the loop unambiguous. Rust carries the identical pattern and is safe only because its engine is a non-backtracking automaton; Python and TypeScript carry it on backtracking engines and appear to share the exposure. C# is unaffected. Not fixed for the other runtimes in this PR.CodeQL was not scanning Java, and its
Autobuildstep was a no-op for Rust. Java had no build-and-test workflow either, so this suite would never have run on a pull request. Both are fixed:prompty-java-check.ymlruns the offline suite on Linux and Windows, and CodeQL now declares a build mode per language — CodeQL rejectsautobuildfor Rust outright, so the previous conditional step was doing nothing there.Strict structured output omitted nested keys from
required. OpenAI requires every property name to appear inrequiredat every nesting level, with optionality expressed as a nullable union instead. Java widened only the top level, so a nested optional was emitted as["string","null"]yet left out ofrequired— an internally inconsistent shape the service rejects with a 400 naming the missing key. Fixed in the shared recursive property walk, which covers all four strict entry points at once: chat tool parameters, chat structured output, and the Responses API tool andtext.formatbuilders. A live test now exercises a genuinely-absent nested optional and gets{city=London, postcode=null}back. Rust has the identical top-level-only widening and will hit the same 400; the canonical cross-runtime session owns that fix. The shared vectors contain only flat schemas, which is why this survived unnoticed in every runtime.Vector files disagree on shape — some are bare arrays, some are
{description, vectors: [...]}objects. Handled, but worth normalizing.Evidence
"Hostile env" means the suite was re-run with every credential the tests assert the
absence of exported into the process:
AZURE_INFERENCE_CREDENTIAL,AZURE_OPENAI_{ENDPOINT,API_KEY,DEPLOYMENT},ANTHROPIC_API_KEY,OPENAI_API_KEY,PROMPTY_TEST_VALUE,MISSING_VAR,NONEXISTENT. That run used to fail. Seven testsasserted a variable was missing without any way to say so -- a JVM cannot unset its own
environment, so they were green only by luck of the machine.
Environment.maskfixesthat, and the live Foundry test that previously carried a "a green result here is not
evidence" caveat now asserts unconditionally.
Generation is deterministic: two consecutive
npm run generateruns produce byte-identical Java — 187 model files, same SHA-256 digest. Averify Java generation is reproduciblejob now enforces that in CI, which also makes it the one place the shim's count floors actually execute.Live runs hit real OpenAI (11 tests), Anthropic (5 tests, 9.4s), and Foundry (3 tests, 2 skipped for absent Azure credentials) — real latency, not mocks. Credentials came from a local
.envthat is gitignored and untracked; I verified both before pushing.Every behavioural rule in this PR was checked by reverting it and confirming a test fails. That is not ceremony — it found five genuine gaps that passing tests had hidden, including a context-trimming suite that was entirely vacuous because the mock executor replayed fixed responses no matter what it was sent, and an OAuth failure path that would have polled until its deadline against a service that had already answered.
Notes for review
runtime/java/andschema/, this PR changes four files: the agent-vector fixture correction, two CI workflows, and.gitattributes(which had entries for every language except Java and Gradle).FoundryArm's paging loop has no direct unit test — it needs an injected transport seam like the one OAuth has.disabled each time. The published
0.4.10fixes three of the nine (J15SaveContextknobs, and both halves of J16 optional-default materialization) but does not compile:
it emits an invalid Java identifier for nested typed assertions on a dotted path
(
ApiKeyConnection instance1.modelConnectionValue = ..., 24 sites) and re-appends.valuetoString-typed discriminators (114 sites, a regression of J19). J13 ishalf-fixed — the two numeric guards are disjoint now, but
LongandDoublematchneither, so a YAML
3.14loads with no kind at all. All reported upstream withvectors and required replacements. Staying pinned at 0.4.3 until a build compiles. I have since probed four further
candidate builds as the emitter's downstream consumer. Emitter commit
7595113is thefirst that compiles: it clears the dotted-identifier and
.valuedefects, fixes thegenerated-test escaping and enum-case assertions, and brings the residual down to two
emitter defects — J13's numeric families, and the collection half of J16, where only
Property.enumValues's field default is wrong (its save guard is already correct, sosuppressing empty lists there would wrongly break explicit
[]round-trips). Both arereported upstream with verbatim snippets. The full retirement criteria — every integration
reference, not just this file — are recorded in the shim header.