Skip to content

fix(openai): require nested strict schema fields - #441

Open
Seth Juarez (sethjuarez) wants to merge 1 commit into
microsoft:mainfrom
sethjuarez:sejuare-microsoft/fix-strict-nested-schemas
Open

fix(openai): require nested strict schema fields#441
Seth Juarez (sethjuarez) wants to merge 1 commit into
microsoft:mainfrom
sethjuarez:sejuare-microsoft/fix-strict-nested-schemas

Conversation

@sethjuarez

@sethjuarez Seth Juarez (sethjuarez) commented Aug 4, 2026

Copy link
Copy Markdown
Member

Summary

  • make OpenAI strict JSON Schema conversion recursively include every nested object key in required
  • preserve Prompty optionality by rendering non-required fields as nullable unions
  • pin the behavior in the canonical shared wire vectors and specification
  • load generated property subtypes in the TypeScript vector harness so nested schemas are exercised faithfully
  • exercise the exact nested optional case in the ignored live Rust integration test

Runtime audit

Runtime Result
Rust Fixed recursive nested required generation
Python Already compliant; new shared vector passes
TypeScript Provider already compliant; vector harness now preserves nested generated properties
C# Already compliant; new shared vector passes
Go / Swift No OpenAI provider surface on main
Java Matching fix and live regression coverage are in PR #444 (2355dbe1, da5a3f6e)

Verification

  • cargo test -p prompty-openai
  • cargo fmt --all -- --check
  • uv run pytest tests/test_spec_vectors.py tests/test_executor.py tests/test_structured.py -q (219 passed, 17 skipped)
  • npm run test --workspace @prompty/core -- spec-vectors.test.ts (140 passed, 17 skipped)
  • npm run test --workspace @prompty/openai (91 passed, 18 skipped)
  • npm run lint --workspace @prompty/core
  • C# SpecVectorWireTests assembly run (25 passed)
  • live OpenAI test_structured_output E2E: accepted the nested strict schema and returned {"style":{"border":null,"color":"blue"}}
  • Java PR Add a Java runtime built on generated models #444: the canonical vector passes unmodified; reverting Java's one-line fix creates exactly one failure at response_format.json_schema.schema.properties.style.required (expected 2 keys, got 1), then the restored fix passes all 695 tests

The live Java negative check also reproduced OpenAI's nested-schema rejection verbatim: In context=('properties', 'address'), 'required' is required to be supplied and to be an array including every key in properties. Missing 'postcode'.

The contract matches OpenAI's Structured Outputs guidance: all fields must be required, optional fields are modeled with null, and the rule applies to nested object schemas.

Align the recursive strict JSON Schema contract with OpenAI by marking every nested object property required while representing Prompty optional fields as nullable. Add a shared wire vector, cross-runtime harness coverage, and a live nested structured-output check.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 4, 2026 09:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR tightens Prompty’s OpenAI Structured Outputs “strict” wire contract so that every object property is listed in required recursively, while Prompty-optional fields are modeled as nullable unions (e.g., type: ["string", "null"]). It updates both the normative spec/docs and the shared wire vectors, and adjusts harnesses/tests so nested schemas are exercised end-to-end.

Changes:

  • Updated the wire-format specification/docs to define strict-mode recursive required behavior and nullable modeling for optional fields.
  • Added a new canonical wire vector (structured_output_nested_optional) that asserts nested optional handling in strict mode.
  • Updated TypeScript vector harness loading to use generated Property.load(...) (preserving nested property shapes) and updated Rust wire/tests to require nested properties under strict mode.
Show a summary per file
File Description
web/src/content/docs/specification/wire-format.mdx Documents strict-mode recursive required + nullable optionals in schema conversion.
spec/vectors/wire/wire_vectors.json Adds canonical nested-optional structured output wire vector + expected strict schema.
spec/spec.md Updates normative spec text/pseudocode for strict schema generation rules.
runtime/typescript/packages/core/tests/spec-vectors.test.ts Uses Property.load so vectors preserve generated nested property subtypes.
runtime/rust/prompty-openai/tests/wire_vectors.rs Exercises the new vector and updates strict nested required assertions.
runtime/rust/prompty-openai/tests/integration.rs Updates ignored live test to validate nested optional strict output is nullable + required.
runtime/rust/prompty-openai/src/wire.rs Ensures strict mode includes nested properties in required arrays (recursive behavior).

Review details

  • Files reviewed: 7/7 changed files
  • Comments generated: 0
  • Review effort level: Lite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants