[gary-mu]

OpenAI retires gpt-4o-2024-11-20 and gpt-4.1-2025-04-14 on Jan 14, 2027. Vocabulary Complexity uses gpt-4o to write students' background knowledge (stage 1, every grade) and gpt-4.1 to rate grades 5–12 (stage 2). This PR moves both steps to gpt-6-luna.

Reports (private Learning Commons Pages site; sign in with a chanzuckerberg GitHub account):

• Grades 3–4, stage 1: Greenlight gpt-6-luna for grades 3–4?
• Grades 5–12, both stages: Greenlight gpt-6-luna for grades 5–12?

What changed

Step	Before	After
background_knowledge (all grades)	openai/gpt-4o-2024-11-20, temperature 0	openai/gpt-6-luna, no temperature sent (model default)
vocab_complexity_grades_3_4	google/gemini-2.5-pro, temperature 0	unchanged
vocab_complexity_other_grades (5–12)	openai/gpt-4.1-2025-04-14, temperature 0	openai/gpt-6-luna, no temperature sent (model default)

• evals/text-complexity/ela-reading/vocabulary-complexity/config.json: model and temperature for the two OpenAI steps. Prompts, prompt hashes, schemas and parsers are unchanged.
• Python SDK contract copy regenerated with make generate-contracts (generated file, not hand-edited).
• Vocabulary unit tests in both SDKs now expect the new model labels, and the TypeScript test checks that no temperature is sent.
• Two TypeScript doc comments (evaluators/base.ts, vocabulary-complexity.ts) now name gpt-6-luna.
• TypeScript model label (second commit). With both grades 5–12 steps on gpt-6-luna, the TypeScript SDK reported openai:gpt-6-luna+openai:gpt-6-luna while Python reported openai:gpt-6-luna. SPEC §3.4 gives one model one label, so TypeScript now names a shared model once, as Python and the modelOverride path already did. The registry conformance test derives its expected label the same way, and two tests that counted three distinct vocabulary models now count two.

Why no temperature (third commit). gpt-6-luna takes only its default: temperature: 0 returns "Unsupported parameter: 'temperature' is not supported with this model", while 1 and no parameter are both accepted. config.schema.json reserves null for exactly this case (a model that rejects any non-default value), and review asked to run models at their own defaults, so both steps pin null and neither SDK sends the parameter. The evaluation sent 1, which is also the OpenAI API default, so this should not change outputs. The first version of this PR pinned 1.

Why gpt-6-luna with no date. OpenAI lists it as the model's only snapshot, so it meets SPEC's rule for pinned model IDs (an immutable release with no dated alias).

Reasoning effort. gpt-6-luna is a reasoning model. Both SDKs send no effort setting, so it runs at the vendor default (medium); the contract has no field for it.

Expected effect

Paired runs against today's pipeline on the same texts, with the request shape the Python SDK sends (Responses API, strict structured output, no output cap). "Pass" means not significantly worse, which is the bar the migration agreed on.

Grades 3–4 (stage 1 only; 233 texts, run 2026_10_26_23_38_20):

Metric	Today	gpt-6-luna	Difference [95% CI]	p
Exact match	52.8%	52.8%	0.0 [−6.4, +6.4]	1.00
Within one level	94.8%	93.6%	−1.3 [−3.9, +1.3]	0.51
Critical error (≥2 levels off)	5.2%	6.4%	+1.3 [−1.3, +3.9]	0.51

Grades 5–12 (both stages; ≥1 Match Accuracy rule, 81 three-rater texts, run 2026_10_28_16_13_25):

Group	Today	gpt-6-luna	Difference [95% CI]	p
All grades	86.4%	82.7%	−3.7 [−12.3, +6.2]	0.61
Grades 5–8	83.0%	80.9%	−2.1 [−14.9, +10.6]	1.00
Grades 9–11	91.2%	85.3%	−5.9 [−20.6, +8.8]	0.69

What callers will notice:

• Labels move, mostly one level harder. Grades 3–4: 67 of 233 labels changed, 51 up and 16 down (p = 2×10⁻⁵), with no net change in accuracy. Grades 5–12: 35 up and 22 down (p = 0.11, not significant).
• Exceedingly complex is used in grades 5–12. gpt-6-luna assigned it 7 times, each on a text where a rater gave a 4; gpt-4.1 never did. Calibration against the raters' median moves from −0.10 to −0.02 levels.
• Labels vary between calls. gpt-6-luna samples at its default temperature, so the same text can get a different label. Across three grades 5–12 runs, 38–45% of labels changed between runs (today's pipeline: 5%), nearly always to an adjacent level. Accuracy across those runs was 79.0–85.2% (today: 86.4–87.7%); none was significantly below today (p ≥ 0.21).
• Slower. Grades 5–12 take about 9.8 s per text, up from 6.8 s. Grades 3–4 stage 1 averages 2.9 s per call.
• metadata.model changes to openai:gpt-6-luna for grades 5–12 and openai:gpt-6-luna+google:gemini-2.5-pro for grades 3–4, in both SDKs.
• One fewer failure. On one grade 5 text, gpt-4.1 kept generating until it hit its output limit; gpt-6-luna answers it.

The output shape is unchanged.

Test plan

• python3 scripts/check.py --fix: all 6 checks pass (config hashes, schemas, fixtures, notebooks, requirements).
• sdks/python: make verify passes (1,755 passed, 108 skipped, after merging main).
• sdks/typescript: npm run lint && npm run typecheck && npm run test:unit pass (1,648 tests, after merging main).
• Offline evaluation in Databricks, in the paired runs above. They call OpenAI the way the Python SDK does.
• Live smoke test of the TypeScript path, which the evaluation did not exercise (it goes through the Vercel AI SDK): RUN_INTEGRATION_TESTS=true npm run test:integration -- vocabulary-complexity with OPENAI_API_KEY and GOOGLE_API_KEY. The suite already accepts an adjacent level and retries up to 3 times, so sampling variation shouldn't make it flaky.
• Optional: run example_notebook.ipynb for one grade 3–4 and one grade 5–12 text and check metadata.model.

Not in this PR

• The hurricanes_grade7 fixture reuses an example from the grades 5–12 prompt and should be swapped before release.
• Sentence Structure's gpt-4o-2024-08-06 retires on the same date; that is a separate migration.

[Copilot]

🔵 Needs a closer look

Validation covered only Python, leaving the TypeScript Vercel AI SDK path awaiting a credentialed live smoke test to confirm the new default model supports both prose generation and strict structured output.

No open issues 🎉

What changed in this PR

Migrates Vocabulary Complexity’s OpenAI steps to gpt-6-luna ahead of the scheduled model retirements, while keeping the grades 3–4 rating model and output shape unchanged.

Changes:

• Sets background knowledge and grades 5–12 rating to gpt-6-luna at temperature 1, including the generated Python contract.
• Deduplicates TypeScript model labels when both stages use the same model, matching Python.
• Updates SDK test expectations and TypeScript documentation comments.

File	Description
sdks/​typescript/​tests/​unit/​registry-conformance.test.ts	Expects distinct model labels in execution order.
sdks/​typescript/​tests/​unit/​evaluators/​vocabulary-complexity.test.ts	Updates model labels, temperatures, and grade-specific expectations.
sdks/​typescript/​tests/​unit/​evaluators/​anthropic-override.test.ts	Updates the expected distinct model count to two.
sdks/​typescript/​src/​evaluators/​text-complexity/​ela-reading/​vocabulary-complexity.ts	Deduplicates shared model labels and updates the rating-model comment.
sdks/​typescript/​src/​evaluators/​base.ts	Updates model-override documentation to describe the two models.
sdks/​python/​tests/​unit/​evaluators/​text_complexity/​ela_reading/​test_vocabulary_complexity.py	Updates model expectations and verifies the shared-model label.
sdks/​python/​src/​learning_commons_evaluators/​contracts/​_generated/​text_complexity/​ela_reading/​vocabulary_complexity/​config.json	Synchronizes the generated Python contract with the migration.
evals/​text-complexity/​ela-reading/​vocabulary-complexity/​config.json	Sets both OpenAI steps to gpt-6-luna at temperature 1.

_{Review effort: Balanced}

[adnanrhussain]

LGTM 🚀

[slee430]

@gary-mu My understanding is that temperature can only be set when reasoning.effort is none. Since we use the default medium here, is temperature: 1 still valid, or should we set it to null so it is omitted from the request?

also tagging @adnanrhussain here since I think we had this conversation back in August.

[gary-mu]

Thanks @slee430. No change made: temperature: 1 is valid for gpt-6-luna at the default reasoning effort, so I've kept it.

• Tested directly. Both evaluation runs (Oct 26 and Oct 28) probed gpt-6-luna with no reasoning parameter, i.e. default effort. temperature: 0 returned a 400 ("Unsupported parameter: 'temperature' is not supported with this model"); temperature: 1 and omitting it were both accepted. Every call in the evaluation was then sent with temperature: 1, so 1 is the setting the results were validated at.
• The schema's rule (from docs(evals): correct the temperature guidance in the config schema #207, which corrected the guidance feat(evals): allow temperature: null so newer models use their own default #186 added) is to use null only for models that reject any non-default value, and otherwise to write the number out so the validated value is recorded rather than inherited from a model default. gpt-6-luna accepts its default of 1, so the contract records 1.
• Same as existing contracts. The seven feedback evaluators on gpt-5.4-2026-03-05, also a reasoning model at default effort, pin temperature: 1.

If OpenAI later starts rejecting any temperature at non-none effort, switching to null is a one-line change: both SDKs omit the parameter for null.

[slee430]

@gary-mu Thanks! I thought we agreed to follow the model's default settings going forwad. @aychi1 @adnanrhussain, could you confirm if that matches your understanding?

If so, I think we should use null instead of 1, since null omits the parameter and lets the model use its default. But, given that this eval may be revamped again and this is a quick fix, I don't think this needs to block the PR. But I think it would be good to follow a consistent approach going forward :)

[gary-mu]

@gary-mu Thanks! I thought we agreed to follow the model's default settings going forwad. @aychi1 @adnanrhussain, could you confirm if that matches your understanding?

If so, I think we should use null instead of 1, since null omits the parameter and lets the model use its default. But, given that this eval may be revamped again and this is a quick fix, I don't think this needs to block the PR. But I think it would be good to follow a consistent approach going forward :)

Sounds good! Changed to null to be consistent !