feat(ai): add Spring AI, and fix the stem math it surfaced

.claude/skills/release-pipeline/SKILL.md

@@ -13,9 +13,9 @@ before answering "is X supported" — they answer different questions.
 
 | | |
 |---|---|
-| Projects | `boot`, `framework`, `security` |
+| Projects | `ai`, `boot`, `framework`, `security` |
 | Version format | GA `major.minor.patch` only — M/RC/SNAPSHOT are rejected by `isGaVersion` |
-| Buildable ranges | `boot` → `3.3.0`–`<4.0.0` and `4.0.0`–`<4.0.8` (both synthesized, different paths) and `>= 4.0.8` (archive); `framework` → `>= 6.1.0` (overlay); `security` → `>= 6.2.0` (overlay, two eras) |
+| Buildable ranges | `ai` → `>= 1.0.0` (overlay); `boot` → `3.3.0`–`<4.0.0` and `4.0.0`–`<4.0.8` (both synthesized, different paths) and `>= 4.0.8` (archive); `framework` → `>= 6.1.0` (overlay); `security` → `>= 6.2.0` (overlay, two eras) |
 | Published | check `catalog.json`; an empty `projects` object means nothing has shipped yet |
 
 A project is a sequence of **layout eras** (`LayoutEra`, ADR-0004), not a single floor. An era
@@ -30,10 +30,14 @@ without the other yields a tree that classifies but converts wrongly:
 | `framework` `>= 6.1.0` | `framework-docs` | `overlay` | the committed `antora.yml`, topped up with the version and the attributes the build contributes | no |
 | `security` `6.2.0` – `<6.5.1` | `docs` | `overlay` | the committed `antora.yml`, topped up with attributes derived from `gradle/libs.versions.toml` and `gradle.properties` | no |
 | `security` `>= 6.5.1` | `docs` | `overlay` | the same, plus the `modules/ROOT/examples/docs-src` symlink that era added | no |
+| `ai` `>= 1.0.0` | `spring-ai-docs/src/main/antora` | `overlay` | the committed `antora.yml` unchanged — its build contributes no attribute at all | no |
 
 `overlay` is the cheapest to add and the one to reach for first on a new project: check what
 that project's `generateAntoraResources` actually produces. Spring Framework's is one
-attribute (`spring-version`), so nothing is downloaded at all.
+attribute (`spring-version`), so nothing is downloaded at all. Spring AI's is the degenerate
+case — its whole generated template is `version` plus `prerelease`, neither of which is an
+asciidoc attribute, so the overlay is pure passthrough and `generatedAttributesFor` returns
+`{}`.
 
 Eras need not be contiguous, and `eraFor` returns `undefined` below the oldest floor:
 
@@ -48,6 +52,11 @@ Eras need not be contiguous, and `eraFor` returns `undefined` below the oldest f
 - **4.1.0** is a *publication* fact, not a layout one: it is inside the archive era but its
   zip is unpublished, so it is probed over the network (below) instead of being encoded as an
   unbuildable range that would keep refusing it after upstream publishes.
+- **Spring AI 0.8.x** is a publication fact too, and a permanent one, so it *is* encoded as a
+  floor: `spring-ai-docs/src/main/antora/antora.yml` is byte-identical at v0.8.0 and v2.0.1, but
+  `org/springframework/ai/spring-ai-bom` on Maven Central begins at `1.0.0-M5` — 0.8.0 and 0.8.1
+  went to Spring's milestone repository only, and no consumer can pin a dependency to the
+  versions those 50 pages describe.
 
 The archive era's floor is not a compatibility guess either: upstream published that zip for
 2.2.x–2.4.2, then not again until 4.0.8. The synthesized era's 3.3.0 floor has nothing to do
@@ -150,6 +159,15 @@ exercise every AsciiDoc construct: adding Spring Framework surfaced hand-written
 `floating_title`, block images and role `<span>`s — none of which Boot uses. Run `--strict` and
 add a rule per construct.
 
+**`--strict` does not catch everything, so read the output too.** It gates on the converter's
+own unknown-construct warnings, which fire only for a node the walker does not recognise.
+Anything Asciidoctor substitutes *before* the walker runs is invisible to it: Spring AI's 16
+inline `stem:[…]` expressions arrive as plain text in MathJax delimiters (`\$…\$` for asciimath,
+`\(…\)` for latexmath), and were escaped as prose into `\\$\\vec{a}\\$` across a page of vector
+maths while `--strict` reported zero warnings. `escapeText` in `inline-html.ts` now carries them
+through as `$…$`. Diff a page or two of a new corpus against the upstream site before believing
+a clean run.
+
 Then:
 
 - extend `tests/unit/upstream-sources.test.ts`

NOTICE

@@ -34,6 +34,11 @@ Spring Security
   License:   Apache License, Version 2.0
   Notice:    https://github.com/spring-projects/spring-security/blob/main/NOTICE.txt
 
+Spring AI
+  Copyright (c) 2023-present, the Spring AI authors.
+  Source:    https://github.com/spring-projects/spring-ai
+  License:   Apache License, Version 2.0
+
 Spring Data (and its sub-projects: JPA, JDBC, MongoDB, Redis, etc.)
   Copyright (c) 2008-present, the Spring Data authors.
   Source:    https://github.com/spring-projects/spring-data-*

README.md

@@ -2,7 +2,7 @@
 
 > Pre-converted, version-pinned Spring documentation in Markdown — distributed as GitHub Release archives.
 
-This repository hosts LLM-friendly Markdown versions of the Spring ecosystem reference docs (Framework, Boot, Security, Data, Cloud, etc.) generated from upstream AsciiDoc sources. Each `(project, version)` pair is published as a separate GitHub Release so consumers can fetch exactly the docs that match their project's dependencies.
+This repository hosts LLM-friendly Markdown versions of the Spring ecosystem reference docs (Framework, Boot, Security, AI, Data, Cloud, etc.) generated from upstream AsciiDoc sources. Each `(project, version)` pair is published as a separate GitHub Release so consumers can fetch exactly the docs that match their project's dependencies.
 
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](./LICENSE)
 [![Upstream](https://img.shields.io/badge/upstream-Apache--2.0-green)](./NOTICE)
@@ -180,7 +180,7 @@ Consumers should prefer `catalog.json` over scraping the Releases page — it's
 ## Generation pipeline
 
 1. **Detect new upstream release** — `nightly-detect.yml` runs `detect-upstream-versions.ts`, which diffs upstream's tags against `catalog.json` and files one issue per missing GA version. It builds nothing: a new upstream line can change the documentation layout, so a human decides.
-2. **Fetch** — `fetch-upstream.ts` assembles one Antora content source from two halves. The authored half is always a sparse checkout of the docs subtree at the release tag. The generated half — the resolved `antora.yml` attributes, the sample sources `include-code::` reads, and the configuration-property metadata `configprop:` validates against — depends on the version's layout era ([ADR-0004](./.please/docs/decisions/0004-synthesize-3x-component.md)): Spring Boot 4.0.8+ merge the content archive published to Maven Central; Boot 3.3-4.0.7 reconstruct it from the tag plus the published `spring-boot-*` jars, because those archives are excluded from Spring's Maven Central sync until 4.0.8 ([ADR-0006](./.please/docs/decisions/0006-synthesize-4-0-x-component.md) covers 4.0.0-4.0.7, which reconstruct the same way from paths 4.0.0 moved); and Spring Framework 6.1+ and Spring Security 6.2+ need neither, because the tag already carries a complete descriptor and its own examples — the committed `antora.yml` is overlaid rather than rebuilt, with the version for Framework and, for Security, with the documentation URLs and four dependency versions its build resolves, all read out of the committed version catalog and `gradle.properties`. Either way: no submodules, no Gradle, no JVM.
+2. **Fetch** — `fetch-upstream.ts` assembles one Antora content source from two halves. The authored half is always a sparse checkout of the docs subtree at the release tag. The generated half — the resolved `antora.yml` attributes, the sample sources `include-code::` reads, and the configuration-property metadata `configprop:` validates against — depends on the version's layout era ([ADR-0004](./.please/docs/decisions/0004-synthesize-3x-component.md)): Spring Boot 4.0.8+ merge the content archive published to Maven Central; Boot 3.3-4.0.7 reconstruct it from the tag plus the published `spring-boot-*` jars, because those archives are excluded from Spring's Maven Central sync until 4.0.8 ([ADR-0006](./.please/docs/decisions/0006-synthesize-4-0-x-component.md) covers 4.0.0-4.0.7, which reconstruct the same way from paths 4.0.0 moved); and Spring Framework 6.1+, Spring Security 6.2+ and Spring AI 1.0+ need neither, because the tag already carries a complete descriptor and its own examples — the committed `antora.yml` is overlaid rather than rebuilt, with the version for Framework and, for Security, with the documentation URLs and four dependency versions its build resolves, all read out of the committed version catalog and `gradle.properties`. Spring AI tops up nothing at all: its build generates only the component version, which the overlay writes anyway. Either way: no submodules, no Gradle, no JVM.
 3. **Convert** — `convert.ts` drives Antora's own pipeline modules with Spring's Asciidoctor extensions registered, so `xref:`, `include::`, `include-code::`, `javadoc:` and `configprop:` are resolved by the same code that produces docs.spring.io. Our converter then emits Markdown from the resolved AST: GFM alerts for admonitions, headed code fences for tab groups, relative `.md` links for internal xrefs, absolute `docs.spring.io` URLs for references into components we do not build. An unhandled construct fails the build rather than being dropped.
 4. **Package** — `package-release.ts` writes `NOTICE`, checksums every file, and builds a reproducible `tar.gz`: entries sorted, timestamps and ownership pinned, gzip's mtime field suppressed. The same converted tree always yields byte-identical bytes.
 5. **Release** — a `<project>-<version>` tag push runs `release.yml`, which rebuilds from the tag, verifies the manifest against it, and publishes the archive, its checksum and the manifest.
@@ -194,7 +194,7 @@ The pipeline is matrix-parallelized: typical full ecosystem rebuild (~10 project
 |---|---|
 | **Tag immutability** | Once a `<project>-<version>` tag is published, it is not deleted. Re-generation creates a new tag suffix (`framework-6.2.0+rebuild.1`) and updates `catalog.json` to point at the latest |
 | **Pre-release versions** | Not built. Only GA versions of upstream projects |
-| **Buildable versions** | Spring Boot `3.3.0`+. 3.2 and older predate the Antora component entirely. 4.0.0-4.0.7 moved to the 4.x layout but published no content archive, so they are reconstructed from the tag like 3.3-3.x rather than downloaded; 4.1.0 is tagged with no archive published yet |
+| **Buildable versions** | Spring Boot `3.3.0`+, Spring Framework `6.1.0`+, Spring Security `6.2.0`+, Spring AI `1.0.0`+. Boot 3.2 and older predate the Antora component entirely; 4.0.0-4.0.7 moved to the 4.x layout but published no content archive, so they are reconstructed from the tag like 3.3-3.x rather than downloaded, and 4.1.0 is tagged with no archive published yet. Spring AI 0.8.x shares the 1.x layout, but its artifacts never reached Maven Central, so no consumer can pin a dependency to what those docs describe |
 | **Reconstructed appendix** | The generated appendix — auto-configuration class listings and configuration-property tables — is a Gradle build output with no published equivalent, so every reconstructed version omits it: ~101 pages for 3.3-3.x, 92 for 4.0.0-4.0.7. The prose corpus (reference, how-to, tutorial, specification) is complete |
 | **Coverage window** | The newest N missing GA versions per project, N being the `limit` input of `matrix-build.yml` (default 3) |
 | **Backfill** | Older versions can be requested via issue and built on-demand |

scripts/lib/inline-html.ts

@@ -70,6 +70,43 @@ const BACKTICK_RUN = /`+/g
 /** Markdown characters that change how a text run parses. */
 const MARKDOWN_SPECIALS = /[\\*_[\]]/g
 
+/**
+ * One inline `stem:[…]` expression, as Asciidoctor has already rendered it.
+ *
+ * The stem macro is substituted away before a converter ever sees the node: it
+ * arrives here as ordinary text wrapped in the MathJax delimiters Asciidoctor
+ * emits — `\$…\$` for asciimath, which is the default and what Spring AI's
+ * playbook selects, and `\(…\)` for latexmath.
+ *
+ * That makes it invisible to the unknown-construct warning `--strict` gates on,
+ * so it has to be recognized here or not at all: escaped as prose,
+ * `\$\vec{a}\$` reaches the reader as `\\$\\vec{a}\\$`, and every backslash the
+ * expression carries doubles with it.
+ *
+ * A delimiter Asciidoctor emits carries exactly one backslash. Source that
+ * escapes one in order to write about it — `\\$x\\$`, or `\\(x\\)` — arrives
+ * with two, and is prose. The lookbehinds tell the two apart: without them the
+ * scan starts at the second backslash and publishes `$x\$`, live math where
+ * the page meant to show a delimiter.
+ *
+ * A singly escaped `\$x\$` written as prose stays indistinguishable, because
+ * Asciidoctor emits the same text for it as for `stem:[x]`. Nothing here can
+ * separate those: the substitution runs before the walker does and leaves the
+ * text node no provenance — not even the `stem` attribute, which is unset on
+ * the very pages that carry the macro.
+ */
+const STEM_SPAN = /(?<!\\)\\\$([\s\S]*?)(?<!\\)\\\$|(?<!\\)\\\(([\s\S]*?)(?<!\\)\\\)/g
+
+/**
+ * Any single unescaped stem delimiter.
+ *
+ * Used on what {@link STEM_SPAN} did not consume: a run whose delimiters all
+ * pair up leaves none of these behind. An escaped delimiter is skipped for the
+ * reason {@link STEM_SPAN} skips it — prose, not a stray half of a pair, so it
+ * must not make the run look unpaired.
+ */
+const STEM_DELIMITER = /(?<!\\)\\[$()]/
+
 /** A leading `#`, which would otherwise start a heading. */
 const LEADING_HASH = /^(\s*)#/
 
@@ -294,11 +331,77 @@ function codeSpan(text: string): string {
  * Deliberately conservative: over-escaping ordinary prose costs more readability
  * than the rare under-escape costs correctness. `#` only matters at line start.
  */
-function escapeText(text: string, atLineStart: boolean): string {
+function escapeProse(text: string, atLineStart: boolean): string {
   const escaped = text.replace(MARKDOWN_SPECIALS, '\\$&')
   return atLineStart ? escaped.replace(LEADING_HASH, '$1\\#') : escaped
 }
 
+/**
+ * Escape a text run, carrying any `stem:[…]` expression in it through unescaped.
+ *
+ * A stem expression is notation, not prose: `\vec{A}`, `a_1` and `^2` all mean
+ * something to a formula renderer and nothing to Markdown, so the escaping that
+ * protects prose is exactly what destroys them. They are re-emitted between `$`
+ * delimiters — the one inline-math spelling Markdown renderers and LLM consumers
+ * both read — with the expression itself byte-for-byte as upstream wrote it.
+ *
+ * Only the text between the delimiters is exempt; everything around it is
+ * escaped as before, including the `#` rule, which still applies to whichever
+ * segment actually begins the line.
+ *
+ * **A run whose delimiters do not all pair up is withheld, not guessed at.**
+ * Pairing is positional, so given `\$ a \$ b \$` there is no way to tell which
+ * delimiter is the stray one, and pairing the first two would splice ` a ` into
+ * a formula and strip its escaping — turning any `*`, `_` or `[` it holds into
+ * live Markdown on the published page. That is the silent half-rendering
+ * ADR-0004 forbids, so the whole run is escaped as ordinary prose instead and
+ * reported through {@link InlineOptions.onUnpairedStem}: `--strict` then fails
+ * the build and names the page, which is how a new construct earns a rule. An
+ * empty expression is withheld on the same grounds — `$$` is display-math
+ * delimiters to most renderers, so emitting it would change the construct.
+ */
+function escapeText(
+  text: string,
+  atLineStart: boolean,
+  onUnpairedStem?: (run: string) => void,
+): string {
+  // Every stem delimiter starts with one, and ordinary prose rarely contains
+  // any — so this skips the scan below for almost every run in a corpus.
+  if (!text.includes('\\'))
+    return escapeProse(text, atLineStart)
+
+  const spans = [...text.matchAll(STEM_SPAN)]
+  if (spans.length === 0)
+    return escapeProse(text, atLineStart)
+
+  let out = ''
+  let remainder = ''
+  let cursor = 0
+  let lineStart = atLineStart
+
+  for (const span of spans) {
+    const before = text.slice(cursor, span.index)
+    remainder += before
+    out += escapeProse(before, lineStart)
+    // One alternative matched, so exactly one group is defined: asciimath's
+    // `\$…\$` or latexmath's `\(…\)`.
+    out += `$${span[1] ?? span[2] ?? ''}$`
+    lineStart = false
+    cursor = span.index + span[0].length
+  }
+
+  const tail = text.slice(cursor)
+  remainder += tail
+
+  const empty = spans.some(span => (span[1] ?? span[2] ?? '') === '')
+  if (empty || STEM_DELIMITER.test(remainder)) {
+    onUnpairedStem?.(text)
+    return escapeProse(text, atLineStart)
+  }
+
+  return out + escapeProse(tail, lineStart)
+}
+
 /**
  * Rewrite a resolved Antora xref target for the Markdown tree.
  *
@@ -342,6 +445,15 @@ export interface InlineOptions {
   readonly imageBase?: string
   /** Called with the `src` of an image that could not be given a base URL. */
   readonly onImageWithoutBase?: (src: string) => void
+  /**
+   * Called with a text run whose `stem:[…]` delimiters do not pair up, or that
+   * holds an empty expression.
+   *
+   * The run is escaped as ordinary prose rather than converted, so the caller
+   * has to surface it: left unreported it would ship a visibly wrong page, and
+   * guessing a pairing instead would ship a silently wrong one.
+   */
+  readonly onUnpairedStem?: (run: string) => void
 }
 
 /**
@@ -371,7 +483,13 @@ const ABSOLUTE_URL = /^[a-z][\w+.-]*:\/\//i
  * @returns Markdown equivalent of `html`.
  */
 export function inlineHtmlToMarkdown(html: string, options: InlineOptions = {}): string {
-  const { onUnknownTag, externalComponents = {}, imageBase, onImageWithoutBase } = options
+  const {
+    onUnknownTag,
+    externalComponents = {},
+    imageBase,
+    onImageWithoutBase,
+    onUnpairedStem,
+  } = options
   let out = ''
   const emit = (text: string): void => {
     out += text
@@ -380,7 +498,11 @@ export function inlineHtmlToMarkdown(html: string, options: InlineOptions = {}):
   const walk = (nodes: readonly HtmlNode[]): void => {
     for (const node of nodes) {
       if (node.kind === 'text') {
-        emit(escapeText(decodeEntities(node.value), out === '' || out.endsWith('\n')))
+        emit(escapeText(
+          decodeEntities(node.value),
+          out === '' || out.endsWith('\n'),
+          onUnpairedStem,
+        ))
         continue
       }
 
@@ -448,7 +570,11 @@ export function inlineHtmlToMarkdown(html: string, options: InlineOptions = {}):
           // It carries no children — the alt text is an attribute — so nothing
           // is walked.
           const src = node.attrs.get('src') ?? ''
-          const alt = escapeText(decodeEntities(node.attrs.get('alt') ?? ''), false)
+          const alt = escapeText(
+            decodeEntities(node.attrs.get('alt') ?? ''),
+            false,
+            onUnpairedStem,
+          )
           if (src === '' || (imageBase === undefined && !ABSOLUTE_URL.test(src))) {
             // Same rule as a block image with no base: an empty destination
             // reads as a broken image, while the alt text alone still says what