[Paper][0.81.9] Baseline-aligned nested Text layout aborts with ‘Expect custom baseline function to not return NaN’ #3121

Environment

Installed versions, read directly from package manifests and the native Pod lockfile:

• react-native-macos: 0.81.9; native React-Core Pod: 0.81.9.
• Base react-native JavaScript package: 0.81.6; hermes-engine Pod: 0.81.6.
• React: 19.1.4.
• react-native-test-app: 5.3.3.
• macOS: 27.0.1, arm64.
• node -v: v24.18.0.
• npm -v: 11.16.0.
• yarn --version: 4.18.0 (Yarn monorepo, pnpm linker; package versions above were inspected rather than running npm ls).
• xcodebuild -version: Xcode 26.4.1, build 17E202.
• Renderer: Paper / Legacy Architecture, RCT_NEW_ARCH_ENABLED=0, fabric_enabled: false, bridgeless_enabled: false. Metro's Legacy Architecture warning confirms the runtime architecture.

The installed RCTTextShadowView.mm and Yoga Baseline.cpp are byte-for-byte matches, by Git blob hashes, for upstream tag v0.81.9, commit 3693ac91a00df6374e610a0a06b0f90aaa1fe4b3.

Steps to reproduce the bug

Verified reproduction: Fluent UI React Native's macOS Storybook catalog, with Paper enabled and the original baseline-aligned CompoundItemLayout demos intact.

1. Use the public CompoundItemLayout fixture. Keep textBaseline: { alignItems: 'baseline' } applied to the row's style and inline contentStyle; do not apply a centered-layout workaround.
2. Configure and build the macOS test app for Paper with the environment/options above. Pod preparation, JavaScript bundling, and the native build all succeed.
3. Launch the app and traverse stories in catalog order. The first 24 stories render, including primitives-compound-item-layout--default.
4. Switch from that Default story to primitives-compound-item-layout--overview. The native ReactTestApp process terminates with the error below. Repeated normal Paper catalog smoke runs reproduce this transition failure.

The local tooling sequence is:

# From apps/storybook; requires the local, not-yet-published --paper CLI patch.
yarn storybook prep --macos --paper
yarn storybook bundle --macos --paper
yarn storybook build --macos --paper
yarn storybook smoke --macos --paper --mode stories-and-tests

Publication caveat: --paper support is currently an unpublished local change, not a command available at the pinned public fixture revision. There is no published PR to link yet.

To configure the public app without that CLI patch, change the existing use_test_app! options in apps/storybook/macos/Podfile to the following, retaining its existing block:

use_test_app!(fabric_enabled: false, bridgeless_enabled: false) do |target|

After installing repository dependencies with Yarn, generate stories and install Paper Pods:

# From apps/storybook
yarn prebuild
cd macos
RCT_NEW_ARCH_ENABLED=0 pod install
cd ..
yarn start

Build and run the generated macOS Xcode workspace/test-app target, then navigate the same Storybook catalog sequence through Default to Overview. Do not use the public default CLI prep/run commands to select the renderer: they configure Fabric. This is the equivalent raw Paper setup; the repeatable crash evidence comes from the local automated catalog traversal, not a separately validated raw-Xcode or isolated RNTester reproduction. Whether navigation directly to Overview on a fresh launch also reproduces has not been established.

Expected Behavior

Nested View/Text rows using alignItems: 'baseline' should lay out without returning an invalid custom baseline or aborting the native process. The Overview story should render, as it does in the Fabric comparison.

Actual Behavior

Native STDERR contains exactly:

Expect custom baseline function to not return NaN
libc++abi: terminating due to uncaught exception of type std::logic_error: Expect custom baseline function to not return NaN

This is a native process termination, not a JavaScript exception or just a Storybook navigation timeout. The later SecondaryPosition and ConstrainedContent story timeouts are consequences of the terminated app, not three independent crash causes.

Fabric control: After returning to the default Fabric configuration with yarn storybook prep --macos, yarn storybook smoke --macos --mode stories-and-tests renders all 168 stories. Its 30 authored cases report 7 passes and 23 explicit capability skips; those skips are not test passes. The baseline demo styling is retained for this comparison.

Reproducible Demo

Public source at an immutable revision, retaining the original failing baseline styling:

• Story fixture, including Default and Overview.
• CompoundItemLayout implementation.
• Storybook app.

The primitive renders an ordinary row View, nested content/primary/secondary region Views, and optional leading/trailing region Views. It has no state hooks or theme machinery. Overview wraps multiple rows in vertical group Views containing labels and nested Text; the single-row Default renders before the composite Overview crashes.

No minimal isolated React Native example has been validated. These links identify the real fixture from the verified app-level reproduction; extracting and independently validating a small RNTester case remains follow-up work.

Additional context

Observed: The native error matches Yoga's assertion that a custom baseline callback must not return NaN:

• calculateBaseline, Yoga Baseline.cpp, lines 17–31.

Inferred, not a confirmed root cause: The Paper text baseline path is a relevant place to investigate:

• RCTTextShadowView registers YGNodeSetBaselineFunc, lines 25–31.
• lastBaselineForSize:, lines 399–415 measures attributed text/font descenders and returns size.height + maximumDescender.
• RCTTextShadowViewBaseline, lines 455–464 converts Yoga width/height to a CoreGraphics size, calls lastBaselineForSize:, and converts the result back with RCTYogaFloatFromCoreGraphicsFloat.

The message verifies the invalid-baseline invariant failure, but we have not captured a symbolicated native backtrace, callback input dimensions, the intermediate baseline value, or proof identifying the particular callback instance responsible. It would be premature to attribute the NaN to a specific dimension, font metric, conversion, or Yoga layout ordering issue. iOS and other RN-macos versions have not been tested for this report.

The Paper driver's metadata was corrected to renderer: paper, endpoint: macos; correcting that metadata did not resolve this native crash.

A temporary Storybook-demo-only workaround is being prepared: centered alignment only on macOS Paper, preserving baseline alignment on Fabric and other platforms and leaving the production primitive unchanged. This is not an upstream/native fix; preserve the original baseline fixture when investigating.

Follow-up requested: Identify why a custom baseline becomes NaN during this Paper layout, extract a verified standalone reproduction, and add a native regression test/fix. Callback instrumentation plus a symbolicated backtrace would help distinguish invalid input dimensions from an invalid text-baseline result.

Before filing, bounded searches across open and closed RN-macos issues for baseline, alignItems baseline, custom baseline, NaN, logic_error, RCTTextShadowView, and Paper crash found no issue precisely covering this crash. The only returned matches were unrelated closed issues (#621 and #465).