docs: persist the first paper alongside the current one

.gitignore

@@ -35,8 +35,8 @@ src/main/resources/data/output
 paper/.olcli.json
 .olauth
 paper/.olauth
-paper/main.bbl
+paper/*/main.bbl
 
 # generated manuscript bundles
-paper/dist/
-paper/main.pdf
+paper/*/dist/
+paper/*/main.pdf

ARTIFACT.md

@@ -21,9 +21,9 @@ Rule: if a claim is not marked `closed` here, do not write it in the paper as al
 
 ## Claim matrix
 
-Artifact claims are numbered `AC*` and research claims, in `paper/RESEARCH-DESIGN.md`, are numbered `RC*`. The two
-sets were both numbered `C1..Cn` and said different things under the same label. A claim cited without its prefix
-is now ambiguous on its face, which is the point.
+Artifact claims are numbered `AC*`, and the manuscript's own research claims are numbered `RC*`. The two sets were
+both numbered `C1..Cn` and said different things under the same label. A claim cited without its prefix is now
+ambiguous on its face, which is the point. This file is the `AC*` set and nothing else.
 
 `paper/evidence/claims.json` is this table in machine-readable form, pinned to the revision it was generated at.
 Regenerate it with `python3 paper/scripts/claims_ledger.py`; it fails rather than emits if any evidence path in the

benchmarks/results/2026-10-08-local/environment.txt

@@ -5,7 +5,7 @@ git_branch=optionality-carriers
 repo_scala_version=3.3.6
 scalac_version=Scala compiler version 3.3.8 -- Copyright 2002-2026, LAMP/EPFL
 java_version=openjdk version "21.0.12.1" 2026-08-18 LTS OpenJDK Runtime Environment Temurin-21.0.12.1+1 (build 21.0.12.1+1-LTS) OpenJDK 64-Bit Server VM Temurin-21.0.12.1+1 (build 21.0.12.1+1-LTS, mixed mode, sharing) 
-uname=Darwin Mac.fritz.box 25.6.0 Darwin Kernel Version 25.6.0: Fri Jul 31 19:18:48 PDT 2026; root:xnu-12377.161.14~5/RELEASE_ARM64_T6020 arm64
+uname=Darwin 25.6.0 arm64
 cpu_model=Apple M2 Max
 memory_bytes=68719476736
 compile_warmups=1

benchmarks/run-benchmarks.sh

@@ -55,7 +55,8 @@ collect_environment() {
     printf 'repo_scala_version=%s\n' "$(awk -F'\"' '$1 ~ /^val scala3[[:space:]]*=/ { print $2; exit }' "$ROOT_DIR/build.sbt")"
     printf 'scalac_version=%s\n' "$(scalac -version 2>&1)"
     printf 'java_version=%s\n' "$(java -version 2>&1 | tr '\n' ' ' | sed 's/  */ /g')"
-    printf 'uname=%s\n' "$(uname -a)"
+    # -srm, not -a: the kernel build string carries the machine's hostname, and a saved run is published.
+    printf 'uname=%s\n' "$(uname -srm)"
     printf 'cpu_model=%s\n' "$(detect_cpu_model)"
     printf 'memory_bytes=%s\n' "$(detect_memory_bytes)"
     printf 'compile_warmups=%s\n' "$WARMUPS"

joss/paper.bib

@@ -92,12 +92,22 @@ @article{deequ
 
 @misc{great_expectations,
   author = {{Great Expectations}},
-  title  = {Great Expectations: Expectations reference},
+  title  = {Great Expectations: Expectation Gallery},
   year   = {2025},
-  url    = {https://docs.greatexpectations.io/docs/reference/expectations/},
+  url    = {https://greatexpectations.io/expectations/},
   note   = {Accessed 2026-10-08}
 }
 
+@misc{mirji_shift_left_2026,
+  author        = {Mirji, Vittal},
+  title         = {Shift schema drift left: policy-aware compile-time contracts for typed JVM and Spark pipelines},
+  year          = {2026},
+  eprint        = {2604.16986},
+  archivePrefix = {arXiv},
+  primaryClass  = {cs.PL},
+  url           = {https://arxiv.org/abs/2604.16986}
+}
+
 @misc{spark_53717,
   author = {{Apache Spark}},
   title  = {SPARK-53717: Fix misleading documentation for MapType.valueContainsNull},

joss/paper.md

@@ -132,8 +132,9 @@ support.
 
 The software is used in the author's own research and the repository is the reproducible
 material for it: the 2668-cell matrix, the harnesses that produced it, and the saved runs are
-checked in and regenerable by documented commands. A companion paper developing the carrier
-analysis in full draws every number from those runs.
+checked in and regenerable by documented commands. The mechanism this software implements, and
+the policy family behind it, are described in a preprint [@mirji_shift_left_2026]. A second
+paper developing the carrier analysis in full draws every number from those runs.
 
 The evidence a reader can check without taking the author's word for it is organised as a claim
 ledger. Every claim the project makes is listed with the test or saved run that backs it in

paper/README.md

@@ -1,45 +1,64 @@
 # Paper scaffold
 
-This directory is the local Overleaf-facing manuscript scaffold for the `compile-time-data-contracts` paper.
+This directory is the manuscript scaffold for `compile-time-data-contracts`. It holds two papers,
+each in its own directory, and the tooling and evidence they share.
 
-It is bound to the Overleaf project `paper` through [paper/.olcli.json](.olcli.json).
+| Directory | Paper |
+|---|---|
+| [three-carriers/](three-carriers/README.md) | "Three carriers, one bit: optionality in schema-equality checking, measured in Spark". The current one, not yet submitted. |
+| [arxiv-2604.16986/](arxiv-2604.16986/README.md) | "Shift schema drift left: policy-aware compile-time contracts for typed JVM and Spark pipelines". Announced on arXiv, awaiting a v2 replacement. |
 
-## Local sources of truth
+They are separate papers rather than two drafts of one. The section filenames are the same in both,
+which is why they cannot share a `sections/` directory.
 
-The detailed paper prep package has been moved to local-only internal docs under `docs/internals/prep-docs/`.
-Those files are intentionally ignored by git at this stage.
+## What stays at this level
 
-## `olcli` workflow
+- `scripts/` builds the arXiv bundles, fetches the `.bbl` from Overleaf, and generates the claim
+  ledger and the Spark predicate hashes
+- `evidence/` holds the generated evidence files the current paper cites by repo-relative path
+- `corpus/` holds the corpus manifest and fetch script, but not the schema payloads
+- `.olcli.json` is the local Overleaf binding, and is ignored by git
+
+Both scripts that need to know which paper they are working on take it as a `PAPER_DIR`
+environment variable, a directory name under `paper/`, defaulting to `three-carriers`.
 
-The CLI is sync-oriented. Project creation happens in the browser first, then local work is pulled, edited, and pushed.
+## `olcli` workflow
 
-Useful commands:
+The CLI is sync-oriented. Project creation happens in the browser first, then local work is pulled,
+edited, and pushed.
 
 ```bash
 npx -y @aloth/olcli sync paper
 npx -y @aloth/olcli push paper --all
 npx -y @aloth/olcli pdf paper -o /tmp/paper.pdf
-npx -y @aloth/olcli output bbl --project paper -o paper/output.bbl
 node paper/scripts/fetch-overleaf-bbl.mjs
 paper/scripts/build-arxiv-bundle.sh
+PAPER_DIR=arxiv-2604.16986 paper/scripts/build-arxiv-bundle.sh
+```
+
+Notes on this setup that still hold:
+
+- the local scaffold is bound to the Overleaf project `paper`, and `olcli` upload/push works
+- remote compilation and PDF download work via `olcli pdf`; remote logs show `pdfTeX` on TeX Live 2025
+- `olcli output log` confirms the manuscript reads `./output.bbl` during compile
+- `olcli output bbl` is flaky in this setup, so `paper/scripts/fetch-overleaf-bbl.mjs` is the
+  reliable local fallback
+
+## Building locally
+
+`tectonic` is the local source of truth for refreshed submission artifacts between remote syncs. It
+runs bibtex itself.
+
+```bash
+cd paper/three-carriers && tectonic -X compile main.tex
+```
+
+`tectonic` does not write the `.bbl` to disk unless asked, and the arXiv bundler refuses a `.bbl`
+older than the sources that can change it, so refresh it with the sources:
+
+```bash
+cd paper/three-carriers && tectonic -X compile main.tex --keep-intermediates
 ```
 
-## Current status
-
-- `main.tex` uses `acmart` with the `sigplan` option and `nonacm` for local drafting
-- `00README.json` records the intended top-level source and current submission-target compiler metadata
-- Sections 1-8 exist as separate files under `paper/sections/`
-- The first prose pass covers Sections 1-8, with the strongest polish so far in Sections 1-4 and 7-8
-- Mermaid source and rendered PNG/PDF assets exist under `paper/figures/`
-- The local scaffold is bound to the Overleaf project `paper`, and `olcli` upload/push currently works again in this environment
-- Remote Overleaf compilation and PDF download work via `olcli pdf`; current remote logs show `pdfTeX` on TeX Live 2025
-- Local `tectonic` output remains the source of truth for refreshed submission artifacts between remote syncs
-- `olcli output log` works and confirms the manuscript reads `./output.bbl` during compile
-- `olcli output bbl` is still flaky in this setup, so `paper/scripts/fetch-overleaf-bbl.mjs` is the reliable local fallback
-- detailed submission and release prep notes are kept in local-only internal docs
-
-## Next edits
-
-1. tighten Sections 4-8 in TeX against the current compiled PDF
-2. trim remaining overfull/underfull warnings where they harm layout
-3. fetch `output.bbl` later through browser or a fixed CLI path when needed for arXiv packaging
+That also leaves `main.aux`, `main.log`, `main.blg` and `main.out` behind; they are build output and
+are not tracked.

paper/arxiv-2604.16986/README.md

@@ -0,0 +1,95 @@
+# arXiv:2604.16986 [cs.PL]
+
+"Shift schema drift left: policy-aware compile-time contracts for typed JVM and Spark pipelines",
+Vittal Mirji, announced 18 April 2026. Seven pages, two figures, one table.
+
+These sources started as the tree at commit `6c5399f` (2026-04-23), byte for byte, which is the state
+the announced v1 was built from. They now carry the v2 corrections described below. The figures are
+kept here too because `fig02-policy-family` has since changed in the current paper.
+
+The body still describes the artifact as it was evaluated, at commit `6c5399f`. Where a statement was
+true then and is not true of the current artifact, the divergence is flagged where it occurs. The
+alternative, rewriting the description to match `0.2.0`, would make the paper describe code its own
+evaluation never ran.
+
+## Why it is a separate directory
+
+This is a different paper from the one in [../three-carriers/](../three-carriers/README.md), not an
+earlier draft of it. This one describes the mechanism and the policy family. The other is a
+measurement study of how the three carriers of optionality behave in schema-equality checking. The
+section filenames collide between the two, which is why each needs its own `sections/`.
+
+## What 0.2.0 made inaccurate
+
+One claim in the announced abstract no longer holds for the artifact, not two. The abstract said the
+runtime comparator "adds a nested-collection-optionality check Spark's built-in comparators omit and
+implements structural subset semantics for backward- and forward-compatible field sets".
+
+- The optionality check is wrong as attributed. `SparkCore.scala:63` builds the pin's rules as
+  `ComparisonRules.of(policy).ignoringOptionality`, which drops all three carriers. The capability
+  was not deleted, it moved: `ComparisonRules.scala:151-161` gives every policy except `Unchecked` a
+  non-`Ignored` optionality axis at compile time, and `ShapeDiff.scala:90-94` is where the sequence
+  and map carriers are compared. The reason for the move is in the 0.2.0 entry of `CHANGELOG.md`:
+  Spark's readers return the permissive value for each carrier on every format that does not record
+  the claim, so the pin would compare a reader default rather than a producer's claim.
+- The subset semantics are intact. `SparkCore.scala:91-104` still implements `missingTolerated` and
+  `extraTolerated` per tolerance, so `Backward` still allows producer extras and optional or
+  defaulted contract omissions, and `Forward` still allows the contract to hold more fields. What
+  changed for these two policies is the optionality direction at compile time, which is a different
+  axis from the field-set subsetting the abstract describes.
+
+Correcting this needs a v2 replacement on this identifier, not a new submission. arXiv's own rule is
+to replace rather than resubmit for a correction.
+
+## v2 replacement
+
+What changed in the sources, all of it surgical:
+
+- `main.tex`: the abstract sentence above is split, the optionality clause is restated as a
+  compile-time-only check with the reader-default reason, and it points at the companion paper. A
+  short "Note on version 2" paragraph follows `\maketitle`. Two macros are added: `\companionpreprint`
+  and `\carriermoved`.
+- `01-introduction.tex`: the contribution bullet says the subset semantics are unchanged and the
+  optionality check is compile-time only. The §1 assertion carries `\carriermoved`.
+- `02-background-and-model.tex`: the claim that field-level optionality is ignored under all
+  non-`Full` policies is marked as inverted by 0.2.0, with the `Full` to `Unchecked` rename noted.
+- `03-framework-design.tex`: "the runtime layer" becomes "the comparison" where the sentence is about
+  the internal representation, and the parity judgement gets a two-sentence correction.
+- `04-artifact-and-evaluation.tex`: the artifact scope names commit `6c5399f`, which scopes the whole
+  section to the evaluated code.
+
+Before submitting, in order:
+
+1. Fill in `\companionpreprint` in `main.tex`. The placeholder is `arXiv:XXXX.XXXXX`, which is not a
+   well-formed identifier, so it will not pass unnoticed. It appears twice in the PDF.
+2. Replace the abstract field on arXiv with the revised abstract. The abstract field and the PDF
+   abstract have to match, so take it from `main.tex` rather than retyping it:
+   `PAPER_DIR=arxiv-2604.16986 python3 paper/scripts/abstract-field.py`. It expands
+   `\companionpreprint` to whatever `main.tex` currently defines, placeholder included, which is why
+   step 1 comes first.
+3. Set the comments field to the text below.
+4. Update the page count. v1 is seven pages; the corrected paper is eight, because the references
+   move onto a new page once the note is added.
+
+Comments field for the replacement:
+
+```
+v2: corrects the attribution of the nested-collection-optionality check, which release 0.2.0 of the
+artifact moved from the runtime sink pin to the compile-time comparison, because Spark's readers
+return the permissive value for each carrier on every format that does not record the claim. The
+design, evaluation and benchmarks are unchanged and describe the repository at commit 6c5399f. The
+measurement study of the carriers is a companion paper, arXiv:XXXX.XXXXX. 8 pages, 2 figures, 1
+table. Code at https://github.com/vim89/compile-time-data-contracts
+```
+
+## Building
+
+```bash
+cd paper/arxiv-2604.16986 && tectonic -X compile main.tex
+PAPER_DIR=arxiv-2604.16986 paper/scripts/build-arxiv-bundle.sh
+```
+
+The bundle script defaults to the other paper, so this one needs `PAPER_DIR`. Note that the bundle
+holds seven sections: `sections/05-industrial-context-and-lessons.tex` is present in this directory
+but is not `\input` by `main.tex`, and the script copies only the sections `main.tex` actually
+names.

paper/arxiv-2604.16986/figures/README.md

@@ -0,0 +1,18 @@
+# Figure sources
+
+This directory holds manuscript figure sources.
+
+Current state:
+
+- Mermaid source is stored as `.mmd`
+- manuscript-ready PNG renders already exist for the first two figures
+
+Render locally with Mermaid CLI:
+
+```bash
+npx -y @mermaid-js/mermaid-cli -i paper/arxiv-2604.16986/figures/fig01-architecture.mmd -o paper/arxiv-2604.16986/figures/fig01-architecture.png
+npx -y @mermaid-js/mermaid-cli -i paper/arxiv-2604.16986/figures/fig02-policy-family.mmd -o paper/arxiv-2604.16986/figures/fig02-policy-family.png
+```
+
+The `.mmd` sources are the only place the figure wording is written; the renders are generated from
+them by the commands above.

paper/arxiv-2604.16986/figures/fig02-policy-family.mmd

@@ -0,0 +1,35 @@
+flowchart TD
+  ROOT["SchemaPolicy family"]
+
+  subgraph EXACT["Exact-style policies"]
+    EXACT_DEFAULT["Exact / ExactUnorderedCI<br/>unordered by name<br/>case-insensitive<br/>no extras, no missing<br/>nested optionality checked"]
+    EXACT_ORDERED["ExactOrdered<br/>ordered by name<br/>case-sensitive<br/>no extras, no missing<br/>nested optionality checked"]
+    EXACT_ORDERED_CI["ExactOrderedCI<br/>ordered by name<br/>case-insensitive<br/>no extras, no missing<br/>nested optionality checked"]
+    EXACT_POS["ExactByPosition<br/>by position<br/>names ignored<br/>no extras, no missing<br/>nested optionality checked"]
+  end
+
+  subgraph SUBSET["Subset-style policies"]
+    BACKWARD["Backward<br/>contract subset of producer<br/>producer extras allowed<br/>missing contract fields allowed only when optional or defaulted<br/>nested optionality checked"]
+    FORWARD["Forward<br/>producer subset of contract<br/>producer extras rejected<br/>contract may contain more fields<br/>nested optionality checked"]
+  end
+
+  subgraph ESCAPE["Escape hatch"]
+    FULL["Full<br/>accept all<br/>used only when enforcement is intentionally disabled"]
+  end
+
+  ROOT --> EXACT_DEFAULT
+  ROOT --> EXACT_ORDERED
+  ROOT --> EXACT_ORDERED_CI
+  ROOT --> EXACT_POS
+  ROOT --> BACKWARD
+  ROOT --> FORWARD
+  ROOT --> FULL
+
+  style ROOT fill:#e1f5ff,stroke:#0066cc,color:#000
+  style EXACT_DEFAULT fill:#f0e1ff,stroke:#8800cc,color:#000
+  style EXACT_ORDERED fill:#f0e1ff,stroke:#8800cc,color:#000
+  style EXACT_ORDERED_CI fill:#f0e1ff,stroke:#8800cc,color:#000
+  style EXACT_POS fill:#f0e1ff,stroke:#8800cc,color:#000
+  style BACKWARD fill:#fff4e1,stroke:#cc8800,color:#000
+  style FORWARD fill:#fff4e1,stroke:#cc8800,color:#000
+  style FULL fill:#ffe1e1,stroke:#cc0000,color:#000