diff --git a/.gitignore b/.gitignore
index 6c82ed4..cfba0c4 100644
--- a/.gitignore
+++ b/.gitignore
@@ -35,8 +35,8 @@ src/main/resources/data/output
paper/.olcli.json
.olauth
paper/.olauth
-paper/main.bbl
+paper/*/main.bbl
# generated manuscript bundles
-paper/dist/
-paper/main.pdf
+paper/*/dist/
+paper/*/main.pdf
diff --git a/ARTIFACT.md b/ARTIFACT.md
index 800ed71..a5ce30b 100644
--- a/ARTIFACT.md
+++ b/ARTIFACT.md
@@ -21,9 +21,9 @@ Rule: if a claim is not marked `closed` here, do not write it in the paper as al
## Claim matrix
-Artifact claims are numbered `AC*` and research claims, in `paper/RESEARCH-DESIGN.md`, are numbered `RC*`. The two
-sets were both numbered `C1..Cn` and said different things under the same label. A claim cited without its prefix
-is now ambiguous on its face, which is the point.
+Artifact claims are numbered `AC*`, and the manuscript's own research claims are numbered `RC*`. The two sets were
+both numbered `C1..Cn` and said different things under the same label. A claim cited without its prefix is now
+ambiguous on its face, which is the point. This file is the `AC*` set and nothing else.
`paper/evidence/claims.json` is this table in machine-readable form, pinned to the revision it was generated at.
Regenerate it with `python3 paper/scripts/claims_ledger.py`; it fails rather than emits if any evidence path in the
diff --git a/benchmarks/results/2026-10-08-local/environment.txt b/benchmarks/results/2026-10-08-local/environment.txt
index 6596d0e..1f49c67 100644
--- a/benchmarks/results/2026-10-08-local/environment.txt
+++ b/benchmarks/results/2026-10-08-local/environment.txt
@@ -5,7 +5,7 @@ git_branch=optionality-carriers
repo_scala_version=3.3.6
scalac_version=Scala compiler version 3.3.8 -- Copyright 2002-2026, LAMP/EPFL
java_version=openjdk version "21.0.12.1" 2026-08-18 LTS OpenJDK Runtime Environment Temurin-21.0.12.1+1 (build 21.0.12.1+1-LTS) OpenJDK 64-Bit Server VM Temurin-21.0.12.1+1 (build 21.0.12.1+1-LTS, mixed mode, sharing)
-uname=Darwin Mac.fritz.box 25.6.0 Darwin Kernel Version 25.6.0: Fri Jul 31 19:18:48 PDT 2026; root:xnu-12377.161.14~5/RELEASE_ARM64_T6020 arm64
+uname=Darwin 25.6.0 arm64
cpu_model=Apple M2 Max
memory_bytes=68719476736
compile_warmups=1
diff --git a/benchmarks/run-benchmarks.sh b/benchmarks/run-benchmarks.sh
index bbe2bec..f2b683f 100755
--- a/benchmarks/run-benchmarks.sh
+++ b/benchmarks/run-benchmarks.sh
@@ -55,7 +55,8 @@ collect_environment() {
printf 'repo_scala_version=%s\n' "$(awk -F'\"' '$1 ~ /^val scala3[[:space:]]*=/ { print $2; exit }' "$ROOT_DIR/build.sbt")"
printf 'scalac_version=%s\n' "$(scalac -version 2>&1)"
printf 'java_version=%s\n' "$(java -version 2>&1 | tr '\n' ' ' | sed 's/ */ /g')"
- printf 'uname=%s\n' "$(uname -a)"
+ # -srm, not -a: the kernel build string carries the machine's hostname, and a saved run is published.
+ printf 'uname=%s\n' "$(uname -srm)"
printf 'cpu_model=%s\n' "$(detect_cpu_model)"
printf 'memory_bytes=%s\n' "$(detect_memory_bytes)"
printf 'compile_warmups=%s\n' "$WARMUPS"
diff --git a/joss/paper.bib b/joss/paper.bib
index 7d2b96c..44741b7 100644
--- a/joss/paper.bib
+++ b/joss/paper.bib
@@ -92,12 +92,22 @@ @article{deequ
@misc{great_expectations,
author = {{Great Expectations}},
- title = {Great Expectations: Expectations reference},
+ title = {Great Expectations: Expectation Gallery},
year = {2025},
- url = {https://docs.greatexpectations.io/docs/reference/expectations/},
+ url = {https://greatexpectations.io/expectations/},
note = {Accessed 2026-10-08}
}
+@misc{mirji_shift_left_2026,
+ author = {Mirji, Vittal},
+ title = {Shift schema drift left: policy-aware compile-time contracts for typed JVM and Spark pipelines},
+ year = {2026},
+ eprint = {2604.16986},
+ archivePrefix = {arXiv},
+ primaryClass = {cs.PL},
+ url = {https://arxiv.org/abs/2604.16986}
+}
+
@misc{spark_53717,
author = {{Apache Spark}},
title = {SPARK-53717: Fix misleading documentation for MapType.valueContainsNull},
diff --git a/joss/paper.md b/joss/paper.md
index 9e15a37..d5d2028 100644
--- a/joss/paper.md
+++ b/joss/paper.md
@@ -132,8 +132,9 @@ support.
The software is used in the author's own research and the repository is the reproducible
material for it: the 2668-cell matrix, the harnesses that produced it, and the saved runs are
-checked in and regenerable by documented commands. A companion paper developing the carrier
-analysis in full draws every number from those runs.
+checked in and regenerable by documented commands. The mechanism this software implements, and
+the policy family behind it, are described in a preprint [@mirji_shift_left_2026]. A second
+paper developing the carrier analysis in full draws every number from those runs.
The evidence a reader can check without taking the author's word for it is organised as a claim
ledger. Every claim the project makes is listed with the test or saved run that backs it in
diff --git a/paper/README.md b/paper/README.md
index dda7a1d..60a3435 100644
--- a/paper/README.md
+++ b/paper/README.md
@@ -1,45 +1,64 @@
# Paper scaffold
-This directory is the local Overleaf-facing manuscript scaffold for the `compile-time-data-contracts` paper.
+This directory is the manuscript scaffold for `compile-time-data-contracts`. It holds two papers,
+each in its own directory, and the tooling and evidence they share.
-It is bound to the Overleaf project `paper` through [paper/.olcli.json](.olcli.json).
+| Directory | Paper |
+|---|---|
+| [three-carriers/](three-carriers/README.md) | "Three carriers, one bit: optionality in schema-equality checking, measured in Spark". The current one, not yet submitted. |
+| [arxiv-2604.16986/](arxiv-2604.16986/README.md) | "Shift schema drift left: policy-aware compile-time contracts for typed JVM and Spark pipelines". Announced on arXiv, awaiting a v2 replacement. |
-## Local sources of truth
+They are separate papers rather than two drafts of one. The section filenames are the same in both,
+which is why they cannot share a `sections/` directory.
-The detailed paper prep package has been moved to local-only internal docs under `docs/internals/prep-docs/`.
-Those files are intentionally ignored by git at this stage.
+## What stays at this level
-## `olcli` workflow
+- `scripts/` builds the arXiv bundles, fetches the `.bbl` from Overleaf, and generates the claim
+ ledger and the Spark predicate hashes
+- `evidence/` holds the generated evidence files the current paper cites by repo-relative path
+- `corpus/` holds the corpus manifest and fetch script, but not the schema payloads
+- `.olcli.json` is the local Overleaf binding, and is ignored by git
+
+Both scripts that need to know which paper they are working on take it as a `PAPER_DIR`
+environment variable, a directory name under `paper/`, defaulting to `three-carriers`.
-The CLI is sync-oriented. Project creation happens in the browser first, then local work is pulled, edited, and pushed.
+## `olcli` workflow
-Useful commands:
+The CLI is sync-oriented. Project creation happens in the browser first, then local work is pulled,
+edited, and pushed.
```bash
npx -y @aloth/olcli sync paper
npx -y @aloth/olcli push paper --all
npx -y @aloth/olcli pdf paper -o /tmp/paper.pdf
-npx -y @aloth/olcli output bbl --project paper -o paper/output.bbl
node paper/scripts/fetch-overleaf-bbl.mjs
paper/scripts/build-arxiv-bundle.sh
+PAPER_DIR=arxiv-2604.16986 paper/scripts/build-arxiv-bundle.sh
+```
+
+Notes on this setup that still hold:
+
+- the local scaffold is bound to the Overleaf project `paper`, and `olcli` upload/push works
+- remote compilation and PDF download work via `olcli pdf`; remote logs show `pdfTeX` on TeX Live 2025
+- `olcli output log` confirms the manuscript reads `./output.bbl` during compile
+- `olcli output bbl` is flaky in this setup, so `paper/scripts/fetch-overleaf-bbl.mjs` is the
+ reliable local fallback
+
+## Building locally
+
+`tectonic` is the local source of truth for refreshed submission artifacts between remote syncs. It
+runs bibtex itself.
+
+```bash
+cd paper/three-carriers && tectonic -X compile main.tex
+```
+
+`tectonic` does not write the `.bbl` to disk unless asked, and the arXiv bundler refuses a `.bbl`
+older than the sources that can change it, so refresh it with the sources:
+
+```bash
+cd paper/three-carriers && tectonic -X compile main.tex --keep-intermediates
```
-## Current status
-
-- `main.tex` uses `acmart` with the `sigplan` option and `nonacm` for local drafting
-- `00README.json` records the intended top-level source and current submission-target compiler metadata
-- Sections 1-8 exist as separate files under `paper/sections/`
-- The first prose pass covers Sections 1-8, with the strongest polish so far in Sections 1-4 and 7-8
-- Mermaid source and rendered PNG/PDF assets exist under `paper/figures/`
-- The local scaffold is bound to the Overleaf project `paper`, and `olcli` upload/push currently works again in this environment
-- Remote Overleaf compilation and PDF download work via `olcli pdf`; current remote logs show `pdfTeX` on TeX Live 2025
-- Local `tectonic` output remains the source of truth for refreshed submission artifacts between remote syncs
-- `olcli output log` works and confirms the manuscript reads `./output.bbl` during compile
-- `olcli output bbl` is still flaky in this setup, so `paper/scripts/fetch-overleaf-bbl.mjs` is the reliable local fallback
-- detailed submission and release prep notes are kept in local-only internal docs
-
-## Next edits
-
-1. tighten Sections 4-8 in TeX against the current compiled PDF
-2. trim remaining overfull/underfull warnings where they harm layout
-3. fetch `output.bbl` later through browser or a fixed CLI path when needed for arXiv packaging
+That also leaves `main.aux`, `main.log`, `main.blg` and `main.out` behind; they are build output and
+are not tracked.
diff --git a/paper/00README.json b/paper/arxiv-2604.16986/00README.json
similarity index 100%
rename from paper/00README.json
rename to paper/arxiv-2604.16986/00README.json
diff --git a/paper/arxiv-2604.16986/README.md b/paper/arxiv-2604.16986/README.md
new file mode 100644
index 0000000..9514b9f
--- /dev/null
+++ b/paper/arxiv-2604.16986/README.md
@@ -0,0 +1,95 @@
+# arXiv:2604.16986 [cs.PL]
+
+"Shift schema drift left: policy-aware compile-time contracts for typed JVM and Spark pipelines",
+Vittal Mirji, announced 18 April 2026. Seven pages, two figures, one table.
+
+These sources started as the tree at commit `6c5399f` (2026-04-23), byte for byte, which is the state
+the announced v1 was built from. They now carry the v2 corrections described below. The figures are
+kept here too because `fig02-policy-family` has since changed in the current paper.
+
+The body still describes the artifact as it was evaluated, at commit `6c5399f`. Where a statement was
+true then and is not true of the current artifact, the divergence is flagged where it occurs. The
+alternative, rewriting the description to match `0.2.0`, would make the paper describe code its own
+evaluation never ran.
+
+## Why it is a separate directory
+
+This is a different paper from the one in [../three-carriers/](../three-carriers/README.md), not an
+earlier draft of it. This one describes the mechanism and the policy family. The other is a
+measurement study of how the three carriers of optionality behave in schema-equality checking. The
+section filenames collide between the two, which is why each needs its own `sections/`.
+
+## What 0.2.0 made inaccurate
+
+One claim in the announced abstract no longer holds for the artifact, not two. The abstract said the
+runtime comparator "adds a nested-collection-optionality check Spark's built-in comparators omit and
+implements structural subset semantics for backward- and forward-compatible field sets".
+
+- The optionality check is wrong as attributed. `SparkCore.scala:63` builds the pin's rules as
+ `ComparisonRules.of(policy).ignoringOptionality`, which drops all three carriers. The capability
+ was not deleted, it moved: `ComparisonRules.scala:151-161` gives every policy except `Unchecked` a
+ non-`Ignored` optionality axis at compile time, and `ShapeDiff.scala:90-94` is where the sequence
+ and map carriers are compared. The reason for the move is in the 0.2.0 entry of `CHANGELOG.md`:
+ Spark's readers return the permissive value for each carrier on every format that does not record
+ the claim, so the pin would compare a reader default rather than a producer's claim.
+- The subset semantics are intact. `SparkCore.scala:91-104` still implements `missingTolerated` and
+ `extraTolerated` per tolerance, so `Backward` still allows producer extras and optional or
+ defaulted contract omissions, and `Forward` still allows the contract to hold more fields. What
+ changed for these two policies is the optionality direction at compile time, which is a different
+ axis from the field-set subsetting the abstract describes.
+
+Correcting this needs a v2 replacement on this identifier, not a new submission. arXiv's own rule is
+to replace rather than resubmit for a correction.
+
+## v2 replacement
+
+What changed in the sources, all of it surgical:
+
+- `main.tex`: the abstract sentence above is split, the optionality clause is restated as a
+ compile-time-only check with the reader-default reason, and it points at the companion paper. A
+ short "Note on version 2" paragraph follows `\maketitle`. Two macros are added: `\companionpreprint`
+ and `\carriermoved`.
+- `01-introduction.tex`: the contribution bullet says the subset semantics are unchanged and the
+ optionality check is compile-time only. The ยง1 assertion carries `\carriermoved`.
+- `02-background-and-model.tex`: the claim that field-level optionality is ignored under all
+ non-`Full` policies is marked as inverted by 0.2.0, with the `Full` to `Unchecked` rename noted.
+- `03-framework-design.tex`: "the runtime layer" becomes "the comparison" where the sentence is about
+ the internal representation, and the parity judgement gets a two-sentence correction.
+- `04-artifact-and-evaluation.tex`: the artifact scope names commit `6c5399f`, which scopes the whole
+ section to the evaluated code.
+
+Before submitting, in order:
+
+1. Fill in `\companionpreprint` in `main.tex`. The placeholder is `arXiv:XXXX.XXXXX`, which is not a
+ well-formed identifier, so it will not pass unnoticed. It appears twice in the PDF.
+2. Replace the abstract field on arXiv with the revised abstract. The abstract field and the PDF
+ abstract have to match, so take it from `main.tex` rather than retyping it:
+ `PAPER_DIR=arxiv-2604.16986 python3 paper/scripts/abstract-field.py`. It expands
+ `\companionpreprint` to whatever `main.tex` currently defines, placeholder included, which is why
+ step 1 comes first.
+3. Set the comments field to the text below.
+4. Update the page count. v1 is seven pages; the corrected paper is eight, because the references
+ move onto a new page once the note is added.
+
+Comments field for the replacement:
+
+```
+v2: corrects the attribution of the nested-collection-optionality check, which release 0.2.0 of the
+artifact moved from the runtime sink pin to the compile-time comparison, because Spark's readers
+return the permissive value for each carrier on every format that does not record the claim. The
+design, evaluation and benchmarks are unchanged and describe the repository at commit 6c5399f. The
+measurement study of the carriers is a companion paper, arXiv:XXXX.XXXXX. 8 pages, 2 figures, 1
+table. Code at https://github.com/vim89/compile-time-data-contracts
+```
+
+## Building
+
+```bash
+cd paper/arxiv-2604.16986 && tectonic -X compile main.tex
+PAPER_DIR=arxiv-2604.16986 paper/scripts/build-arxiv-bundle.sh
+```
+
+The bundle script defaults to the other paper, so this one needs `PAPER_DIR`. Note that the bundle
+holds seven sections: `sections/05-industrial-context-and-lessons.tex` is present in this directory
+but is not `\input` by `main.tex`, and the script copies only the sections `main.tex` actually
+names.
diff --git a/paper/arxiv-2604.16986/figures/README.md b/paper/arxiv-2604.16986/figures/README.md
new file mode 100644
index 0000000..1dfd2bd
--- /dev/null
+++ b/paper/arxiv-2604.16986/figures/README.md
@@ -0,0 +1,18 @@
+# Figure sources
+
+This directory holds manuscript figure sources.
+
+Current state:
+
+- Mermaid source is stored as `.mmd`
+- manuscript-ready PNG renders already exist for the first two figures
+
+Render locally with Mermaid CLI:
+
+```bash
+npx -y @mermaid-js/mermaid-cli -i paper/arxiv-2604.16986/figures/fig01-architecture.mmd -o paper/arxiv-2604.16986/figures/fig01-architecture.png
+npx -y @mermaid-js/mermaid-cli -i paper/arxiv-2604.16986/figures/fig02-policy-family.mmd -o paper/arxiv-2604.16986/figures/fig02-policy-family.png
+```
+
+The `.mmd` sources are the only place the figure wording is written; the renders are generated from
+them by the commands above.
diff --git a/paper/figures/fig01-architecture.mmd b/paper/arxiv-2604.16986/figures/fig01-architecture.mmd
similarity index 100%
rename from paper/figures/fig01-architecture.mmd
rename to paper/arxiv-2604.16986/figures/fig01-architecture.mmd
diff --git a/paper/figures/fig01-architecture.pdf b/paper/arxiv-2604.16986/figures/fig01-architecture.pdf
similarity index 100%
rename from paper/figures/fig01-architecture.pdf
rename to paper/arxiv-2604.16986/figures/fig01-architecture.pdf
diff --git a/paper/figures/fig01-architecture.png b/paper/arxiv-2604.16986/figures/fig01-architecture.png
similarity index 100%
rename from paper/figures/fig01-architecture.png
rename to paper/arxiv-2604.16986/figures/fig01-architecture.png
diff --git a/paper/arxiv-2604.16986/figures/fig02-policy-family.mmd b/paper/arxiv-2604.16986/figures/fig02-policy-family.mmd
new file mode 100644
index 0000000..65466c1
--- /dev/null
+++ b/paper/arxiv-2604.16986/figures/fig02-policy-family.mmd
@@ -0,0 +1,35 @@
+flowchart TD
+ ROOT["SchemaPolicy family"]
+
+ subgraph EXACT["Exact-style policies"]
+ EXACT_DEFAULT["Exact / ExactUnorderedCI
unordered by name
case-insensitive
no extras, no missing
nested optionality checked"]
+ EXACT_ORDERED["ExactOrdered
ordered by name
case-sensitive
no extras, no missing
nested optionality checked"]
+ EXACT_ORDERED_CI["ExactOrderedCI
ordered by name
case-insensitive
no extras, no missing
nested optionality checked"]
+ EXACT_POS["ExactByPosition
by position
names ignored
no extras, no missing
nested optionality checked"]
+ end
+
+ subgraph SUBSET["Subset-style policies"]
+ BACKWARD["Backward
contract subset of producer
producer extras allowed
missing contract fields allowed only when optional or defaulted
nested optionality checked"]
+ FORWARD["Forward
producer subset of contract
producer extras rejected
contract may contain more fields
nested optionality checked"]
+ end
+
+ subgraph ESCAPE["Escape hatch"]
+ FULL["Full
accept all
used only when enforcement is intentionally disabled"]
+ end
+
+ ROOT --> EXACT_DEFAULT
+ ROOT --> EXACT_ORDERED
+ ROOT --> EXACT_ORDERED_CI
+ ROOT --> EXACT_POS
+ ROOT --> BACKWARD
+ ROOT --> FORWARD
+ ROOT --> FULL
+
+ style ROOT fill:#e1f5ff,stroke:#0066cc,color:#000
+ style EXACT_DEFAULT fill:#f0e1ff,stroke:#8800cc,color:#000
+ style EXACT_ORDERED fill:#f0e1ff,stroke:#8800cc,color:#000
+ style EXACT_ORDERED_CI fill:#f0e1ff,stroke:#8800cc,color:#000
+ style EXACT_POS fill:#f0e1ff,stroke:#8800cc,color:#000
+ style BACKWARD fill:#fff4e1,stroke:#cc8800,color:#000
+ style FORWARD fill:#fff4e1,stroke:#cc8800,color:#000
+ style FULL fill:#ffe1e1,stroke:#cc0000,color:#000
diff --git a/paper/arxiv-2604.16986/figures/fig02-policy-family.pdf b/paper/arxiv-2604.16986/figures/fig02-policy-family.pdf
new file mode 100644
index 0000000..50a5708
Binary files /dev/null and b/paper/arxiv-2604.16986/figures/fig02-policy-family.pdf differ
diff --git a/paper/arxiv-2604.16986/figures/fig02-policy-family.png b/paper/arxiv-2604.16986/figures/fig02-policy-family.png
new file mode 100644
index 0000000..aa2342e
Binary files /dev/null and b/paper/arxiv-2604.16986/figures/fig02-policy-family.png differ
diff --git a/paper/arxiv-2604.16986/main.tex b/paper/arxiv-2604.16986/main.tex
new file mode 100644
index 0000000..4a6ccc0
--- /dev/null
+++ b/paper/arxiv-2604.16986/main.tex
@@ -0,0 +1,68 @@
+\documentclass[sigplan,screen,nonacm]{acmart}
+
+\usepackage{booktabs}
+\usepackage{graphicx}
+\usepackage{microtype}
+\usepackage{enumitem}
+\usepackage{float}
+\usepackage{xurl}
+\graphicspath{{figures/}}
+
+\settopmatter{printfolios=true}
+\renewcommand{\shortauthors}{Mirji}
+\setlength{\emergencystretch}{1.5em}
+\Urlmuskip=0mu plus 1mu
+
+% The companion paper's arXiv identifier, used twice below and nowhere else. The placeholder is
+% deliberately not a well-formed identifier: v2 must not be submitted until it is filled in.
+\newcommand{\companionpreprint}{arXiv:XXXX.XXXXX}
+
+% Marks a statement that was true of the evaluated commit and is not true of the current artifact.
+% The body otherwise describes the commit it evaluated, so the divergence is flagged where it occurs
+% rather than by rewriting the description around it.
+\newcommand{\carriermoved}{\footnote{Release \texttt{0.2.0} removed this comparison from the runtime pin and compares the carriers at compile time instead. See the note on version~2.}}
+
+\title[Shift schema drift left]{Shift schema drift left: policy-aware compile-time contracts for typed JVM and Spark pipelines}
+
+\author{\href{https://vitthalmirji.com/}{Vittal Mirji}}
+\orcid{0009-0005-3376-0457}
+\affiliation{
+ \position{Staff Software Engineer \& Independent researcher}
+ \country{ }
+}
+\email{vitthalmirji@gmail.com}
+\AtBeginDocument{
+ \hypersetup{
+ pdfauthor={Vittal Mirji},
+ pdftitle={Shift schema drift left: policy-aware compile-time contracts for typed JVM and Spark pipelines},
+ pdfsubject={A Scala 3 and Spark mechanism paper on compile-time structural compatibility and sink-boundary runtime schema checks},
+ pdfkeywords={Scala 3, Spark, macros, data contracts, schema drift, compile-time verification}
+ }
+}
+
+\begin{document}
+
+\begin{abstract}
+Schema drift in data pipelines is often caught only when a job touches real data. Typed-Dataset layers close part of this gap but require wholesale adoption; table-level enforcement systems close another part but operate at write time against a stored schema. We present a small Scala~3 framework that occupies the seam: it proves producer-to-contract \emph{structural} compatibility under explicit policies at compile time, derives Spark schemas from the same contract types, and re-checks the actual \texttt{DataFrame} schema at the sink boundary before write. The artifact fuses the compile-time witness with a policy-aware runtime comparator that implements structural subset semantics for backward- and forward-compatible field sets. Optionality is compared at compile time only, against the Scala types, and includes the optionality of sequence elements and map values; the runtime comparator ignores all three carriers of optionality, because Spark's file readers return the permissive value for each carrier on every format that does not record the claim, so a read schema cannot be distinguished from one whose producer stated that nulls are possible. That boundary is the subject of a companion paper~(\companionpreprint). Evaluation covers compile-time proofs, runtime policy tests, builder-path end-to-end tests, and reproducible benchmarks on two environments. This is a narrow, honest mechanism artifact; the broader claim that compile-time structural contracts deliver measurable productivity or reliability in practice is stated as motivation and left for future work.
+\end{abstract}
+
+\keywords{Scala 3, Spark, macros, data contracts, schema drift, compile-time verification}
+
+\maketitle
+\pagestyle{plain}
+
+\paragraph*{Note on version 2}
+Version~1 described the nested-collection-optionality check as part of the policy-aware runtime comparator. That was accurate for the artifact evaluated here, the repository at commit \texttt{6c5399f}, and it is no longer accurate for the artifact. Release \texttt{0.2.0} removed the optionality comparison from the runtime sink pin for the reason given in the abstract, and compares all three carriers at compile time instead; a row-level check that the contract's always-present positions actually carry values replaces it at the sink. Nothing else in this paper changes: the design, the evaluation, and the benchmarks describe the commit named above. The companion paper~(\companionpreprint) is the measurement study of the carriers themselves.
+
+\input{sections/01-introduction}
+\input{sections/02-background-and-model}
+\input{sections/03-framework-design}
+\input{sections/04-artifact-and-evaluation}
+\input{sections/06-related-work}
+\input{sections/07-limitations}
+\input{sections/08-conclusion}
+
+\bibliographystyle{ACM-Reference-Format}
+\bibliography{references}
+
+\end{document}
diff --git a/paper/arxiv-2604.16986/references.bib b/paper/arxiv-2604.16986/references.bib
new file mode 100644
index 0000000..0c64bb7
--- /dev/null
+++ b/paper/arxiv-2604.16986/references.bib
@@ -0,0 +1,149 @@
+@misc{scala3_reflection,
+ author = {{Scala Documentation}},
+ title = {Reflection | Macros in Scala 3},
+ year = {2026},
+ url = {https://docs.scala-lang.org/scala3/guides/macros/reflection.html},
+ note = {Accessed 2026-04-17}
+}
+
+@misc{spark_datatype,
+ author = {{Apache Spark}},
+ title = {DataType Companion Object | Spark 3.5.6 Scala API},
+ year = {2024},
+ url = {https://spark.apache.org/docs/3.5.6/api/scala/org/apache/spark/sql/types/DataType$.html},
+ note = {Accessed 2026-04-17}
+}
+
+@misc{spark_structtype,
+ author = {{Apache Spark}},
+ title = {StructType | Spark 3.5.6 Scala API},
+ year = {2024},
+ url = {https://spark.apache.org/docs/3.5.6/api/scala/org/apache/spark/sql/types/StructType.html},
+ note = {Accessed 2026-04-17}
+}
+
+@inproceedings{contracts_in_practice,
+ author = {Estler, H.-Christian and Furia, Carlo A. and Nordio, Martin and Piccioni, Marco and Meyer, Bertrand},
+ title = {Contracts in Practice},
+ booktitle = {FM 2014: Formal Methods},
+ series = {Lecture Notes in Computer Science},
+ volume = {8442},
+ pages = {230--246},
+ year = {2014},
+ publisher = {Springer},
+ doi = {10.1007/978-3-319-06410-9_17},
+ url = {https://doi.org/10.1007/978-3-319-06410-9_17}
+}
+
+@article{effectful_software_contracts,
+ author = {Moy, Cameron and Dimoulas, Christos and Felleisen, Matthias},
+ title = {Effectful Software Contracts},
+ journal = {Proceedings of the ACM on Programming Languages},
+ volume = {8},
+ number = {POPL},
+ pages = {2639--2666},
+ year = {2024},
+ doi = {10.1145/3632930},
+ url = {https://doi.org/10.1145/3632930}
+}
+
+@article{trace_contracts,
+ author = {Moy, Cameron and Felleisen, Matthias},
+ title = {Trace Contracts},
+ journal = {Journal of Functional Programming},
+ volume = {33},
+ year = {2023},
+ doi = {10.1017/S0956796823000096},
+ url = {https://doi.org/10.1017/S0956796823000096}
+}
+
+@article{dynamic_contract_analysis,
+ author = {Oraji, Yussur Mustafa and H{\"u}ck, Alexander and Bischof, Christian},
+ title = {Dynamic Contract Analysis for Parallel Programming Models},
+ journal = {arXiv preprint arXiv:2603.03023},
+ year = {2026},
+ url = {https://arxiv.org/abs/2603.03023}
+}
+
+@misc{compile_time_resolved_contracts,
+ author = {Persson, Jonas},
+ title = {Compile Time Resolved Contracts},
+ year = {2024},
+ url = {https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2024/p3317r0.pdf},
+ note = {WG21 P3317R0}
+}
+
+@misc{frameless,
+ author = {{Typelevel}},
+ title = {Frameless},
+ year = {2016},
+ url = {https://typelevel.org/frameless/},
+ note = {Accessed 2026-04-17}
+}
+
+@misc{iskra,
+ author = {{VirtusLab}},
+ title = {iskra},
+ year = {2023},
+ url = {https://github.com/VirtusLab/iskra},
+ note = {Accessed 2026-04-17}
+}
+
+@misc{chimney,
+ author = {{Scalalandio}},
+ title = {Supported transformations | Chimney},
+ year = {2019},
+ url = {https://chimney.readthedocs.io/en/stable/supported-transformations/},
+ note = {Accessed 2026-04-17}
+}
+
+@misc{delta_lake,
+ author = {{Delta Lake}},
+ title = {Batch reads and writes},
+ year = {2020},
+ url = {https://docs.delta.io/delta-batch/},
+ note = {Accessed 2026-04-17}
+}
+
+@misc{apache_iceberg,
+ author = {{Apache Iceberg}},
+ title = {Evolution},
+ year = {2020},
+ url = {https://iceberg.apache.org/docs/latest/evolution/},
+ note = {Accessed 2026-04-17}
+}
+
+@misc{shapeless,
+ author = {Sabin, Miles},
+ title = {shapeless},
+ year = {2014},
+ url = {https://github.com/milessabin/shapeless},
+ note = {Accessed 2026-04-17}
+}
+
+@misc{avro_compatibility,
+ author = {{Confluent}},
+ title = {Schema evolution and compatibility for Schema Registry on Confluent Platform},
+ year = {2011},
+ url = {https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html},
+ note = {Accessed 2026-04-17}
+}
+
+@inproceedings{aldrich_typestate_2009,
+ author = {Aldrich, Jonathan and Sunshine, Joshua and Saini, Darpan and Sparks, Zachary},
+ title = {Typestate-oriented programming},
+ booktitle = {Proceedings of the 24th ACM SIGPLAN Conference Companion on Object Oriented Programming Systems Languages and Applications},
+ year = {2009},
+ publisher = {ACM},
+ pages = {1015--1022},
+ doi = {10.1145/1639950.1640073},
+ url = {https://doi.org/10.1145/1639950.1640073}
+}
+
+@misc{avro_spec,
+ author = {{Apache Avro}},
+ title = {Apache Avro Specification},
+ year = {2025},
+ url = {https://avro.apache.org/docs/1.12.0/specification/},
+ note = {Accessed 2026-04-17}
+}
diff --git a/paper/arxiv-2604.16986/sections/01-introduction.tex b/paper/arxiv-2604.16986/sections/01-introduction.tex
new file mode 100644
index 0000000..b1da269
--- /dev/null
+++ b/paper/arxiv-2604.16986/sections/01-introduction.tex
@@ -0,0 +1,23 @@
+\section{Introduction}
+
+Schema drift is still found too late in many data systems. A producer renames a field, reorders a record, widens optionality, or changes the nullability of a nested collection, and the break often appears only when a job touches real data. Spark offers runtime schema comparators and schema-aware readers, but those checks still run after the program has been built and after an external boundary has already been crossed. A pipeline can therefore look typed in code while the key compatibility question is deferred to execution time~\cite{spark_datatype}.
+
+Typed-Dataset layers and table-level schema enforcement each close part of this gap but leave a seam. Typed-Dataset libraries such as \texttt{frameless} and \texttt{iskra} require that raw Spark \texttt{DataFrame} code be replaced with a typed wrapper across the job, which raises adoption cost in practice~\cite{frameless,iskra}. Table-level schema enforcement such as Delta Lake and Apache Iceberg operates at write time against a stored schema rather than at compile time against a declared Scala type, and therefore still runs after job submission~\cite{delta_lake,apache_iceberg}. Neither layer currently provides compile-time proof that a declared producer type conforms to a declared contract type under an explicit policy before the job is submitted at all. That is the seam this paper's artifact targets. The value of closing it is argued here and left for future work to measure.
+
+For typed JVM pipelines, much of that question is structural and available earlier. If the producer and sink contract are both represented as Scala types, then field names, nesting, collection shape, optionality, and defaulted fields are visible to the compiler. Earlier Scala derivation routes such as \texttt{shapeless} and Scala~3 \texttt{Mirror}-based derivation show adjacent implementation paths, but this artifact uses quoted reflection directly because it exposes \texttt{TypeRepr}, symbols, and type structure without introducing a separate generic wrapper layer~\cite{shapeless,scala3_reflection}. That gives us a concrete way to compute a normalized structural model at compile time and reject incompatible producer-to-contract pairs before a pipeline ships.
+
+Compile time is not enough. Even if application code proves that a declared producer type conforms to a contract, the actual runtime schema can still drift at the data boundary. CSV, JSON, and other external inputs do not obey the Scala types used in pipeline code. The artifact in this paper therefore combines two checks: compile-time proof that the declared producer type conforms to the target contract under an explicit policy, and a runtime pin that validates the actual Spark \texttt{DataFrame} schema before write. The first shifts structural drift detection into compilation and CI; the second protects the external boundary where runtime data can still violate assumptions.
+
+This paper presents a deliberately narrow framework for that combination. It derives normalized structural shapes from Scala~3 case classes, materializes compile-time evidence \texttt{SchemaConforms[Out, Contract, P]} under an explicit policy family, derives Spark schemas from the same contract types, and enforces a policy-aware runtime comparator at the sink boundary. Where Spark's documented comparators stop, the runtime layer adds a deeper check for nested collection optionality, which the artifact shows as a real drift source in tests~\cite{spark_datatype}.\carriermoved
+
+The artifact is structural rather than semantic. It does not model business rules, temporal properties, cross-record invariants, or external schema-registry workflows. That boundary matters. Prior work on software contracts shows that contracts are useful in practice, but also that the space is much richer than boundary shape checking; temporal and effectful contracts, for example, are well beyond the scope of this artifact~\cite{contracts_in_practice,trace_contracts,effectful_software_contracts}.
+
+Within that scope, this paper makes three contributions:
+
+\begin{enumerate}[leftmargin=1.5em]
+ \item A Scala~3 macro derivation of normalized structural shapes that rejects incompatible producer-to-contract pairs at compile time under an explicit, tested policy family, with path-rich drift diagnostics.
+ \item A sink-boundary design that fuses that compile-time witness with a policy-aware Spark runtime comparator, including subset semantics for \texttt{Backward}/\texttt{Forward} and a nested-collection-optionality check that Spark's built-in comparators omit. The subset semantics are unchanged in the current artifact; the optionality check is a compile-time check only as of release \texttt{0.2.0}.
+ \item A reproducible artifact demonstrating that this combination can be carried at low additional compile cost and at write-time runtime cost that is higher than Spark's built-in comparators but still within a single-digit microsecond per schema comparison.
+\end{enumerate}
+
+The paper is therefore best read as a mechanism paper with an honest artifact, not as a production-effectiveness paper. What is proven here is that a focused class of structural drift checks can move from runtime failure toward compile-time proof and CI while still retaining a runtime guard at the actual data boundary.
diff --git a/paper/arxiv-2604.16986/sections/02-background-and-model.tex b/paper/arxiv-2604.16986/sections/02-background-and-model.tex
new file mode 100644
index 0000000..2d65052
--- /dev/null
+++ b/paper/arxiv-2604.16986/sections/02-background-and-model.tex
@@ -0,0 +1,52 @@
+\section{Background and model}
+
+\subsection{Structural contracts in this paper}
+
+The term \emph{data contract} is overloaded. In some systems it includes business semantics, quality thresholds, temporal guarantees, ownership, and operational policy. In this paper, a contract is narrower: it is the structural shape expected at a checked sink boundary. That shape includes field names, field order where the chosen policy cares about order, nesting through product types, sequence and map structure, field optionality, nested collection optionality, and whether a contract field has a default value. It does not include domain constraints such as valid ranges, allowed enum values, uniqueness across records, or temporal properties across events.
+
+This choice is deliberate. The artifact is strongest where the compiler and the Spark schema model have a direct structural correspondence. Scala~3 macros can inspect case-class structure at compile time, and Spark represents row schemas using \texttt{StructType}, \texttt{StructField}, \texttt{ArrayType}, and \texttt{MapType}~\cite{scala3_reflection,spark_structtype}. That makes a focused structural contract model practical to derive and compare from both sides of the pipeline.
+
+We therefore use the following working definition:
+
+\begin{quote}
+A structural data contract is a typed description of the record shape expected at a checked boundary, together with a comparison policy that defines what counts as compatible producer-to-contract drift.
+\end{quote}
+
+This framing leaves room for richer contract systems in future work while keeping the current claims honest. Field-level optionality is intentionally ignored under all non-\texttt{Full} policies to match Spark's default field-nullability semantics; nested optionality inside arrays and map values is preserved and compared explicitly. Release \texttt{0.2.0} inverted both halves of that sentence, which is the largest difference between the artifact described here and the current one: field-level optionality is now compared at compile time under every policy except \texttt{Full}, renamed \texttt{Unchecked}, and no carrier is compared at the pin.
+
+\subsection{Policy family}
+
+Compatibility is not a single relation. Different boundaries need different answers to ``does this producer conform to that contract?'' The artifact makes that choice explicit through a small policy family:
+
+\begin{itemize}[leftmargin=1.5em]
+ \item \texttt{Exact} and \texttt{ExactUnorderedCI}: unordered matching by field name, case-insensitive, with no extras and no missing fields.
+ \item \texttt{ExactOrdered}: ordered matching by field name, case-sensitive.
+ \item \texttt{ExactOrderedCI}: ordered matching by field name, case-insensitive.
+ \item \texttt{ExactByPosition}: by-position matching where field names are ignored.
+ \item \texttt{Backward}: contract-by-name subset semantics. Producer extras are allowed; missing contract fields are allowed only when the contract field is optional or has a default value.
+ \item \texttt{Forward}: producer-by-name subset semantics. Producer fields must all exist in the contract, while the contract may contain additional fields.
+ \item \texttt{Full}: accept all structural combinations.
+\end{itemize}
+
+These policies separate intent from implementation. A sink can declare whether it wants exact matching, position-based matching, or subset-style compatibility with default relaxations. The compile-time and runtime layers then enforce that declared intent at their respective boundaries.
+The subset-style policies are intentionally case-sensitive at both layers to avoid ambiguity in subset direction. A case-insensitive variant would be a reasonable extension, but it is not part of the current artifact.
+
+\subsection{Why compile-time proof and runtime pin both exist}
+
+Compile-time proof answers a question about declared program structure: given producer type \texttt{Out}, contract type \texttt{R}, and policy \texttt{P}, can the compiler prove that \texttt{Out} structurally conforms to \texttt{R} under \texttt{P}? That is a property of code. Runtime pinning answers a different question: does the actual Spark schema reaching the sink still satisfy the declared contract under the same policy? That is a property of runtime data.
+
+The separation matters because data pipelines cross external boundaries. A typed source or transform can be declared against a case class, while the data actually read from files or tables may still drift independently. Compile-time proof cannot see those runtime schemas. Conversely, runtime validation alone comes too late to catch declared drift in the code that wires the pipeline. The artifact therefore treats the two checks as complementary rather than redundant. This split is deliberate: unlike designs that collapse structural validation into a single pass, the artifact separates the checks because they consume different inputs at different times, namely declared Scala types at compile time and external Spark schemas at the sink boundary.
+
+Spark's documented comparator surface helps explain the runtime side. The three relevant comparison entry points are:
+
+\begin{itemize}[leftmargin=1.5em]
+ \item \texttt{equalsIgnoreCaseAndNullability} for unordered name-based comparison with case-insensitive resolution and ignored field nullability,
+ \item \texttt{equalsStructurally} for by-position comparison, and
+ \item \texttt{equalsStructurallyByName} for ordered-by-name comparison with a supplied name resolver~\cite{spark_datatype}.
+\end{itemize}
+
+These comparators are a useful baseline for exact-style runtime matching, but they are not enough for the artifact's full policy story. The artifact also needs subset semantics for \texttt{Backward} and \texttt{Forward}, and it needs nested collection optionality checks for arrays and map values because Spark's default comparators do not cover that drift.\carriermoved
+
+\subsection{Relationship to broader contract work}
+
+The model here is intentionally smaller than the broader software-contract literature. Empirical work shows that contracts matter in real software projects, while more recent work on effectful and trace-oriented contracts shows that the contract design space extends far beyond record shape compatibility~\cite{contracts_in_practice,effectful_software_contracts,trace_contracts}. The right comparison for this paper is therefore not ``have we solved contracts?'' but ``have we built and evidenced a useful compile-time plus runtime mechanism for a focused structural boundary problem?''
diff --git a/paper/arxiv-2604.16986/sections/03-framework-design.tex b/paper/arxiv-2604.16986/sections/03-framework-design.tex
new file mode 100644
index 0000000..028ec90
--- /dev/null
+++ b/paper/arxiv-2604.16986/sections/03-framework-design.tex
@@ -0,0 +1,59 @@
+\section{Framework design}
+
+\subsection{Overview}
+
+The framework has one central design rule: the same contract type should drive both the compile-time proof path and the runtime sink pin. The compile-time path starts from Scala types and produces a structural witness. The runtime path starts from a Spark \texttt{DataFrame} schema and checks it against a \texttt{StructType} derived from the same contract type. The typed builder then fuses the two by requiring compile-time evidence at \texttt{addSink[R, P]} and by re-checking the actual schema before write.
+
+That design is what prevents the framework from becoming only a macro demo or only a runtime checker. If the compile-time path is not tied to the actual sink boundary, the paper risks proving a property of declarations while ignoring the data path. If the runtime path is not tied back to the same contract type, the paper becomes a Spark schema-checking utility without a compile-time story. The artifact keeps those two sides connected through the contract type \texttt{R} and the chosen policy \texttt{P}.
+
+\begin{figure}[H]
+ \centering
+ \includegraphics[width=\columnwidth]{fig01-architecture.pdf}
+ \caption{The same contract type \texttt{R} drives both compile-time proof and the runtime sink pin. Compile time rejects declared producer-to-contract drift in code. Runtime re-checks the actual Spark schema before write because external data boundaries can still violate the declared type path.}
+ \Description{Architecture overview of the framework. The compile-time path derives a normalized shape from producer type Out, contract type R, and policy P, then materializes SchemaConforms evidence required by the typed sink builder. The runtime path derives SparkSchema from R, applies PolicyRuntime for P to the actual DataFrame schema, and either allows or rejects the sink write.}
+ \label{fig:architecture}
+\end{figure}
+
+\subsection{Normalized structural model}
+
+The compile-time side needs an internal representation that is small enough to reason about and expressive enough to cover the supported shapes. The artifact uses a normalized \texttt{TypeShape} model with primitive shapes, optional shapes, sequence shapes, map shapes with atomic keys, and struct shapes whose fields carry four pieces of information: field name, nested shape, whether the field has a default value, and whether the field is optional.
+
+This representation is intentionally not user-facing. Its job is to normalize different Scala surface forms into one comparison model. Once two Scala types have been reduced to \texttt{TypeShape}, policy-specific comparison becomes a deterministic structural walk instead of ad hoc pattern matching over raw reflection nodes.
+
+The important detail is that optionality is tracked in two places. Field-level optionality is part of a field's own metadata. Nested optionality inside sequences and maps is represented in the nested shape itself. That distinction is what later allows the comparison to preserve and compare nested collection optionality instead of flattening it away.
+
+\subsection{Compile-time evidence materialization}
+
+Scala~3 quoted reflection makes it possible to materialize compatibility evidence directly at the call site~\cite{scala3_reflection}. The artifact defines a type class \texttt{SchemaConforms[Out, Contract, P]}. An inline given macro derives that witness by inspecting \texttt{Out} and \texttt{Contract} through \texttt{quotes.reflect}, computing normalized shapes for both sides, comparing them under the selected \texttt{SchemaPolicy}, and either returning the witness or aborting compilation with a path-rich drift report.
+
+The drift report is part of the design, not a side effect. If the framework only failed with a generic missing-given error, it would be much less useful in practice. The macro therefore records missing fields, extra fields, and mismatches with structural paths so that a failed compile points at the actual drift shape rather than only at the location where evidence was requested.
+
+\subsection{Deriving runtime schemas from the same contract type}
+
+The runtime path begins by deriving a Spark \texttt{StructType} from the contract case class. This derivation mirrors the supported compile-time shape family: primitives map to Spark atomic types, nested case classes map to nested \texttt{StructType}, sequence-like containers map to \texttt{ArrayType}, maps with atomic keys map to \texttt{MapType}, and \texttt{Option[T]} affects field or nested nullability. Default-valued contract fields are recorded in metadata so that runtime subset policies can make the same allowance that compile time makes for \texttt{Backward}.
+
+The result is a runtime contract schema that is not manually duplicated. The same Scala contract type \texttt{R} is used for both compile-time comparison and runtime schema derivation. That keeps the sink boundary honest and avoids the common failure mode where one contract exists in typed code and a second, loosely synchronized version exists in runtime configuration.
+
+\subsection{Policy-aware runtime comparison}
+
+At runtime the framework selects a \texttt{PolicyRuntime[P]} instance. The exact-style policies mirror Spark's documented comparator families: unordered-by-name exact matching, ordered-by-name exact matching, and by-position exact matching~\cite{spark_datatype}. The artifact does not simply call Spark's built-in comparators and stop there. Instead, it implements a comparator that follows the same broad name-order directions while adding two behaviors that matter for the contract story: nested collection optionality is checked explicitly for arrays and map values, and \texttt{Backward} and \texttt{Forward} are implemented as real subset relations rather than being collapsed into equality.\carriermoved
+
+This is the most important runtime design decision in the artifact. Without it, the framework would overclaim semantic parity between compile-time and runtime enforcement. With it, the runtime layer becomes a true boundary check for the policies the paper claims to support.
+
+Release \texttt{0.2.0} reverses that judgement for the optionality carriers, and only for them: parity is not available where the pin receives a reader default rather than a producer's claim, so the carriers are checked at compile time and the sink asks the rows instead. The subset relations for \texttt{Backward} and \texttt{Forward} are unaffected.
+
+\begin{figure}[H]
+ \centering
+ \includegraphics[width=\columnwidth]{fig02-policy-family.pdf}
+ \caption{Policy family supported by the artifact. Exact-style policies differ on name and order sensitivity. Subset-style policies make direction explicit. Nested collection optionality is checked across all policies except \texttt{Full}.}
+ \Description{Policy overview diagram. Exact and ExactUnorderedCI are unordered by name and case-insensitive. ExactOrdered and ExactOrderedCI preserve field order. ExactByPosition compares by position. Backward allows producer extras and optional or defaulted contract omissions. Forward allows the contract to contain more fields than the producer. Full accepts all combinations.}
+ \label{fig:policy-family}
+\end{figure}
+
+Figure~\ref{fig:policy-family} summarizes the supported policy family at a glance. The exact-style policies differ on name and order sensitivity, while the subset-style policies make direction explicit.
+
+\subsection{Typed builder fuse}
+
+The user-facing enforcement point is the typed builder. A sink is added through \texttt{addSink[R, P]}, and that call requires compile-time evidence that the current producer type conforms to contract \texttt{R} under policy \texttt{P}. If evidence cannot be derived, the pipeline does not compile. If evidence is derived, the built pipeline still performs the runtime schema pin before write.
+
+This builder encodes a lightweight typestate discipline~\cite{aldrich_typestate_2009}: the phantom type parameter tracks which capabilities the builder has acquired, and the type system prevents illegal transitions such as adding a sink before a source exists. This builder surface matters because it ties the proof obligation to the checked boundary itself. The compile-time witness is not a separate demonstration step that users can forget to wire in. It is part of the sink-construction path. The runtime pin is not an optional logging step after the fact. It is part of the write path. Mid-pipeline pins intentionally use the default unordered case-insensitive comparator to catch gross shape drift during reshaping; policy-aware enforcement is applied at the sink boundary only, where the declared contract type is tied to the actual write path. That fusion is what turns the artifact from a library of helpers into a coherent boundary-enforcement design.
diff --git a/paper/arxiv-2604.16986/sections/04-artifact-and-evaluation.tex b/paper/arxiv-2604.16986/sections/04-artifact-and-evaluation.tex
new file mode 100644
index 0000000..e1e31eb
--- /dev/null
+++ b/paper/arxiv-2604.16986/sections/04-artifact-and-evaluation.tex
@@ -0,0 +1,58 @@
+\section{Artifact and evaluation}
+
+\subsection{Artifact scope}
+
+We evaluate the clean reference repository at commit \texttt{6c5399f} rather than the larger framework effort that motivated it. The repository is intentionally small. It contains the compile-time policy model, the Scala~3 macro derivation for \texttt{SchemaConforms}, Spark schema derivation from contract types, a policy-aware runtime comparator, a typed builder path that enforces sink checks, and a reproducible benchmark harness.\footnote{A practitioner-oriented tutorial exposition of the same mechanism appears separately at \url{https://rockthejvm.com/articles/compile-time-data-contracts-scala-3}.} This is a mechanism evaluation: the question is whether the repository proves the structural properties it claims, not whether it already delivers industrial-scale outcomes.
+
+Accordingly, the evaluation asks four narrow questions: whether compile-time witness generation accepts and rejects the intended structural cases; whether the runtime layer catches drift that declared Scala types cannot rule out, including subset-policy behavior and nested collection optionality drift; whether the typed builder path enforces both layers at the actual sink boundary; and whether the mechanism overhead is measurable and reproducible across the two saved environments.
+
+\subsection{Compile-time and runtime coverage}
+
+Compile-time coverage is centered on direct witness generation and compile-fail behavior. The tests exercise positive conformance for exact-style, backward-compatible, and forward-compatible policies, and rejection paths for reordering, missing required fields, and nested optionality drift. The covered shape family includes nested case classes, sequences, maps, optional fields, and defaulted contract fields.
+
+The runtime layer is evaluated separately because it has a different job. The compile-time witness proves compatibility for declared Scala types. The runtime layer validates the actual Spark schema that reaches the sink. The runtime tests therefore ask whether the exact-style pin rejects nested collection optionality drift that Spark ignores by default,\carriermoved whether \texttt{Backward} and \texttt{Forward} behave as true subset relations at runtime, and whether contract-derived metadata allows missing defaulted fields where the policy permits them.
+
+\subsection{Builder-path end-to-end checks}
+
+The builder-path tests are the most operational part of the evaluation. They show that the paper is not only about a macro witness in isolation. \texttt{PipelineBuilder.addSink[R, P]} requires compile-time evidence for the current producer type, and the built pipeline still validates the actual \texttt{DataFrame} schema before write. The end-to-end checks cover a compile-time rejection at the sink, a green path where both the declared type and runtime schema satisfy the sink policy, a red path where the runtime schema violates the policy after compile time has succeeded, and no-transform subset-policy paths for \texttt{Backward} and \texttt{Forward}. This is the closest point in the artifact to the intended application boundary, so it is the strongest evidence that the two enforcement layers are wired together rather than merely coexisting in the same repository.
+
+\subsection{Benchmark harness}
+
+The benchmark harness is intentionally small. It measures compile-time witness-generation overhead on synthetic but representative schemas and runtime comparator overhead on nested \texttt{StructType} comparisons. It does not claim end-to-end Spark job performance.
+
+The saved snapshots show compile-time deltas of \texttt{+0.270s}, \texttt{+0.397s}, and \texttt{+0.513s} on local \texttt{macOS arm64} for \texttt{10}, \texttt{25}, and \texttt{50} schema pairs. On GitHub-hosted \texttt{Ubuntu x86\_64}, the same sizes measure \texttt{+0.847s}, \texttt{+1.000s}, and \texttt{+1.880s}. Runtime unordered exact matching stays in the low-microsecond range per schema comparison on both environments, while exact-by-position matching stays below microsecond scale in both saved runs. The custom unordered comparator is roughly \texttt{17--25$\times$} slower than Spark's built-in \texttt{equalsIgnoreCaseAndNullability} baseline in the saved runs, but that cost remains acceptable for this artifact because the check runs once per sink write, not per row.
+
+The right interpretation is limited but useful: the harness reproduces across two environments and shows that the mechanism is measurable and stable enough to package as artifact evidence. The wrong interpretation would be a broad claim about low overhead across machines or organizations. Table~\ref{tab:benchmarks} therefore reports bounded artifact evidence, not a general performance baseline.
+
+\begin{table}[t]
+ \caption{Saved benchmark snapshots for the artifact head. These numbers show reproducible harness output, not a broad cross-machine performance baseline.}
+ \Description{Two-part benchmark table. The left half reports compile-time deltas for 10, 25, and 50 schema pairs on local macOS arm64 and GitHub-hosted Ubuntu x86_64. The right half reports runtime comparator averages for by-position matching, unordered exact matching, and two Spark baseline comparators on the same two environments.}
+ \label{tab:benchmarks}
+ \centering
+ \scriptsize
+ \textbf{Compile-time overhead}
+
+ \begin{tabular}{lrrrr}
+ \toprule
+ Pairs & Local (s) & Local (\%) & Ubuntu (s) & Ubuntu (\%) \\
+ \midrule
+ 10 & 0.270 & 11.8 & 0.847 & 12.6 \\
+ 25 & 0.397 & 13.5 & 1.000 & 11.1 \\
+ 50 & 0.513 & 13.9 & 1.880 & 16.6 \\
+ \bottomrule
+ \end{tabular}
+
+ \vspace{0.6em}
+ \textbf{Runtime comparator averages}
+
+ \begin{tabular}{lrr}
+ \toprule
+ Benchmark & Local (ns) & Ubuntu (ns) \\
+ \midrule
+ By-position & 116.82 & 180.55 \\
+ Unordered exact & 4736.41 & 8149.74 \\
+ Spark ignore-case & 278.92 & 331.42 \\
+ Spark structural & 332.13 & 380.36 \\
+ \bottomrule
+ \end{tabular}
+\end{table}
diff --git a/paper/arxiv-2604.16986/sections/05-industrial-context-and-lessons.tex b/paper/arxiv-2604.16986/sections/05-industrial-context-and-lessons.tex
new file mode 100644
index 0000000..64f950b
--- /dev/null
+++ b/paper/arxiv-2604.16986/sections/05-industrial-context-and-lessons.tex
@@ -0,0 +1,7 @@
+\section{Industrial context and lessons}
+
+The larger FlowForge effort is useful here only as industrial context. It informed the motivation, parts of the application-side API shape, and the decision to split out a smaller clean artifact. It does not serve as proof for the closed claims in this paper.
+
+Two lessons from that effort matter most. First, typed examples can drift away from the real execution path surprisingly easily. A framework can present a convincing compile-time story while the dataset actually written follows a different route. That is exactly the kind of gap a paper can overclaim if it treats examples or demos as evidence. Second, framework narratives naturally accumulate connectors, evolution, observability, and product positioning long before the enforcement path is fully closed.
+
+Those lessons are why this paper evaluates the small artifact directly. FlowForge still contributes value here, but the value is different: it shows why typed boundaries matter operationally and how easily a framework story can get ahead of the actual checked data path. That is useful context for this section. It is not a substitute for the direct proof in Sections~3 and~4.
diff --git a/paper/arxiv-2604.16986/sections/06-related-work.tex b/paper/arxiv-2604.16986/sections/06-related-work.tex
new file mode 100644
index 0000000..85e1172
--- /dev/null
+++ b/paper/arxiv-2604.16986/sections/06-related-work.tex
@@ -0,0 +1,19 @@
+\section{Related work}
+
+This paper sits at the intersection of software contracts, static-versus-dynamic checking, and typed data-pipeline boundaries. The closest primary sources for the implementation itself are the Scala~3 reflection documentation and the Spark schema-comparator APIs, because the artifact is built directly on those mechanisms~\cite{scala3_reflection,spark_datatype}. These are mechanism sources, not related-work contributions in the research sense.
+
+The closest implementation-level neighbors are typed-Dataset libraries. \texttt{frameless} and \texttt{iskra} provide typed Spark layers that carry schema information through Dataset operations, but they do so by asking the user to replace raw \texttt{DataFrame} code with typed wrappers across the job~\cite{frameless,iskra}. This artifact targets a narrower seam: producer-to-contract conformance is required only at the checked boundary, and the rest of the job may remain in raw Spark code.
+
+Record-to-record derivation libraries solve a different adjacent problem. Chimney, and earlier Scala generic-derivation work built on \texttt{shapeless}, derive field-to-field transformations with rich compile-time diagnostics~\cite{chimney,shapeless}. That macro technique is close in spirit to the compile-time side of this artifact, but the goal is different: Chimney proves transformation existence, whereas this artifact proves structural conformance under an explicit policy family and couples that witness to a Spark sink-boundary pin.
+
+Table-level schema enforcement is also close, but at a different layer. Delta Lake and Apache Iceberg validate writes against stored table metadata and support schema evolution at that boundary~\cite{delta_lake,apache_iceberg}. Those systems are valuable runtime seatbelts, but they operate after job submission and against stored schemas rather than against declared Scala producer and contract types. The artifact here is therefore complementary rather than competing.
+
+The policy names \texttt{Backward} and \texttt{Forward} deliberately reuse vocabulary that is familiar from Avro and schema-registry compatibility systems~\cite{avro_spec,avro_compatibility}. In this paper, however, the relation is structural over Scala product types and Spark schemas. It is not a value-level compatibility proof for serialized messages or registered schemas.
+
+The broader contract literature provides the right contrast. Empirical work such as \emph{Contracts in Practice} shows that contract disciplines remain relevant in real software, which supports the motivation for making contract intent explicit~\cite{contracts_in_practice}. More expressive systems such as \emph{Effectful Software Contracts} and \emph{Trace Contracts} show how much richer the contract space becomes when effects, histories, or traces are part of the property being checked~\cite{effectful_software_contracts,trace_contracts}. By comparison, this artifact is deliberately narrower: it checks structural compatibility at a typed boundary.
+
+There is also a useful static-versus-dynamic comparison point. \emph{Dynamic Contract Analysis for Parallel Programming Models} treats dynamic contracts as complementary to static analysis rather than as a complete replacement~\cite{dynamic_contract_analysis}. That lesson aligns with this paper's design choice: compile-time proof and runtime checking do different jobs and should be combined at the boundary instead of being presented as competitors.
+
+Finally, work outside the JVM ecosystem helps position the contribution. P3317R0 argues for contracts that can often be resolved at compile time so that runtime cost can be reduced or removed~\cite{compile_time_resolved_contracts}. The artifact here is not a direct analogue of that C++ proposal, but it shares the same directional idea: when part of a contract question is structural and visible to the compiler, deferring everything to runtime is unnecessary.
+
+The novelty claim of this paper is therefore narrow. It is not a new general theory of contracts. It is a small, evidenced combination of compile-time structural proof and runtime boundary checking for a typed Scala and Spark setting.
diff --git a/paper/arxiv-2604.16986/sections/07-limitations.tex b/paper/arxiv-2604.16986/sections/07-limitations.tex
new file mode 100644
index 0000000..fc3a2d5
--- /dev/null
+++ b/paper/arxiv-2604.16986/sections/07-limitations.tex
@@ -0,0 +1,13 @@
+\section{Limitations}
+
+The first limitation is scope. The artifact checks structural compatibility, not business semantics. It can prove that two typed schemas match under a policy, but it does not know whether an \texttt{age} field must be positive, whether a \texttt{currency} field must come from an approved domain, or whether two records must satisfy a temporal or cross-row invariant. Those are different kinds of contract problems.
+
+The second limitation is supported-shape breadth. The artifact intentionally focuses on product-like case-class schemas, nested products, sequence-like containers, maps with atomic keys, optional fields, nested collection optionality, and defaulted contract fields. That is enough for the mechanism paper, but it is not a full schema system for all Spark or Scala data representations.
+
+The third limitation is ecosystem reach. There is no external schema-registry integration, no industrial deployment pack in the clean repo, and no measured user-study evidence about productivity or usability. The builder API may be teachable, but the paper should not convert that into a measured productivity claim.
+
+The fourth limitation is evaluation breadth. The benchmark results come from two saved snapshots on one local machine and one GitHub-hosted Ubuntu runner. They show reproducibility of the harness, not a stable cross-machine baseline. The runtime numbers are micro-bench results for schema comparison, not end-to-end Spark job measurements.
+
+The fifth limitation is value-add evidence. The paper argues that boundary-only compile-time proof may lower adoption cost relative to a typed-Dataset rewrite and may catch drift earlier than write-time schema enforcement, but it does not measure those claims. A follow-up evaluation should compare time-to-first-drift-catch in CI, drift-catches-per-month in a deployed pipeline, and adoption-cost delta versus a typed-Dataset rewrite or a \texttt{DataFrame} plus runtime-only pin.
+
+The final limitation is the role of FlowForge. FlowForge informed the motivation, application-side realism, and overclaim analysis, but it does not serve as proof for the clean artifact's closed claims. That separation is a strength for honesty, but it also means the paper stops short of industrial-effectiveness claims for now.
diff --git a/paper/arxiv-2604.16986/sections/08-conclusion.tex b/paper/arxiv-2604.16986/sections/08-conclusion.tex
new file mode 100644
index 0000000..431e14d
--- /dev/null
+++ b/paper/arxiv-2604.16986/sections/08-conclusion.tex
@@ -0,0 +1,7 @@
+\section{Conclusion}
+
+This paper presents a focused mechanism for shifting structural contract drift detection left in typed JVM and Spark pipelines. The artifact proves compile-time structural compatibility under an explicit policy family, derives runtime schemas from the same contract types, and checks the actual Spark schema at the sink boundary before write.
+
+The contribution is narrow by design: a bounded class of structural mismatches can be rejected at compile time and re-checked at runtime where external data still matters. The artifact does not claim semantic contracts, industrial metrics, or broad performance generalization.
+
+That is the main result. Compile-time proof and runtime boundary checking belong together, and this artifact shows a small honest way to combine them while leaving a clean base for richer contract models and stronger industrial evidence.
diff --git a/paper/evidence/claims.json b/paper/evidence/claims.json
index 2b12f57..a37df76 100644
--- a/paper/evidence/claims.json
+++ b/paper/evidence/claims.json
@@ -2,10 +2,10 @@
"generated_from": "ARTIFACT.md",
"generator": "paper/scripts/claims_ledger.py",
"revision": {
- "commit": "2c214f323e0f76d240de66ad6d23d0230541349c",
+ "commit": "1b1ce22a0fbba0cc063dbfe7826de931996ffe54",
"tracked_tree_matches_commit": true
},
- "note": "Artifact claims only. Research claims are numbered RC* in paper/RESEARCH-DESIGN.md and are not part of this file. Status values: closed, partial, open.",
+ "note": "Artifact claims only. The manuscript's research claims are numbered RC* and are not part of this file. Status values: closed, partial, open.",
"status_counts": {
"closed": 7,
"partial": 0,
diff --git a/paper/figures/README.md b/paper/figures/README.md
deleted file mode 100644
index 6b36680..0000000
--- a/paper/figures/README.md
+++ /dev/null
@@ -1,17 +0,0 @@
-# Figure sources
-
-This directory holds manuscript figure sources.
-
-Current state:
-
-- Mermaid source is stored as `.mmd`
-- manuscript-ready PNG renders already exist for the first two figures
-
-Render locally with Mermaid CLI:
-
-```bash
-npx -y @mermaid-js/mermaid-cli -i paper/figures/fig01-architecture.mmd -o paper/figures/fig01-architecture.png
-npx -y @mermaid-js/mermaid-cli -i paper/figures/fig02-policy-family.mmd -o paper/figures/fig02-policy-family.png
-```
-
-The canonical figure wording now lives in the local-only prep docs under `docs/internals/prep-docs/`.
diff --git a/paper/scripts/abstract-field.py b/paper/scripts/abstract-field.py
new file mode 100755
index 0000000..cdb10b6
--- /dev/null
+++ b/paper/scripts/abstract-field.py
@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+"""Print a paper's abstract as the plain text arXiv's abstract field wants.
+
+arXiv keeps the abstract twice: once in the submitted source, once in the metadata field, and the two are
+expected to say the same thing. Transcribing the second by hand is how they drift, so this derives it from
+main.tex instead. main.tex stays the only place the abstract is written.
+
+Run from the repository root:
+
+ python3 paper/scripts/abstract-field.py
+ PAPER_DIR=arxiv-2604.16986 python3 paper/scripts/abstract-field.py
+
+Exit code 1 means the abstract still holds markup this script does not know how to render as plain text,
+which is a prompt to extend it rather than to paste the result.
+"""
+
+import os
+import re
+import sys
+from pathlib import Path
+
+# Commands whose argument is the text itself: the markup carries font or emphasis, which the field drops.
+INLINE_TEXT_COMMANDS = ("texttt", "code", "emph", "textbf", "textit")
+
+# Spacing and punctuation control sequences, mapped to what a reader of plain text should see.
+LITERAL_REPLACEMENTS = {
+ r"\,": "", # thin space inside a quantity, e.g. 25\,\% reads as 25%
+ r"\%": "%",
+ r"\&": "&",
+ r"\_": "_",
+ "~": " ", # non-breaking space
+ "--": "-",
+}
+
+
+def abstract_source(main_tex: Path) -> str:
+ match = re.search(r"\\begin\{abstract\}(.*?)\\end\{abstract\}", main_tex.read_text(), re.S)
+ if match is None:
+ sys.exit(f"No abstract environment found in {main_tex}")
+ return match.group(1).strip()
+
+
+def own_macros(main_tex: Path) -> dict[str, str]:
+ """The paper's own argument-less \\newcommand definitions, so the field reads what the PDF reads."""
+ definitions = re.findall(r"\\newcommand\{\\([A-Za-z]+)\}\{([^{}]*)\}", main_tex.read_text())
+ return {name: body for name, body in definitions}
+
+
+def to_plain_text(latex: str, macros: dict[str, str]) -> str:
+ text = latex
+ for name, body in macros.items():
+ text = re.sub(rf"\\{name}(?![A-Za-z])", body, text)
+ for command in INLINE_TEXT_COMMANDS:
+ # Repeat until stable: an inline command may wrap another.
+ pattern = re.compile(rf"\\{command}\{{([^{{}}]*)\}}")
+ while pattern.search(text):
+ text = pattern.sub(r"\1", text)
+ for source, replacement in LITERAL_REPLACEMENTS.items():
+ text = text.replace(source, replacement)
+ return re.sub(r"\s+", " ", text).strip()
+
+
+def main() -> None:
+ paper_dir = Path(__file__).resolve().parent.parent / os.environ.get("PAPER_DIR", "three-carriers")
+ main_tex = paper_dir / "main.tex"
+ text = to_plain_text(abstract_source(main_tex), own_macros(main_tex))
+
+ leftover = sorted(set(re.findall(r"\\[A-Za-z]+|[{}\\]", text)))
+ if leftover:
+ sys.exit(f"Abstract still holds markup this script does not render: {', '.join(leftover)}")
+
+ print(text)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/paper/scripts/build-arxiv-bundle.sh b/paper/scripts/build-arxiv-bundle.sh
index 42837ad..49c906e 100755
--- a/paper/scripts/build-arxiv-bundle.sh
+++ b/paper/scripts/build-arxiv-bundle.sh
@@ -1,8 +1,13 @@
#!/usr/bin/env bash
set -euo pipefail
-root_dir="$(cd "$(dirname "$0")/.." && pwd)"
+# Which paper to bundle. There are two of them under paper/, so this is a parameter with the
+# current one as the default. PAPER_DIR is a directory name under paper/, not a path.
+root_dir="$(cd "$(dirname "$0")/../${PAPER_DIR:-three-carriers}" && pwd)"
out_dir="${1:-$root_dir/dist/arxiv}"
+# The parent has to exist before it can be resolved to an absolute path, and on a clean checkout
+# it does not: dist/ is build output and is not tracked.
+mkdir -p "$(dirname "$out_dir")"
dist_dir="$(cd "$(dirname "$out_dir")" && pwd)"
zip_path="${2:-$dist_dir/compile-time-data-contracts-arxiv.zip}"
@@ -28,10 +33,32 @@ cp "$root_dir/references.bib" "$out_dir/"
if [ -f "$root_dir/00README.json" ]; then
cp "$root_dir/00README.json" "$out_dir/"
fi
-if [ -f "$root_dir/main.bbl" ]; then
- cp "$root_dir/main.bbl" "$out_dir/"
+# arXiv typesets the .bbl supplied in the bundle and does not run bibtex, so a .bbl that is missing
+# or older than anything able to change the bibliography publishes the wrong references, and does it
+# silently: the PDF builds, the citations just say something else.
+bbl="$root_dir/main.bbl"
+if [ ! -f "$bbl" ]; then
+ echo "No $bbl. arXiv does not run bibtex, so the bundle needs one." >&2
+ echo "Generate it: (cd $root_dir && tectonic -X compile main.tex --keep-intermediates)" >&2
+ exit 1
fi
+stale_against=()
+for source in "$root_dir/references.bib" "$root_dir/main.tex" "$root_dir/sections/"*.tex; do
+ if [ "$source" -nt "$bbl" ]; then
+ stale_against+=("${source#"$root_dir"/}")
+ fi
+done
+
+if [ "${#stale_against[@]}" -gt 0 ]; then
+ echo "main.bbl is older than: ${stale_against[*]}" >&2
+ echo "Regenerate it before bundling, or the bundle ships stale references:" >&2
+ echo " (cd $root_dir && tectonic -X compile main.tex --keep-intermediates)" >&2
+ exit 1
+fi
+
+cp "$bbl" "$out_dir/"
+
# Flatten sections into top level
for section in "${section_names[@]}"; do
cp "$root_dir/sections/$section.tex" "$out_dir/"
diff --git a/paper/scripts/claims_ledger.py b/paper/scripts/claims_ledger.py
index 0cb5548..2ba5039 100644
--- a/paper/scripts/claims_ledger.py
+++ b/paper/scripts/claims_ledger.py
@@ -98,8 +98,8 @@ def main() -> int:
"generator": Path(__file__).relative_to(ROOT).as_posix(),
"revision": revision(),
"note": (
- "Artifact claims only. Research claims are numbered RC* in paper/RESEARCH-DESIGN.md and are not "
- "part of this file. Status values: closed, partial, open."
+ "Artifact claims only. The manuscript's research claims are numbered RC* and are not part of "
+ "this file. Status values: closed, partial, open."
),
"status_counts": {s: sum(1 for c in claims if c["status"] == s) for s in ("closed", "partial", "open")},
"claims": claims,
diff --git a/paper/scripts/fetch-overleaf-bbl.mjs b/paper/scripts/fetch-overleaf-bbl.mjs
index aabe33c..66df8d5 100644
--- a/paper/scripts/fetch-overleaf-bbl.mjs
+++ b/paper/scripts/fetch-overleaf-bbl.mjs
@@ -5,8 +5,11 @@ import { dirname, join, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
const __dirname = dirname(fileURLToPath(import.meta.url));
-const paperDir = resolve(__dirname, '..');
-const projectMetaPath = join(paperDir, '.olcli.json');
+// The Overleaf binding belongs to the scaffold, which holds both papers. The .bbl belongs to the
+// one that was compiled, so that is a parameter with the current paper as the default.
+const scaffoldDir = resolve(__dirname, '..');
+const paperDir = resolve(scaffoldDir, process.env.PAPER_DIR || 'three-carriers');
+const projectMetaPath = join(scaffoldDir, '.olcli.json');
const defaultOutputPath = join(paperDir, 'main.bbl');
const baseUrl = process.env.OVERLEAF_BASE_URL || 'https://www.overleaf.com';
const cookieName = process.env.OVERLEAF_COOKIE_NAME || 'overleaf_session2';
@@ -26,7 +29,7 @@ function getSessionCookie() {
const candidates = [
join(process.cwd(), '.olauth'),
- join(paperDir, '.olauth')
+ join(scaffoldDir, '.olauth')
];
for (const candidate of candidates) {
diff --git a/paper/three-carriers/00README.json b/paper/three-carriers/00README.json
new file mode 100644
index 0000000..fb87470
--- /dev/null
+++ b/paper/three-carriers/00README.json
@@ -0,0 +1,13 @@
+{
+ "sources": [
+ {
+ "usage": "toplevel",
+ "filename": "main.tex"
+ }
+ ],
+ "spec_version": 1,
+ "texlive_version": 2025,
+ "process": {
+ "compiler": "pdflatex"
+ }
+}
diff --git a/paper/three-carriers/README.md b/paper/three-carriers/README.md
new file mode 100644
index 0000000..d9734d4
--- /dev/null
+++ b/paper/three-carriers/README.md
@@ -0,0 +1,61 @@
+# Three carriers, one bit
+
+"Three carriers, one bit: optionality in schema-equality checking, measured in Spark", Vittal
+Mirji. Not submitted yet. The intended arXiv primary category is `cs.PL`, matching
+[arxiv-2604.16986/](../arxiv-2604.16986/README.md), which this paper is the companion to.
+
+It is a measurement study of how the three carriers of optionality behave in schema-equality
+checking: `StructField.nullable`, `ArrayType.containsNull`, and `MapType.valueContainsNull`. The
+other paper describes the mechanism and the policy family.
+
+## Layout
+
+- `main.tex` uses `acmart` with the `sigplan` option, and `nonacm` for local drafting
+- `00README.json` records the intended top-level source and the submission-target compiler metadata
+- `sections/` holds sections 1-8, one file each
+- `figures/` holds the Mermaid sources and the rendered PNG and PDF assets
+- `main.bbl` and `main.pdf` are build output and are ignored by git
+
+The evidence the paper cites is one level up, under `paper/evidence/`, `paper/corpus/` and
+`paper/scripts/`, because it is shared and because the paper cites it by repo-relative path.
+
+## Building
+
+```bash
+cd paper/three-carriers && tectonic -X compile main.tex
+paper/scripts/build-arxiv-bundle.sh
+```
+
+The bundle script defaults to this directory. It flattens `sections/` into the top level and
+rewrites the `\input` paths to match, because arXiv wants a flat source tree.
+
+arXiv does not run bibtex, so the bundle has to carry a current `main.bbl`, and the script refuses
+to build one that is older than `references.bib`, `main.tex` or any section. Refresh it with
+`tectonic -X compile main.tex --keep-intermediates`, which is the only way tectonic writes it to
+disk.
+
+## Submitting
+
+Categories: `cs.PL` primary, matching the companion paper, with `cs.SE` and `cs.DB` as cross-lists.
+The paper is about a language mechanism, the subject it measures is a data-platform schema
+boundary, and the requirement it is driven by is a software-engineering one.
+
+The abstract metadata field has to say the same thing as the abstract in the PDF. It is derived from
+`main.tex` rather than transcribed, so the two cannot drift:
+
+```bash
+python3 paper/scripts/abstract-field.py
+```
+
+Current counts, for the comments field: 24 pages, 3 figures, 7 tables.
+
+```
+24 pages, 3 figures, 7 tables. Companion to arXiv:2604.16986, which describes the mechanism and the
+policy family; this paper is the measurement study of the three carriers of optionality. Harnesses,
+saved runs and the full 2668-cell verdict matrix at
+https://github.com/vim89/compile-time-data-contracts
+```
+
+Once this paper has an identifier, it has to be filled into `\companionpreprint` in
+[../arxiv-2604.16986/main.tex](../arxiv-2604.16986/main.tex) before that paper's v2 replacement can
+be posted.
diff --git a/paper/three-carriers/figures/README.md b/paper/three-carriers/figures/README.md
new file mode 100644
index 0000000..21581df
--- /dev/null
+++ b/paper/three-carriers/figures/README.md
@@ -0,0 +1,18 @@
+# Figure sources
+
+This directory holds manuscript figure sources.
+
+Current state:
+
+- Mermaid source is stored as `.mmd`
+- manuscript-ready PNG renders already exist for the first two figures
+
+Render locally with Mermaid CLI:
+
+```bash
+npx -y @mermaid-js/mermaid-cli -i paper/three-carriers/figures/fig01-architecture.mmd -o paper/three-carriers/figures/fig01-architecture.png
+npx -y @mermaid-js/mermaid-cli -i paper/three-carriers/figures/fig02-policy-family.mmd -o paper/three-carriers/figures/fig02-policy-family.png
+```
+
+The `.mmd` sources are the only place the figure wording is written; the renders are generated from
+them by the commands above.
diff --git a/paper/three-carriers/figures/fig01-architecture.mmd b/paper/three-carriers/figures/fig01-architecture.mmd
new file mode 100644
index 0000000..1edf5a0
--- /dev/null
+++ b/paper/three-carriers/figures/fig01-architecture.mmd
@@ -0,0 +1,51 @@
+flowchart LR
+ subgraph CT["Compile-time path"]
+ OUT["Producer type: Out"]
+ CONTRACT["Contract type: R"]
+ POLICY["Policy type: P"]
+ SHAPE["Normalize to TypeShape"]
+ WITNESS["SchemaConforms[Out, R, P]"]
+ FUSE["Typed builder fuse: addSink[R, P]"]
+
+ OUT --> SHAPE
+ CONTRACT --> SHAPE
+ POLICY --> SHAPE
+ SHAPE --> WITNESS
+ WITNESS --> FUSE
+ end
+
+ BUILD["Compiled pipeline"]
+
+ subgraph RT["Runtime sink boundary"]
+ FOUND["Actual Spark schema: DataFrame.schema"]
+ EXPECTED["Expected schema: SparkSchema[R]"]
+ RPOLICY["PolicyRuntime[P]"]
+ COMPARE["Policy-aware runtime comparator"]
+ ALLOW["Sink write allowed"]
+ REJECT["Sink write rejected"]
+
+ FOUND --> COMPARE
+ EXPECTED --> COMPARE
+ RPOLICY --> COMPARE
+ COMPARE -->|Match| ALLOW
+ COMPARE -->|Drift| REJECT
+ end
+
+ FUSE --> BUILD
+ BUILD --> FOUND
+ CONTRACT --> EXPECTED
+ POLICY --> RPOLICY
+
+ style OUT fill:#e1f5ff,stroke:#0066cc,color:#000
+ style CONTRACT fill:#e1f5ff,stroke:#0066cc,color:#000
+ style POLICY fill:#fff4e1,stroke:#cc8800,color:#000
+ style SHAPE fill:#f0e1ff,stroke:#8800cc,color:#000
+ style WITNESS fill:#f0e1ff,stroke:#8800cc,color:#000
+ style FUSE fill:#fff4e1,stroke:#cc8800,color:#000
+ style BUILD fill:#e1f5ff,stroke:#0066cc,color:#000
+ style FOUND fill:#e1f5ff,stroke:#0066cc,color:#000
+ style EXPECTED fill:#e1f5ff,stroke:#0066cc,color:#000
+ style RPOLICY fill:#fff4e1,stroke:#cc8800,color:#000
+ style COMPARE fill:#f0e1ff,stroke:#8800cc,color:#000
+ style ALLOW fill:#e1ffe1,stroke:#2d7a2d,color:#000
+ style REJECT fill:#ffe1e1,stroke:#cc0000,color:#000
diff --git a/paper/three-carriers/figures/fig01-architecture.pdf b/paper/three-carriers/figures/fig01-architecture.pdf
new file mode 100644
index 0000000..80c0899
Binary files /dev/null and b/paper/three-carriers/figures/fig01-architecture.pdf differ
diff --git a/paper/three-carriers/figures/fig01-architecture.png b/paper/three-carriers/figures/fig01-architecture.png
new file mode 100644
index 0000000..1fe77e5
Binary files /dev/null and b/paper/three-carriers/figures/fig01-architecture.png differ
diff --git a/paper/figures/fig02-policy-family.mmd b/paper/three-carriers/figures/fig02-policy-family.mmd
similarity index 100%
rename from paper/figures/fig02-policy-family.mmd
rename to paper/three-carriers/figures/fig02-policy-family.mmd
diff --git a/paper/figures/fig02-policy-family.pdf b/paper/three-carriers/figures/fig02-policy-family.pdf
similarity index 100%
rename from paper/figures/fig02-policy-family.pdf
rename to paper/three-carriers/figures/fig02-policy-family.pdf
diff --git a/paper/figures/fig02-policy-family.png b/paper/three-carriers/figures/fig02-policy-family.png
similarity index 100%
rename from paper/figures/fig02-policy-family.png
rename to paper/three-carriers/figures/fig02-policy-family.png
diff --git a/paper/main.tex b/paper/three-carriers/main.tex
similarity index 98%
rename from paper/main.tex
rename to paper/three-carriers/main.tex
index bc95a45..1c46315 100644
--- a/paper/main.tex
+++ b/paper/three-carriers/main.tex
@@ -26,7 +26,7 @@
\author{\href{https://vitthalmirji.com/}{Vittal Mirji}}
\orcid{0009-0005-3376-0457}
\affiliation{
- \position{Staff Software Engineer}
+ \position{Staff Software Engineer \& Independent researcher}
\country{ }
}
\email{vitthalmirji@gmail.com}
diff --git a/paper/references.bib b/paper/three-carriers/references.bib
similarity index 83%
rename from paper/references.bib
rename to paper/three-carriers/references.bib
index 9d601d8..a7ae447 100644
--- a/paper/references.bib
+++ b/paper/three-carriers/references.bib
@@ -22,34 +22,43 @@ @misc{spark_structtype
note = {Accessed 2026-04-17}
}
-@article{contracts_in_practice,
- author = {Estler, Hans-Christian and Furia, Carlo A. and Nordio, Martin and Piccioni, Marco and Meyer, Bertrand},
- title = {Contracts in Practice},
- journal = {arXiv preprint arXiv:1211.4775},
- year = {2012},
- url = {https://arxiv.org/abs/1211.4775}
+@inproceedings{contracts_in_practice,
+ author = {Estler, H.-Christian and Furia, Carlo A. and Nordio, Martin and Piccioni, Marco and Meyer, Bertrand},
+ title = {Contracts in Practice},
+ booktitle = {FM 2014: Formal Methods},
+ series = {Lecture Notes in Computer Science},
+ volume = {8442},
+ pages = {230--246},
+ year = {2014},
+ publisher = {Springer},
+ doi = {10.1007/978-3-319-06410-9_17},
+ url = {https://doi.org/10.1007/978-3-319-06410-9_17}
}
@article{effectful_software_contracts,
- author = {Moy, Matthew and Dimoulas, Christos and Felleisen, Matthias},
+ author = {Moy, Cameron and Dimoulas, Christos and Felleisen, Matthias},
title = {Effectful Software Contracts},
journal = {Proceedings of the ACM on Programming Languages},
+ volume = {8},
+ number = {POPL},
+ pages = {2639--2666},
year = {2024},
doi = {10.1145/3632930},
url = {https://doi.org/10.1145/3632930}
}
@article{trace_contracts,
- author = {Moy, Matthew and Felleisen, Matthias},
+ author = {Moy, Cameron and Felleisen, Matthias},
title = {Trace Contracts},
journal = {Journal of Functional Programming},
+ volume = {33},
year = {2023},
doi = {10.1017/S0956796823000096},
url = {https://doi.org/10.1017/S0956796823000096}
}
@article{dynamic_contract_analysis,
- author = {Oraji, Mina and H{\"u}ck, Gerhard and Bischof, Christian},
+ author = {Oraji, Yussur Mustafa and H{\"u}ck, Alexander and Bischof, Christian},
title = {Dynamic Contract Analysis for Parallel Programming Models},
journal = {arXiv preprint arXiv:2603.03023},
year = {2026},
@@ -153,10 +162,10 @@ @article{deequ
@misc{great_expectations,
author = {{Great Expectations}},
- title = {Great Expectations: Expectations reference},
+ title = {Great Expectations: Expectation Gallery},
year = {2025},
- url = {https://docs.greatexpectations.io/docs/reference/expectations/},
- note = {Accessed 2026-10-07}
+ url = {https://greatexpectations.io/expectations/},
+ note = {Accessed 2026-10-08}
}
@misc{empirical_standards,
@@ -195,6 +204,16 @@ @misc{spark_source_datatype_42
note = {Accessed 2026-10-07}
}
+@misc{mirji_shift_left_2026,
+ author = {Mirji, Vittal},
+ title = {Shift schema drift left: policy-aware compile-time contracts for typed JVM and Spark pipelines},
+ year = {2026},
+ eprint = {2604.16986},
+ archivePrefix = {arXiv},
+ primaryClass = {cs.PL},
+ url = {https://arxiv.org/abs/2604.16986}
+}
+
@misc{spark_53717,
author = {{Apache Spark}},
title = {SPARK-53717: Fix misleading documentation for \texttt{MapType.valueContainsNull}},
diff --git a/paper/sections/01-introduction.tex b/paper/three-carriers/sections/01-introduction.tex
similarity index 97%
rename from paper/sections/01-introduction.tex
rename to paper/three-carriers/sections/01-introduction.tex
index 0a6a478..6d09ffb 100644
--- a/paper/sections/01-introduction.tex
+++ b/paper/three-carriers/sections/01-introduction.tex
@@ -26,7 +26,7 @@ \subsection{Contributions}
\begin{enumerate}[leftmargin=1.5em]
\item \textbf{A characterisation of optionality in Spark's schema comparison.} Over a drift taxonomy generated from \texttt{StructType}'s grammar rather than chosen by hand, the ten schema-equality configurations admitted by a stated discovery rule over Spark 3.5.6's sources (\S\ref{sec:predicate-family}) are measured in both argument orders, and the whole suite is then re-run against a Spark 4.2.0 build, where all 2668 verdicts come out identical; reading the sources of \texttt{v4.0.4} and \texttt{v4.1.3} shows why, and extends the result to the releases between: the predicate bodies are unchanged, the carrier set is still exactly three slots, and the one comparator Spark~4 adds is \texttt{private[sql]} and does not discriminate (\S\ref{sec:spark4}). The ten shipped configurations partition into accept-all and reject-all over the three carriers, with no discriminating configuration; the two compatible-nullability methods are the only direction-sensitive members of the family and are sensitive on exactly the optionality edits, making them nullability subtype checks rather than equality checks; and \texttt{equalsStructurallyByName} constrains the name tree and nothing else, accepting a struct replaced by a scalar of the same name.
\item \textbf{An externally grounded statement of what is missing, and evidence that it is wanted.} The conjunction of Avro's name-matching rule and a directional containment requirement on the field carrier, which Confluent's optional-field compatibility levels show real systems are specified over, is satisfied by none of Spark's ten configurations and by five of the twenty-three measured, four of them this artifact's comparison engine and the fifth Avro's own resolution check, included as an external baseline. In a corpus of 528 Avro schemas from twenty public repositories, 132 schemas (25\,\%) declare optionality unevenly across the three carriers, which is reported as prevalence of the property and not as a count of schemas no predicate can serve.
- \item \textbf{A mechanism that occupies the empty cell, and the characterisation of why it has to sit where it does.} A Scala~3 macro derivation proves producer-to-contract structural conformance under an explicit policy family, and a sink-boundary runtime pin re-checks the actual frame. The division of labour is not a design preference: the field carrier is checked at compile time because that is the last point at which the claim exists, and the runtime pin deliberately ignores it.
+ \item \textbf{A mechanism that occupies the empty cell, and the characterisation of why it has to sit where it does.} A Scala~3 macro derivation proves producer-to-contract structural conformance under an explicit policy family, and a sink-boundary runtime pin re-checks the actual frame. The division of labour is not a design preference: the field carrier is checked at compile time because that is the last point at which the claim exists, and the runtime pin deliberately ignores it. The mechanism and its policy family are described in a companion paper~\cite{mirji_shift_left_2026}; what is new here is the characterisation of the carriers it has to compare and the measurement of what the shipped predicates do with them.
\end{enumerate}
One bound belongs here rather than only in the limitations section, because it determines how every number below should be read. This is a characterisation, not an evaluation of a technique. It reports what Spark's predicates do, what two external specifications demand, and how often real schemas carry the configuration the two disagree about. It does not report that making contract intent explicit improves anything a practitioner would notice: there is no user study, no deployed-pipeline measurement, no time-to-first-drift-catch figure, and no comparison against the cost of a typed-\texttt{Dataset} rewrite. A reader looking for evidence that the mechanism is \emph{worth adopting} will not find it here and should read \S\ref{sec:limits-effectiveness} for what such evidence would have to be.
diff --git a/paper/sections/02-background-and-model.tex b/paper/three-carriers/sections/02-background-and-model.tex
similarity index 100%
rename from paper/sections/02-background-and-model.tex
rename to paper/three-carriers/sections/02-background-and-model.tex
diff --git a/paper/sections/03-framework-design.tex b/paper/three-carriers/sections/03-framework-design.tex
similarity index 100%
rename from paper/sections/03-framework-design.tex
rename to paper/three-carriers/sections/03-framework-design.tex
diff --git a/paper/sections/04-artifact-and-evaluation.tex b/paper/three-carriers/sections/04-artifact-and-evaluation.tex
similarity index 99%
rename from paper/sections/04-artifact-and-evaluation.tex
rename to paper/three-carriers/sections/04-artifact-and-evaluation.tex
index e27c68f..7f42b56 100644
--- a/paper/sections/04-artifact-and-evaluation.tex
+++ b/paper/three-carriers/sections/04-artifact-and-evaluation.tex
@@ -313,7 +313,7 @@ \subsection{Mechanism overhead}
\subsection{Artifact and reproduction}
-The measured repository is the clean reference implementation, not the larger framework effort that motivated it. It holds the compile-time policy model, the Scala~3 macro derivation of the conformance witness, Spark schema derivation from contract types, the policy-aware runtime comparator, a typed builder path that enforces the sink check, the probe programs that produce every number above, and the benchmark harness.\footnote{A practitioner-oriented exposition of the same mechanism appears separately at \url{https://rockthejvm.com/articles/compile-time-data-contracts-scala-3}.}
+The measured repository is the clean reference implementation, not the larger framework effort that motivated it. It holds the compile-time policy model, the Scala~3 macro derivation of the conformance witness, Spark schema derivation from contract types, the policy-aware runtime comparator, a typed builder path that enforces the sink check, the probe programs that produce every number above, and the benchmark harness.\footnote{The mechanism itself is the subject of the companion paper~\cite{mirji_shift_left_2026}. A practitioner-oriented exposition appears separately at \url{https://rockthejvm.com/articles/compile-time-data-contracts-scala-3}.}
Three probe programs produce the evidence files, and each writes its own inputs' provenance into its output: the comparator matrix, the nullability-consequence transcript, and the corpus facts. The corpus fetch script is separate and is the only step that requires network access. Figure~\ref{fig:reproduce} is the full list of commands, so that every number in this section can be regenerated without reading the build file to find out how. Each writes to the path the committed evidence is at, so a re-run that agrees is a byte-identical file and a re-run that does not is a diff. The harnesses live in an unpublished \texttt{probe} module rather than in \texttt{ctdc-spark}: one of them reaches two \texttt{private[sql]} comparators from a subpackage of \texttt{org.apache.spark.sql}, which is surface no pipeline should resolve. Every number here belongs to one revision of the comparison engine, so the artifact's claim ledger is generated rather than written: \code{paper/evidence/claims.json} carries the commit it was produced at, and is emitted only if every repository path it cites as evidence still exists, which keeps a claim from reading as settled after the file behind it moved. Its claims are numbered \texttt{AC*} and the research claims of the design record \texttt{RC*}, because the two sets were previously numbered alike and said different things.
diff --git a/paper/sections/05-industrial-context-and-lessons.tex b/paper/three-carriers/sections/05-industrial-context-and-lessons.tex
similarity index 100%
rename from paper/sections/05-industrial-context-and-lessons.tex
rename to paper/three-carriers/sections/05-industrial-context-and-lessons.tex
diff --git a/paper/sections/06-related-work.tex b/paper/three-carriers/sections/06-related-work.tex
similarity index 100%
rename from paper/sections/06-related-work.tex
rename to paper/three-carriers/sections/06-related-work.tex
diff --git a/paper/sections/07-limitations.tex b/paper/three-carriers/sections/07-limitations.tex
similarity index 100%
rename from paper/sections/07-limitations.tex
rename to paper/three-carriers/sections/07-limitations.tex
diff --git a/paper/sections/08-conclusion.tex b/paper/three-carriers/sections/08-conclusion.tex
similarity index 100%
rename from paper/sections/08-conclusion.tex
rename to paper/three-carriers/sections/08-conclusion.tex