Add Apache Arrow Go Parquet writer and reader

.github/workflows/ci.yml

@@ -49,6 +49,25 @@ jobs:
       - name: Test (pytest)
         run: uv run pytest -q
 
+  go-sidecar:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version-file: sidecars/go/go.mod
+          cache-dependency-path: sidecars/go/go.sum
+      - name: Build Apache Arrow Go writer and reader
+        working-directory: sidecars/go
+        run: go build -o raincloud-parquet-go .
+      - uses: astral-sh/setup-uv@v5
+      - run: uv sync --python 3.11 --extra dev --extra build --inexact
+      - name: Verify settings and read-back fidelity
+        env:
+          RAINCLOUD_SIDECAR_PARQUET_GO: ${{ github.workspace }}/sidecars/go/raincloud-parquet-go
+          RAINCLOUD_READER_PARQUET_GO: ${{ github.workspace }}/sidecars/go/raincloud-parquet-go
+        run: uv run --no-sync pytest -q tests/test_go_sidecar.py tests/test_parquet_options.py tests/test_reader_fidelity.py
+
   java-sidecars:
     # Validate the JVM conformance lanes: the parquet@java lane is a Gradle
     # composite build over the parquet-arrow-java git submodule, so this job

.gitignore

@@ -97,3 +97,5 @@ docs/snapshot.json
 /clients/c/build/
 # profile promotion lock beside tracked profiles
 docs/**/.profiles-write.lock
+
+/sidecars/go/raincloud-parquet-go

CHANGELOG.md

@@ -7,6 +7,13 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht
 
 ## [Unreleased]
 
+### Added
+
+- `parquet@go`: Apache Arrow Go 18.8.0's `pqarrow` writer and reader sidecars,
+  with streaming IPC input, shared Parquet settings, page indexes on with
+  statistics, and read-back verification before publication. Explicit page
+  checksum and page-row-limit requests fail because the library cannot write them.
+
 ### Changed
 
 - **`parquet@py` writes a page index and page checksums by default.** With

README.md

@@ -323,6 +323,13 @@ its own. To try again with nothing changed, pass `--retry-errors`
 (`raincloud build <slug> --retry-errors`, `raincloud load <slug> --retry-errors`,
 or `load(..., retry_errors=True)`).
 
+### Apache Arrow Go Parquet writer
+
+The optional `parquet@go` sidecar writes and verifies Parquet with Apache Arrow Go's
+`pqarrow`. It accepts the shared writer settings, enables page indexes with
+statistics, and refuses requests to write page checksums or enforce an exact page-row limit. See
+[the build instructions and option matrix](sidecars/README.md#apache-arrow-go).
+
 ## Licensing and redistribution
 
 Each entry carries an SPDX identifier, a source URL, and an explicit

raincloud/_registry.py

@@ -94,6 +94,7 @@
 # class serves all of them, so the whole cell is data — which is why these are
 # declared rather than written out as five near-identical registrations.
 SIDECAR_EXPORTERS: dict[str, tuple[str, str]] = {
+    "parquet@go": ("parquet", "raincloud-export-parquet-go"),
     "parquet@rs": ("parquet", "raincloud-export-parquet-rs"),
     "parquet@java": ("parquet", "raincloud-export-parquet-java"),
     "parquet@hardwood": ("parquet", "raincloud-export-parquet-hardwood"),

raincloud/pipeline/export/readers.py

@@ -472,6 +472,7 @@ def run_reader(
 register_reader(PyarrowParquetReader())
 register_reader(VortexPyReader())
 register_reader(PyarrowOrcReader())
+register_reader(SidecarReader("parquet@go", {"parquet"}, "raincloud-read-parquet-go"))
 register_reader(SidecarReader("parquet@rs", {"parquet"}, "raincloud-read-parquet-rs"))
 register_reader(SidecarReader("vortex@rs", {"vortex"}, "raincloud-read-vortex-rs"))
 register_reader(SidecarReader("parquet@java", {"parquet"}, "raincloud-read-parquet-java"))

sidecars/README.md

@@ -171,7 +171,7 @@ passes to a sidecar in this form:
 | `RAINCLOUD_PARQUET_PAGE_CHECKSUMS` | `1` or `0`: a CRC in every page header, or none | the install, only when set |
 
 Page indexes and checksums default on wherever the pinned writer supports them:
-`parquet@py` and `parquet@java` write both, `parquet@rs` writes indexes but cannot write
+`parquet@py` and `parquet@java` write both, `parquet@rs` and `parquet@go` write indexes but cannot write
 checksums, and `parquet@hardwood` writes checksums but cannot write indexes. Page
 statistics require the recipe's statistics to be enabled. Other unset variables use
 the library's defaults (zstd and statistics on, for the first two). Counts use the
@@ -183,21 +183,21 @@ or none, or out of the codec's range) are refused for every lane before any writ
 A lane refuses, as a failed round-trip naming the variable, a setting its library
 cannot honour:
 
-| | py (pyarrow 24) | rs (arrow-rs 59.2) | java (parquet-arrow-java 0.3.0) | hardwood (1.1.0.Beta1) |
-|---|---|---|---|---|
-| codec | all | all | not `brotli` | all |
-| compression level | yes | yes | yes | no |
-| statistics off | yes | yes | yes | no |
-| statistics, first N columns | yes | yes | yes | no |
-| page index on | yes (default) | yes (default) | yes (default) | no |
-| page index off | yes | yes | no, with statistics on | yes |
-| page index, first N columns | no: all or none | yes | no | no |
-| page bytes | yes | yes | yes | yes |
-| page rows | yes | yes, checked every 1,024 values | yes | no |
-| dictionary on / off | yes | yes | yes | yes (off is PLAIN) |
-| dictionary page bytes | yes | yes | yes | no |
-| page checksums on | yes (default) | no | yes (default) | yes (always) |
-| page checksums off | yes | yes (always) | yes | no |
+| | py (pyarrow 24) | rs (arrow-rs 59.2) | java (parquet-arrow-java 0.3.0) | hardwood (1.1.0.Beta1) | go (Apache Arrow Go 18.8.0) |
+|---|---|---|---|---| --- |
+| codec | all | all | not `brotli` | all | all |
+| compression level | yes | yes | yes | no | yes |
+| statistics off | yes | yes | yes | no | yes |
+| statistics, first N columns | yes | yes | yes | no | yes |
+| page index on | yes (default) | yes (default) | yes (default) | no | yes (default) |
+| page index off | yes | yes | no, with statistics on | yes | yes |
+| page index, first N columns | no: all or none | yes | no | no | yes |
+| page bytes | yes | yes | yes | yes | yes |
+| page rows | yes | yes, checked every 1,024 values | yes | no | no |
+| dictionary on / off | yes | yes | yes | yes (off is PLAIN) | yes |
+| dictionary page bytes | yes | yes | yes | no | yes |
+| page checksums on | yes (default) | no | yes (default) | yes (always) | no |
+| page checksums off | yes | yes (always) | yes | no | yes (always) |
 
 Each library measures a page its own way, so the same `RAINCLOUD_PARQUET_PAGE_BYTES`
 does not give identical pages in every lane. The java column's page index gaps are
@@ -407,3 +407,37 @@ Upgrade their pins together and re-measure supported cells; a package version
 change alone is not evidence of conformance. Preserve skipped and unsupported
 lanes in the results. Software updates do not move stored data or refresh an
 active catalog revision.
+
+## Apache Arrow Go
+
+`parquet@go` uses Apache Arrow Go's `parquet/pqarrow` (18.8.0), which streams
+Arrow IPC record batches directly into its pure-Go Parquet writer. It is a
+different implementation from `github.com/parquet-go/parquet-go`.
+
+```bash
+cd sidecars/go
+go build -o raincloud-parquet-go .
+# Both contracts are served by the same binary (--canonical selects read mode).
+export RAINCLOUD_SIDECAR_PARQUET_GO="$PWD/raincloud-parquet-go"
+export RAINCLOUD_READER_PARQUET_GO="$PWD/raincloud-parquet-go"
+cd ../..
+python -m raincloud.pipeline.export <slug> --format parquet@go
+```
+
+For PATH discovery, install the binary as both `raincloud-export-parquet-go` and
+`raincloud-read-parquet-go`. Use `export.priority: {"parquet": ["go", "py"]}`
+to prefer it. It reads its own output through pqarrow before promotion. Comparison
+is streamed, permits reversible type normalization, ignores field metadata, and
+preserves float sign bits through nested values. The writer stores the Arrow
+schema and preserves native Parquet VARIANT, including null rows; its verdict
+checks both the Parquet logical annotation and the Arrow extension on read.
+
+Row groups accumulate across IPC batches, capped by the recipe's row limit and
+checked every 1,024 rows against encoded bytes and estimated decoded bytes. An
+uncompressed planning encoder runs alongside the writer so compression does not
+move row-group boundaries; disabling the encoded-byte target skips that encoder.
+Selective statistics/index settings are refused when dotted field names create
+ambiguous leaf paths requiring different settings. Page
+rows and checksums-on are refused: Arrow Go exposes neither an exact page-row
+limit nor a checksum writer option. Other controls are listed above. Compression
+and statistics come from the recipe, and page indexes default on with statistics.