collect

A Rust project to collect positional data into Hive-partitioned Parquet files with Zstd compression — the bronze layer of a medallion pipeline for maritime (AIS) data. Source-specific collectors, data parsers, and two batch tools (table maintenance, and derived vessel / track / stop / voyage tables), run either ad hoc or unattended (Nomad, containers).

New here? Start with TUTORIAL.md — a four-stage walkthrough from local storage to Apache Iceberg. This page is a map of the docs, not a guide.

The binaries

• collect-file — recursive file ingestion (plain, gzip, bzip2, zip)
• collect-socket — TCP line-stream ingestion
• collect-kafka — Kafka topic ingestion with at-least-once offset commits
• collect-aisstream — aisstream.io WebSocket ingestion
• ais-parse — silver layer: decode AIS sentences into typed Parquet (positions, statics, meteo, binary, aids to navigation), via ScottSyms/nmea-parser; local or S3 on both sides
• aisstream-parse — silver layer: decode aisstream.io JSON from bronze Parquet into the same typed tables; local or S3 on both sides
• ais-compact — Iceberg table maintenance for RustFS and other catalogs with no compactor: sorted compaction, snapshot expiry, orphan cleanup
• ais-tracks — derived tables built with DataFusion SQL: vessel identity, annotated track points, tracks, stops matched to World Port Index ports, and voyages; written to a namespace you choose
• collect-foundation — foundation reference data in Iceberg (namespace foundation, warehouse maritimedata by default): the GeoNames placenames gazetteer, and vessel evidence (Paris MoU, Tokyo MoU and USCG port state control; TSB Canada marine occurrences)
• vessel-identity — the vessel identity registry in Iceberg (namespace identity, warehouse maritimedata by default): collects Transport Canada, IACS, USCG PSIX and USCG MISLE inspection vessel records with full provenance and history, and reconciles them into one record per physical vessel with a permanent id
• vessel-imagery — ship photos for the registry's vessels (namespace imagery): freely licensed photos from Wikimedia Commons by IMO number, held/restricted photos from a manifest, each with its licence, plus visual attributes (type, colours, funnels, masts, gear) read by a System 1 model (Laya) and rolled up into a per-vessel visual profile

All four collectors support optional S3/MinIO/RustFS upload, and can register or fully decode into Apache Iceberg as they ingest — see TUTORIAL.md stages 3 and 4.

Documentation map

Doc	Covers
TUTORIAL.md	Staged walkthrough: local storage → partitioning → S3 → Iceberg
SCHEMAS.md	Column-by-column reference for every table the programs write
CLI_REFERENCE.md	Every flag/env var, exit codes, health signals, metrics, delivery guarantees
COLLECT_FILE.md / COLLECT_SOCKET.md / COLLECT_KAFKA.md / COLLECT_AISSTREAM.md	Per-collector usage and behavior
AIS_PARSE.md / AISSTREAM_PARSE.md	Decoded (silver) table schemas, incremental/idempotent decoding
AIS_COMPACT.md	Compacting, sorting, expiring and cleaning Iceberg tables
AIS_TRACKS.md	Derived vessel, track, stop and voyage tables: design, columns, run order, queries, limits
COLLECT_FOUNDATION.md	Foundation reference tables: GeoNames placenames and vessel evidence; sources, data model, load semantics, caveats
VESSEL_IDENTITY.md	Vessel identity registry: sources and their authorization, ingest and reconciliation rules, tables, queries, departures from the specification
VESSEL_IMAGERY.md	Vessel imagery: Commons and held photos, licence handling, the System 1 question set, profiles, the eval, tables
DOCKER_HEALTH_CHECK.md	Health-check mechanics for containers
NOMAD.md	Nomad job definitions and cluster deployment

Docker

docker build -t collect .
docker run -d --name data-ingest \
  -e TCP_HOST=153.44.253.27 -e TCP_PORT=5631 -e SOURCE=norway-tcp \
  -v $(pwd)/output:/data \
  collect:latest

The image defaults to collect-socket; use --entrypoint /usr/local/bin/<binary> (collect-file, collect-kafka, collect-aisstream, ais-parse, collect-foundation, vessel-identity, vessel-imagery, aisstream-parse) to run any other binary the image ships. A docker-compose example lives in docker-compose.yml; see NOMAD.md for cluster deployment instead.

Building from source

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
git clone <repository-url>
cd collect
cargo build --release --workspace
./target/release/collect-file --help

Configuration precedence

For any flag: command-line argument > environment variable > --config file > built-in default. See CLI_REFERENCE.md for --config's TOML format.