Loom turns FHIR-shaped data in ArangoDB into authorized, queryable flat dataframes in ClickHouse.
ArangoDB is the canonical graph store: Loom loads NDJSON, extracts FHIR resources and relationships, profiles the populated graph, and compiles typed dataframe recipes into scoped AQL. ClickHouse is the publication store: Loom streams resolved recipe outputs into versioned physical tables, records their schema and visibility in an Arango-backed catalog, and exposes them through a stable reader API.
Loom is deliberately not an arbitrary AQL gateway, ClickHouse SQL proxy, or a replacement FHIR model. It is a recipe-driven graph-to-flat data service with explicit authorization and publication boundaries.
Explorer Builder authoring is intent-driven: the browser submits the versioned V2 authoring document, Loom lowers it to a native recipe, and the existing recipe compiler produces the scoped plan/AQL. The browser never constructs or repairs the recipe AST. See the Explorer authoring contract.
flowchart LR
NDJSON["FHIR NDJSON"] --> Ingest["Ingest and catalog"]
Ingest --> Arango["ArangoDB\nresources, fhir_edge, field catalog"]
Recipe["Versioned dataframe recipe"] --> Compile["Resolve, validate, compile"]
Arango --> Compile
Compile --> AQL["Scoped AQL stream"]
AQL --> Publish["Atomic publication catalog commit"]
Publish --> ClickHouse["Versioned flat tables"]
Publish --> Catalog["Arango publication catalog"]
Catalog --> Reads["REST published rows and aggregations"]
ClickHouse --> Reads
Compile --> Dataframe["REST FHIR dataframe query"]
- FHIR NDJSON ingestion into ArangoDB resource collections and
fhir_edge. - Immutable dataset generations and active-generation selection.
- Populated-field, reference, traversal, and pivot discovery.
- Strict, versioned dataframe recipes and compiler-backed AQL execution.
- Scoped publication of recipe outputs into ClickHouse.
- A durable Arango catalog that maps exact dataframe selectors to current published outputs.
- Multi-project,
project_id-identifiable andauth_resource_path-scoped flat-data reads.
Loom does not expose arbitrary ClickHouse tables, arbitrary SQL, or an Elasticsearch/Guppy fallback. Published dataframe reads require an exact recipe, translation-version, and output selector.
| Surface | Purpose |
|---|---|
arango-fhir-proto |
Operator CLI for loading data, loading immutable generations, catalog discovery, and local dataframe materialization. |
arango-fhir-server |
HTTP server for graph compilation/control and flat dataframe reads. |
POST /api/v1/projects/{project}/dataframes/rows |
Page through a published dataframe. |
POST /api/v1/projects/{project}/dataframes/aggregations |
Run named aggregations on a published dataframe. |
POST /api/v1/projects/{project}/dataframes/run |
Compile and run a FHIR dataframe query against ArangoDB. |
POST /api/v1/projects/:project/explorers/... |
REST Explorer lifecycle and V2 intent authoring used by the Builder. |
POST /api/v1/datasets/:project/generations/:generation |
Multipart immutable-generation loader used by ETL. |
The canonical contract for every server route is
api/openapi.yaml. It includes health, generation
ingestion/activation, recipe execution, dataframe queries, Explorer lifecycle,
and Explorer authoring operations. The :project path parameter is the tenancy
identity used for authorization and becomes the published row project_id
(for example, HTAN_INT-BForePC).
- Load FHIR NDJSON into ArangoDB. Loom creates resource collections,
fhir_edge, andfhir_field_catalog. - For production snapshots, load an immutable dataset generation and let the active manifest select it for a project.
- Resolve a registered recipe against the populated graph and caller scope.
- Compile the resolved recipe to parameterized AQL and stream rows from ArangoDB.
- Publish output streams to new ClickHouse physical tables. Loom injects the
reserved
project_idandauth_resource_pathcolumns, records schema/provenance in Arango, and atomically advances the logical publication pointer only once every output is READY. - Query the logical dataframe through the project-scoped rows endpoint. The reader resolves authorized current outputs from the catalog; it never accepts a physical ClickHouse table name.
The configured ClickHouse database is created by the server during startup if
necessary. Publication creates its tables itself. No DDL or alias-management
HTTP endpoint is required. The ClickHouse account needs database creation plus
CREATE, INSERT, SELECT, and DROP privileges.
To run the complete no-auth Explorer demo with the locked FHIR Aggregator sample, use:
make demo-upOpen the standalone Explorer. See Run the standalone Explorer demo for the seeded dataset, smoke checks, logs, and reset commands.
To launch from a separate repository's hydrated META data and native Loom
CONFIG workspace, including local CONFIG write-back on Publish, see
Launch Loom from a data repository.
For warm backend and Explorer UI iterations with an isolated Compose project
and real browser verification, see Fast local Explorer development.
The public make verify-* targets run native Playwright Test cases. verify-current
and verify-full temporarily edit watched source for HMR checks, so run them one
at a time.
For backend-only development, the lightweight Compose stack starts ArangoDB and ClickHouse:
rtk docker compose --project-directory . -f tests/deploy/compose.local.yaml up -d
make generate-fhir
make buildLoad the bundled example data into a mutable local namespace:
./bin/arango-fhir-proto load \
--url http://127.0.0.1:8529 \
--database fhir_proto \
--meta-dir META \
--project ARANGODB_PROTO \
--auth-resource-path EllrottLab-GDC_DataFor a production-shaped immutable snapshot, use load-generation instead. It
requires an opaque generation identifier and deliberately forbids truncation:
./bin/arango-fhir-proto load-generation \
--url http://127.0.0.1:8529 \
--database fhir_proto \
--meta-dir META \
--project ARANGODB_PROTO \
--generation local-meta-1 \
--auth-resource-path EllrottLab-GDC_DataStart Loom locally with an unrestricted development principal:
./bin/arango-fhir-server \
--listen :8080 \
--no-auth \
--url http://127.0.0.1:8529 \
--database fhir_proto \
--clickhouse-url clickhouse://127.0.0.1:9000 \
--clickhouse-database loom \
--dataframer-recipe /path/to/dataframer.jsonAdd --dataset-generations when reading the active immutable generation. That
mode disables the legacy one-file import endpoint.
Useful local URLs:
Run a saved dataframe request after the server is up:
make dataframe-demo DATAFRAME_REQUEST=/path/to/request.jsonSee the documentation index for the complete guide map, the Quickstart for local setup, and the Explorer authoring guide for the current Builder contract.
The Viewer reads the current published table through the rows and aggregations
endpoints. Both require an authorized project and the exact selector
(recipe, translationVersion, output). Rows support field selection, filters,
sorting, and cursor pagination. Aggregations evaluate named specifications
against the same selected table. See the dataframe API guide.
The run endpoint compiles a FHIR-shaped request against the project's active generation and executes the resulting scoped AQL. Explorer V2 authoring uses the same recipe compiler and owns publication.
In production, Loom uses either basic authentication or Calypr/Fence-backed
authorization. Calypr mode resolves the caller’s allowed
auth_resource_path values and applies them to graph reads and published flat
reads. Publication injects auth_resource_path into every ClickHouse output;
the flat reader treats it as an authorization predicate, never as a
client-controlled filter. project_id identifies the source project for
Explorer filters; it does not replace authorization paths.
Set --no-auth only for local development. It creates an explicit unrestricted
principal and must not be used in a deployment.
The server can be configured with YAML:
server:
listen: ":8080"
url: http://arangodb:8529
database: fhir_proto
clickhouse:
enabled: true
url: clickhouse://clickhouse:9000
database: loom
dataframer:
recipe: /etc/loom/dataframer.json
auth:
mode: calypr
calypr:
trusted_issuer: https://fence.exampleReplace https://fence.example with the deployment's fixed Fence issuer.
When auth.mode is calypr, auth.calypr.trusted_issuer is required; the
LOOM_AUTH_CALYPR_TRUSTED_ISSUER environment variable can supply it when the
YAML value is omitted. Loom requires the bearer token's iss claim to match
this configured issuer exactly, then asks that issuer's /user endpoint to
validate the token and return grants. Loom does not verify the token signature
itself or use an untrusted token claim to choose the authorization server.
Basic mode reads LOOM_AUTH_BASIC_USERNAME and LOOM_AUTH_BASIC_PASSWORD if
they are not supplied in the config file.
server.dataframer.recipe is required only when ClickHouse is enabled. Loom
reads and validates that recipe during startup; deployments can update it
without rebuilding the server image.
Project views are saved in Loom for each project. A view chooses published output tables, tabs, columns, filters, and charts; it contains no recipe or loading instructions. Several views can use the same published tables. The first publication creates a simple table view from that project's actual schema. Later publications keep the project's saved views and default choices.
Project-owned views and their published revisions continue to be refreshed transactionally when repository source updates are published. Their saved definitions remain part of project configuration; the standalone View Builder HTTP API has been removed.
| Path | Responsibility |
|---|---|
cmd/ |
Operator CLI, server executable, and developer tools. |
api/ |
Canonical Loom HTTP specification and generator configuration. |
fhir/ |
Source FHIR graph schema and generated Go code. |
fhir/model |
Public generated FHIR resource structs and validation helpers. |
fhir/schema |
Generated raw FHIR schema metadata. |
internal/fhir/schema |
Server-only FHIR schema metadata and selector semantics. |
internal/ingest |
NDJSON loading, validation, graph extraction, and ingest lifecycle. |
internal/dataset |
Immutable generation and active-manifest contracts. |
internal/catalog |
Evidence of populated fields, references, and authorization paths. |
internal/explorer |
V2 authoring intent, resolved Builder models, server compilation receipts, immutable revisions, publication state, and legacy migration types. |
internal/dataframe/compiler |
Typed plan IR, lowering, optimization, and AQL rendering. |
internal/dataframe/recipe |
Recipe contract, validation, schema resolution, execution, and control services. |
internal/dataframe/publication |
Backend-neutral bounded streaming publication contract. |
internal/dataframe/published |
Safe single-project published-data reads and aggregates. |
internal/store/arango |
ArangoDB boundary. |
internal/store/clickhouse |
Typed ClickHouse driver boundary and DDL/DML. |
internal/api/dataframe/query |
Arango graph and FHIR dataframe API services. |
internal/api/dataframe/dataframe |
Published-dataframe reads, aggregates, export, and response mapping. |
make build # server and CLI binaries
make generate-fhir # generated FHIR structs and schema metadata
make generate-openapi # Loom HTTP models and strict Fiber server
make test # full Go test suite
make conformance # compiler conformance corpusThe generated FHIR metadata and OpenAPI bindings are checked in. Regenerate
them when their inputs change.
docs/CODE_GENERATION.md explains every generated
directory, its source of truth, and the required regeneration order.
For a direct build and full verification:
go build .
go test ./...- Documentation index
- OpenAPI contract
- Quickstart
- Explorer authoring contract
- Default dataframer recipe authoring guide
- Fast local Explorer development
- Explorer dataframe backlog
- Dataframer recipe reference and operating manual
- Dataframe API guide
- Developer architecture
- Experimental local stack
The Helm chart lives in the separate gen3-helm repository under helm/loom.