Skip to content
calyprPublic

About

A visual cohort and dataset builder for FHIR research data.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Loom

Loom turns FHIR-shaped data in ArangoDB into authorized, queryable flat dataframes in ClickHouse.

ArangoDB is the canonical graph store: Loom loads NDJSON, extracts FHIR resources and relationships, profiles the populated graph, and compiles typed dataframe recipes into scoped AQL. ClickHouse is the publication store: Loom streams resolved recipe outputs into versioned physical tables, records their schema and visibility in an Arango-backed catalog, and exposes them through a stable reader API.

Loom is deliberately not an arbitrary AQL gateway, ClickHouse SQL proxy, or a replacement FHIR model. It is a recipe-driven graph-to-flat data service with explicit authorization and publication boundaries.

Explorer Builder authoring is intent-driven: the browser submits the versioned V2 authoring document, Loom lowers it to a native recipe, and the existing recipe compiler produces the scoped plan/AQL. The browser never constructs or repairs the recipe AST. See the Explorer authoring contract.

flowchart LR
    NDJSON["FHIR NDJSON"] --> Ingest["Ingest and catalog"]
    Ingest --> Arango["ArangoDB\nresources, fhir_edge, field catalog"]
    Recipe["Versioned dataframe recipe"] --> Compile["Resolve, validate, compile"]
    Arango --> Compile
    Compile --> AQL["Scoped AQL stream"]
    AQL --> Publish["Atomic publication catalog commit"]
    Publish --> ClickHouse["Versioned flat tables"]
    Publish --> Catalog["Arango publication catalog"]
    Catalog --> Reads["REST published rows and aggregations"]
    ClickHouse --> Reads
    Compile --> Dataframe["REST FHIR dataframe query"]
Loading

What Loom owns

  • FHIR NDJSON ingestion into ArangoDB resource collections and fhir_edge.
  • Immutable dataset generations and active-generation selection.
  • Populated-field, reference, traversal, and pivot discovery.
  • Strict, versioned dataframe recipes and compiler-backed AQL execution.
  • Scoped publication of recipe outputs into ClickHouse.
  • A durable Arango catalog that maps exact dataframe selectors to current published outputs.
  • Multi-project, project_id-identifiable and auth_resource_path-scoped flat-data reads.

Loom does not expose arbitrary ClickHouse tables, arbitrary SQL, or an Elasticsearch/Guppy fallback. Published dataframe reads require an exact recipe, translation-version, and output selector.

Runtime surfaces

Surface Purpose
arango-fhir-proto Operator CLI for loading data, loading immutable generations, catalog discovery, and local dataframe materialization.
arango-fhir-server HTTP server for graph compilation/control and flat dataframe reads.
POST /api/v1/projects/{project}/dataframes/rows Page through a published dataframe.
POST /api/v1/projects/{project}/dataframes/aggregations Run named aggregations on a published dataframe.
POST /api/v1/projects/{project}/dataframes/run Compile and run a FHIR dataframe query against ArangoDB.
POST /api/v1/projects/:project/explorers/... REST Explorer lifecycle and V2 intent authoring used by the Builder.
POST /api/v1/datasets/:project/generations/:generation Multipart immutable-generation loader used by ETL.

The canonical contract for every server route is api/openapi.yaml. It includes health, generation ingestion/activation, recipe execution, dataframe queries, Explorer lifecycle, and Explorer authoring operations. The :project path parameter is the tenancy identity used for authorization and becomes the published row project_id (for example, HTAN_INT-BForePC).

Data lifecycle

  1. Load FHIR NDJSON into ArangoDB. Loom creates resource collections, fhir_edge, and fhir_field_catalog.
  2. For production snapshots, load an immutable dataset generation and let the active manifest select it for a project.
  3. Resolve a registered recipe against the populated graph and caller scope.
  4. Compile the resolved recipe to parameterized AQL and stream rows from ArangoDB.
  5. Publish output streams to new ClickHouse physical tables. Loom injects the reserved project_id and auth_resource_path columns, records schema/provenance in Arango, and atomically advances the logical publication pointer only once every output is READY.
  6. Query the logical dataframe through the project-scoped rows endpoint. The reader resolves authorized current outputs from the catalog; it never accepts a physical ClickHouse table name.

The configured ClickHouse database is created by the server during startup if necessary. Publication creates its tables itself. No DDL or alias-management HTTP endpoint is required. The ClickHouse account needs database creation plus CREATE, INSERT, SELECT, and DROP privileges.

Local development

To run the complete no-auth Explorer demo with the locked FHIR Aggregator sample, use:

make demo-up

Open the standalone Explorer. See Run the standalone Explorer demo for the seeded dataset, smoke checks, logs, and reset commands.

To launch from a separate repository's hydrated META data and native Loom CONFIG workspace, including local CONFIG write-back on Publish, see Launch Loom from a data repository.

For warm backend and Explorer UI iterations with an isolated Compose project and real browser verification, see Fast local Explorer development. The public make verify-* targets run native Playwright Test cases. verify-current and verify-full temporarily edit watched source for HMR checks, so run them one at a time.

For backend-only development, the lightweight Compose stack starts ArangoDB and ClickHouse:

rtk docker compose --project-directory . -f tests/deploy/compose.local.yaml up -d
make generate-fhir
make build

Load the bundled example data into a mutable local namespace:

./bin/arango-fhir-proto load \
  --url http://127.0.0.1:8529 \
  --database fhir_proto \
  --meta-dir META \
  --project ARANGODB_PROTO \
  --auth-resource-path EllrottLab-GDC_Data

For a production-shaped immutable snapshot, use load-generation instead. It requires an opaque generation identifier and deliberately forbids truncation:

./bin/arango-fhir-proto load-generation \
  --url http://127.0.0.1:8529 \
  --database fhir_proto \
  --meta-dir META \
  --project ARANGODB_PROTO \
  --generation local-meta-1 \
  --auth-resource-path EllrottLab-GDC_Data

Start Loom locally with an unrestricted development principal:

./bin/arango-fhir-server \
  --listen :8080 \
  --no-auth \
  --url http://127.0.0.1:8529 \
  --database fhir_proto \
  --clickhouse-url clickhouse://127.0.0.1:9000 \
  --clickhouse-database loom \
  --dataframer-recipe /path/to/dataframer.json

Add --dataset-generations when reading the active immutable generation. That mode disables the legacy one-file import endpoint.

Useful local URLs:

Run a saved dataframe request after the server is up:

make dataframe-demo DATAFRAME_REQUEST=/path/to/request.json

See the documentation index for the complete guide map, the Quickstart for local setup, and the Explorer authoring guide for the current Builder contract.

Dataframe queries

The Viewer reads the current published table through the rows and aggregations endpoints. Both require an authorized project and the exact selector (recipe, translationVersion, output). Rows support field selection, filters, sorting, and cursor pagination. Aggregations evaluate named specifications against the same selected table. See the dataframe API guide.

The run endpoint compiles a FHIR-shaped request against the project's active generation and executes the resulting scoped AQL. Explorer V2 authoring uses the same recipe compiler and owns publication.

Authorization and tenancy

In production, Loom uses either basic authentication or Calypr/Fence-backed authorization. Calypr mode resolves the caller’s allowed auth_resource_path values and applies them to graph reads and published flat reads. Publication injects auth_resource_path into every ClickHouse output; the flat reader treats it as an authorization predicate, never as a client-controlled filter. project_id identifies the source project for Explorer filters; it does not replace authorization paths.

Set --no-auth only for local development. It creates an explicit unrestricted principal and must not be used in a deployment.

The server can be configured with YAML:

server:
  listen: ":8080"
  url: http://arangodb:8529
  database: fhir_proto
  clickhouse:
    enabled: true
    url: clickhouse://clickhouse:9000
    database: loom
  dataframer:
    recipe: /etc/loom/dataframer.json

auth:
  mode: calypr
  calypr:
    trusted_issuer: https://fence.example

Replace https://fence.example with the deployment's fixed Fence issuer. When auth.mode is calypr, auth.calypr.trusted_issuer is required; the LOOM_AUTH_CALYPR_TRUSTED_ISSUER environment variable can supply it when the YAML value is omitted. Loom requires the bearer token's iss claim to match this configured issuer exactly, then asks that issuer's /user endpoint to validate the token and return grants. Loom does not verify the token signature itself or use an untrusted token claim to choose the authorization server.

Basic mode reads LOOM_AUTH_BASIC_USERNAME and LOOM_AUTH_BASIC_PASSWORD if they are not supplied in the config file.

server.dataframer.recipe is required only when ClickHouse is enabled. Loom reads and validates that recipe during startup; deployments can update it without rebuilding the server image.

Project views are saved in Loom for each project. A view chooses published output tables, tabs, columns, filters, and charts; it contains no recipe or loading instructions. Several views can use the same published tables. The first publication creates a simple table view from that project's actual schema. Later publications keep the project's saved views and default choices.

Project-owned views and their published revisions continue to be refreshed transactionally when repository source updates are published. Their saved definitions remain part of project configuration; the standalone View Builder HTTP API has been removed.

Repository guide

Path Responsibility
cmd/ Operator CLI, server executable, and developer tools.
api/ Canonical Loom HTTP specification and generator configuration.
fhir/ Source FHIR graph schema and generated Go code.
fhir/model Public generated FHIR resource structs and validation helpers.
fhir/schema Generated raw FHIR schema metadata.
internal/fhir/schema Server-only FHIR schema metadata and selector semantics.
internal/ingest NDJSON loading, validation, graph extraction, and ingest lifecycle.
internal/dataset Immutable generation and active-manifest contracts.
internal/catalog Evidence of populated fields, references, and authorization paths.
internal/explorer V2 authoring intent, resolved Builder models, server compilation receipts, immutable revisions, publication state, and legacy migration types.
internal/dataframe/compiler Typed plan IR, lowering, optimization, and AQL rendering.
internal/dataframe/recipe Recipe contract, validation, schema resolution, execution, and control services.
internal/dataframe/publication Backend-neutral bounded streaming publication contract.
internal/dataframe/published Safe single-project published-data reads and aggregates.
internal/store/arango ArangoDB boundary.
internal/store/clickhouse Typed ClickHouse driver boundary and DDL/DML.
internal/api/dataframe/query Arango graph and FHIR dataframe API services.
internal/api/dataframe/dataframe Published-dataframe reads, aggregates, export, and response mapping.

Build, generation, and tests

make build                 # server and CLI binaries
make generate-fhir         # generated FHIR structs and schema metadata
make generate-openapi      # Loom HTTP models and strict Fiber server
make test                  # full Go test suite
make conformance           # compiler conformance corpus

The generated FHIR metadata and OpenAPI bindings are checked in. Regenerate them when their inputs change. docs/CODE_GENERATION.md explains every generated directory, its source of truth, and the required regeneration order.

For a direct build and full verification:

go build .
go test ./...

Further reading

The Helm chart lives in the separate gen3-helm repository under helm/loom.

About

A visual cohort and dataset builder for FHIR research data.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages